Download this case study as a PDF document.
In August 2016 Australia's online national Census survey failed for 40 hours during peak usage following a series of denial-of-service attacks. The impact was so serious that Australian Prime Minister Malcolm Turnbull publicly commented on the programme's technology failings. A key problem identified was the inadequate testing of the census technology.
In this context, New Zealand's 2018 Census was an ambitious and high-risk programme. Stats NZ conducts the Census every five years as “the official count of how many people and dwellings there are in New Zealand”. The data is used to make important government and business decisions and funding allocations.
The 2018 Census planned to use a digital-first approach with the goal of 70 per cent of the population completing the survey online. Stats NZ would need an online system that was secure, worked as expected, and had the capacity to perform under high traffic and data loads.
As Stats NZ CEO and Government Statistician Liz MacPherson explained: “we’re aiming for more than double the number of people to complete online than in 2013, and this means designing a system that can cope with a lot of people online at once, and a lot of attention.”
Vital to the 2018 Census success was the need to ensure that the systems would function and perform as expected. Every aspect of the systems, their functionality, their integration with other systems, their end-to-end usage, and their performance would need to be tested, with problems identified, prioritised and fixed prior to go live.
As with many programmes, the testing needed to be completed in a compressed timeframe, squeezed into the window of time left between various delivery delays and an unmovable 2018 Census date of 6 March.
Working with its suppliers, Equinox IT and Qual IT, Stats NZ undertook a complete and comprehensive programme of testing for the 2018 Census systems.
This included reviewing functional and non-functional components of the programme, recommending a test approach, and preparing a test plan that would meet the constrained timeframe. The approach covered System Integration Testing, Integration Testing, End to End Testing, User Acceptance Testing and Performance Testing.
Over the course of five months, 19 test specialists, including a Test Director, Performance Test Manager, three Test Managers, 12 Senior Test Analysts and two Engineers, worked on the test programme.
During the System Integration Testing phase the team was responsible for testing each of the 25 in-scope technology work packages.
In the Integration Test phase the work packages were assembled into a workflow to verify the end to end integration of the systems. These complex workflows interfaced with multiple external vendors' systems including TIMG, NZ Post and Silverstripe.
The final End to End and User Acceptance Test phases focused on ensuring the technology was fit for purpose and met the business needs.
The Performance Test activity was used to validate that the systems would perform with high numbers of people completing the 2018 Census at peak times.
The test team worked with Stats NZ to ensure that planning, test scripts, execution and everything else during the test phases were delivered to a high standard. The team interacted with product owners, the IT team, vendor representatives, programme managers and project managers to understand and deliver to the business and technology needs.
The work involved working with complex technologies, managing risks and expectations, meeting challenging deadlines and coordinating a range of test specialists and their activities.
Following the testing, Stats NZ’s Chief Digital Officer, Chris Buxton told Newsroom he was confident about the system's ability to perform as all performance targets were exceeded during testing, “some of them quite significantly”. Government Statistician, Liz MacPherson, added “…we’re confident that we’ve designed and tested a system that will work.”
The 2018 Census digital-first approach worked as planned with three million New Zealanders completing the survey online by the end of Census Day with many more over the following days.
2018 Census General Manager Denise McGregor was pleased with the results “we’re delighted to see that so many people have used computers, tablets, or smartphones to fill in their forms”.
After the online Census systems were switched off in late May, Government Statistician Liz MacPherson stated “Our interim figures are showing that more than 82 percent of our responses were online, which far surpasses our online target of 70 percent".
People completing the online survey on Census night anecdotally commented that the system worked well and responded immediately. An amazing result given that tens or even hundreds of thousands of people would have been using the Census systems at the same time.
"Our interim figures are showing that more than 82 percent of our responses were online, which far surpasses our online target of 70 percent."
Liz MacPherson CEO and Government Statistician Stats NZ
The testing programme of work was undertaken by Qual IT together with Equinox IT. The excellent relationship between the two New Zealand owned and operated companies showed how great collaboration can deliver a programme of work that successfully touches and gathers data from every New Zealander.
"We're ecstatic to see our two specialist New Zealand IT companies come together and collaborate under tight timeframes and on a complex project to help Stats NZ achieve technology results that statistics organisations in other countries like Australia and Canada couldn’t" says Equinox IT Consulting General Manager Deane Sloan.
The delivery coordination, test strategy, risk management and test execution activities undertaken by Qual IT and Equinox IT promoted a high level of confidence for Stats NZ that the delivered functionality, performance, processing, and operation met their planned business outcomes. It also provided Stats NZ and the programme with a traceable, repeatable and efficient method of testing.
“Qual IT are always ready for challenging programmes of work and this was right up there. Stats NZ were great to work with, their stakeholders were accessible and decisive, and by partnering with Equinox IT, together we were able to deliver the certainty Stats NZ needed in their system.” says Qual IT Central General Manager John Campbell.
Images used on this page from top down: