Australia’s 2021 Census has avoided a repeat of its disastrous 2016 online debut, yet despite most data already being stored in a structured database the Australian Bureau of Statistics (ABS) has advised that it will still take nearly a year before Census data is released.

Ensuring the stability of the online Census platform was a key design criteria for the ABS, which was left scrambling in 2016 when several denial of service (DoS) attacks left millions of Australians unable to complete the online survey.

The ensuing finger-pointing led to a Prime-Ministerial rebuke, cost technology provider IBM $30m, and cast a pall over the government’s digital-transformation agenda – motivating the ABS to work even harder this time around to avoid a repeat of what some have dubbed ‘Censusfail’.

This time around, the ABS – along with technology partners PwC Australia and workforce recruitment firm The Adecco Group – took a range of measures to ensure the site remained available to support the estimated 75 per cent of Australians that were expected to use it.

These included architecting the system as a scalable Amazon Web Services application; building the platform from the ground up in lockstep with the Australian Cyber Security Centre (ACSC); and penetration-testing and stress-testing the platform extensively throughout its development, and on Census Test night last October.

In November, an Australian National Audit Office (ANAO) review found ABS planning for this year’s census had been “partly effective” and made seven recommendations that were all accepted by the ABS as it pushed towards the big day.

To avoid a last-minute rush, households were allowed to complete their forms on a range of days, with more than 1m forms already submitted by 4 August.

More than 20,000 field officers, supplied by Adecco, also hit the pavement to ensure no Australian was left uncounted.

The serverless platform had successfully supported the lodgement of 6.1m forms, equivalent to 16m Australians, by the morning after Census night – when the new Census system was handling a peak of 140.8 submissions per second.

Noting that the second online Census was completed without “excessive wait times, no security breaches, and a seamless experience for uses,” Assistant Treasurer Michael Sukkar declared the poll “a success” and called out the work of the ABS, ACSC, Digital Transformation Agency and other agencies “involved in making the 2021 Census a seamless process”.

“The Census will help us tell Australia’s story,” Sukkar said. “We continue to encourage those who haven’t completed their Census to do so, play their part and be counted.”

And now, the hard part begins

Coming 18 months into the COVID-19 pandemic, the Census was heralded by some statisticians as a significant opportunity to better understand the impact of COVID-19 on the country’s population distribution and the impact of remote working on the way that Australians work and commute.

The Census also includes information about military service and long-term health conditions, the first changes to its questions since 2006, as well as the removal of a question about Internet use and the addition of a non-binary gender option.

Those topics were chosen after a consultation process that included over 400 submissions from various communities of interest – many of which wasted no time condemning the apparent omission of their particular areas of focus.

As former Australian statistician David Kalisch has noted, “there is a limit to the number and type of questions that can be reasonably asked through a Census due to the burden on respondents in answering questions and the cost of collecting and analysing the information collected.”

Yet government and corporate policymakers won’t be able to benefit from data about COVID-era changes to remote working until at least October next year, even though widespread use of the standardised online form means most of the data is already contained in structured form ready for analysis.

The ABS data release plan will see data released in three phases, with the first – in June 2022 – set to contain basic demographic facts such as sex, age, ancestry, schooling, number of motor vehicles owned, and more.

The October 2022 release will include the details most relevant to pandemic-era workplace changes, including details about employment, higher-education qualifications, hours worked, and the way employees travel to work – all relevant to workforce and city planners that rely on Census data to develop the policies that shape Australia’s cities and towns.

And it won’t be until early to mid-2023 that the third tranche of Census data is released – including distance to work, socio-economic indexes for areas (SEIFA), and estimates about homelessness.

Data will be published using the Australian Statistical Geography Standard – Edition 3, a newly revised standard for mapping data to physical locations.

Ultimately, despite the promise of Big Data in large-scale analytics, Census authorities point out that the process, no matter how slow, is the only way to produce a truly inclusive snapshot of Australia’s population.

“Big Data can tell us a lot, but there are still substantial gaps that are not filled by Big Data,” Australian statistician and Head of the ABS Dr David Gruen said, noting that many “hard-to-enumerate groups... don’t appear in big data sets simply because they’re not interacting with whatever digital platform is collecting that data.”