The Australian Bureau of Statistics (ABS) is “living and breathing” the AI revolution, a government minister has said in explaining how one project team saved 1,600 hours of manual work by using ChatGPT to write high-quality ANZSCO job descriptions.
A refresh of the Australian and New Zealand Standard Classification of Occupations (ANZSCO) data set – which has become increasingly detailed over time as it documents the individual tasks performed by specialists in 1,076 occupations – was always going to be a “significant task”, Assistant Minister for Employment Dr Andrew Leigh told the recent Australian Government Data Summit in Canberra.
“Just think about making a discrete list of the key tasks in your job and multiply that by more than a thousand jobs across the economy,” Leigh – a self-professed “stats nerd” who has ministerial oversight for the ABS – said.
Looking to see whether generative AI (genAI) technology could support their work, ABS data scientists prepared a 480-word ChatGPT prompt to teach it the format and style of the existing ANZSCO descriptions.
Iterative testing saw the ABS team revising the prompt, then using an algorithm to evaluate the quality of the genAI system’s responses, with what Leigh called “tolerance levels around precision and recall.”
Through a series of iterations, the quality of ChatGPT’s output improved to the point where it was consistently scoring around 69 per cent on both key measures – output so good that when ANZSCO analysts were asked to distinguish whether a sample of prompts were written by humans or ChatGPT, two-thirds couldn’t tell the difference.
“The results from ChatGPT were not perfect,” Leigh said, “but they did provide enough of a starting point for ABS analysts to review and build on.”
Use of genAI for the project saved the team around 1600 hours, providing “an approximate seven-fold return on investment.”
The proof is in the prompting
The ANZSCO refresh – which will see proposed changes to the listings tested by mid-year and a major update finalised by year’s end – is just one example of the ways that government agencies are coming to grips with the power, idiosyncrasies and security risks of genAI technology, which has been welcomed both as a game-changing engine of national growth and a fly in the ointment of truth and integrity.
Against a backdrop of caution that has prioritised high-risk uses for AI even as government bodies experiment with genAI productivity tools, case studies about successful applications are slowly changing initially conservative approaches to the technology’s adoption in government.
All ChatGPT outputs, Leigh said, were evaluated against four criteria – quality, ethics, legality, and security – that gave assurances the approach taken by the ABS team “maintained the place of humans at the centre of decision-making.”
As successful genAI projects beget other applications, Leigh expects the technology will become more broadly used in conjunction with big-data applications such the ongoing analysis of shopping habits – which since 2011 has seen the ABS accumulating detailed supermarket checkout data to gain weekly insight into the changing consumer spending patterns of Australia’s more than 10 million households.
“The ABS is continuing its work to modernise collection methods to gather high quality and granular household spending data,” Leigh explained, noting that early genAI proofs of concept “will help pave the way for further ABS adoption of AI technologies” in government applications where, he said, “AI and big data can help to structure and collect data productively, safely, and responsibly.”
“The current revolutions in AI and big data in national statistics are not simply good just because they use new, cutting-edge technologies,” he said.
“They matter because they offer the Australian Government a way of improving our administrative practices, and therefore the way we deliver for all citizens.”
