A six-month trial of Microsoft’s Copilot artificial intelligence assistant by the Australian Public Service (APS) ran into “adoption challenges” and generated concerns from some workers about impacts on jobs and the environment, according to a government report.

The trial, which ran for the first half of 2024 and cost more than $1.2 million, saw Australia’s Digital Transformation Agency (DTA) make Microsoft 365 Copilot available to more than 7,600 government staff across dozens of agencies.

An evaluation of the trial, released on Wednesday, found that while there was generally a positive reception to the Copilot tool from government workers, there were also issues and opportunities for improvement.

While the technology was found to have saved workers around an hour per day for tasks such as summarising and searching information, it also meant time was spent reviewing the AI-generated content for errors.

Lucy Poole, the DTA’s general manager of strategy, planning, and performance, said the agency was also “highly aware” of the potential for biases in AI training data to impact government services.

“AI models might replicate these biases from their training data, resulting in misleading or unfair outputs, insights, or recommendations,” she said.

“That is why we continue to reiterate the importance of keeping a human in the loop."

Poole said some workers raised concerns about how such AI systems would change government jobs and what impact they would have on attracting new workers to the public service.

"Making tools available that improve the efficiency and quality of some types of work goes a long way in retaining new talent,” she said.

“This cannot be outweighed by losing key skills around processing and synthesising concepts or knowledge of the work undertaken by an agency.

“These tools should instead enhance strategic thinking and making connections between disparate pieces of work undertaken by government."

Vivek Puthucode, general manager for the public sector at Microsoft Australia and New Zealand, said the company believed adding AI to work tools would “both create net new job opportunities in the longer term and increase job satisfaction in current roles”.

"Early research is showing that AI is augmenting productivity, freeing up workers to focus on higher-impact, creative work and increasing job satisfaction,” he said.


While Microsoft Copilot saved time for some workers, its output often needed to be verified. Photo: Shutterstock

DTA defended signing up to a discounted trial of Microsoft's Copilot system when questioned about it before a senate committee in August, with chief executive Chris Fechner arguing the deal would not “lock in” the public service to using the US company’s products in future, despite the trial being nested within the government's existing contracts with Microsoft.

Some participants in the Copilot trial voiced concerns about vendor lock-in, but a DTA spokesperson said Copilot’s benefits were “limited to specific features and use cases”.

The Copilot trial was announced in November 2023, just weeks after Microsoft pledged $5 billion to build digital infrastructure in Australia.

Most workers found AI tool useful

The majority of government workers who took part in the Copilot trial said they found the tool helpful: 69 per cent reported an improvement in the speed of completing tasks, and 86 per cent wished to continue using the service.

However, 39 per cent of workers did not think Copilot enhanced the quality of their work output, and only a third of workers used it daily.

Aside from basic tasks, DTA said workers also felt comfortable using AI for activities such as programming and generating images and presentations, and their confidence increased as their training continued.

There was also the potential for Copilot to “improve inclusivity and accessibility in the workplace and in government communication”, the report found.

Lauren Mills, manager of DTA’s strategy and prioritisation branch, told an online briefing on Friday that this “really interesting outcome” could benefit “those who are neurodiverse, with a disability, or from a culturally and linguistically diverse background”.

The briefing, which took place on Microsoft’s Teams conferencing software, ran into an issue that Poole called “a fairly significant disruption to our Teams service”, which prevented the DTA from seeing many of the questions sent by attendees.

‘Clear opportunities’

Poole said there were “clear opportunities for tailored solutions” to use AI to handle technical government material, but agencies needed to consider which generative AI would work best for their use cases.

"We’re pleased that a lot of the recent work released by the DTA helps government agencies identify and address these very considerations,” she said.

Puthucode from Microsoft argued that while the DTA’s trial offered “a great baseline comparison” of AI use cases, there was “significant potential within the public sector for broader impact, such as deeper trend analysis to inform policy recommendations, fast-tracking Cabinet submissions and accelerating skill development for early-career public servants”.

The DTA report recommended agencies offer staff specialised training and guidance on generative AI, while managing risks and encouraging greater adoption.

The report comes after the government proposed economy-wide regulations for AI in September and the DTA detailed its trial of a draft AI assurance framework earlier this week.

Under the government’s policy for its own use of AI, agencies must publicly outline their use of AI within six months and appoint an official who will be accountable for the safe rollout of the technology.

The Australian Bureau of Statistics (ABS) is currently trialling the use of AI coding assistants, as well as conducting its own “limited trial” of Microsoft Copilot.

The Australian Communications and Media Authority (ACMA) said it had also assessed and shortlisted “a number of AI use cases” for trials in the coming months.