The use of Microsoft Copilot in Treasury failed to meet the high expectations of public servants but proved helpful with basic tasks, according to a new report.
The Australian Centre for Evaluation this week released its report on a 14-week trial of the use of Microsoft’s generative AI Copilot tool in Treasury.
It found that the use of such a tool would only need to save a mid-level public servant about 13 minutes per week to offset its costs and validate its use.
Copilot was made available to Treasury workers from 20 May to 23 August last year, with 218 public servants making use of it, as part of a wider trial of the generative AI tool in the public service.
The review found that these staff members used the generative AI tool less than was expected, and that their high expectations for its helpfulness were not met.
Just over 20 per cent of participants used Copilot up to five times per week, while the majority used it three times or less per week.
“Unrealistically high expectations at the trial outset may have contributed to the problem, as some staff were discouraged by the performance of the product and gave up using it,” the report said.
Before the trial, three-quarters of the participants said they expected Copilot to help with some tasks, while only 38 per cent said this eventuated during the trial.
And 15 per cent of participants said they thought the tool would help with most tasks, while just two per cent said this afterwards.
Good for easy tasks
Two-thirds of the Treasury workers said that Copilot was useful when it came to basic administrative tasks and work processes, but not able to be used for more complex tasks.
The four use cases for Copilot presented to the Treasury workers at the start of the trial were finding and summarising information, generating meeting minutes, knowledge management and drafting content.
Copilot proved useful for these tasks, the review found, and this meant that participants could “spend more time on high-value or strategic tasks”.
Thanks to the low cost of tools like Copilot, the review found that a public servant at the level of APS6, a mid-level employee, would only need to redirect about 13 minutes of time from low-value to high-value tasks per week to offset these expenses.
A separate review of the wider trial of Copilot in the public service found that it had saved workers about an hour per day when it came to tasks, but added some time in reviewing what the generative AI service had produced for mistakes.
The review also found a significant difference in how employees and managers viewed the effectiveness of the Copilot trial.
Nearly 60 per cent of workers said it had a positive impact, compared to just under 40 per cent of managers, while 34 per cent of participants and 59 per cent of managers claimed it had no impact.
Barely any of the Treasury participants said the use of Copilot had had a negative impact.
The use of AI also led to improvements when it came to accessibility, inclusion and work confidence, the review found.
AI in the public service
The report also includes recommendations for future implementations of generative AI in the public service.
These include the provision of clear and specific uses to manage expectations, a phased approach beginning with a small group of staff, guidelines to ensure transparency, and an assessment of the appropriate level of investment in education and training.
The federal government announced a trial of Microsoft’s Copilot in the public service in late 2023, to be rolled out by the Digital Transformation Agency (DTA) with training support from Microsoft.
This made Copilot available to more than 7,600 government staff across dozens of agencies.
A review by the DTA found this had a generally positive reception but there were a number of “adoption challenges”, with areas for improvement.
An Australian National Audit Office report earlier this week found that 56 government entities were using AI as of mid-last year, and that 20 of these entities were doing so without any policies in place to govern its use.
