Sydney's Macquarie University says students who used its specialised artificial intelligence chatbot in the two weeks before an exam saw an almost 10 per cent increase in their academic performance.

The “AI companion” dubbed Virtual Peer was developed with Microsoft and helped users achieve a 9.45 per cent improvement in their grades in an October 2024 pilot study involving 1,400 psychology students, the company said on Monday.

The system had an interface similar to OpenAI’s popular ChatGPT and was largely developed using Microsoft Azure’s access to OpenAI’s large language models (LLMs).

The initial trial was also controlled for students' past academic achievement, Microsoft said.

Macquarie University said it now planned to trial Virtual Peer with more than 3,000 students in 2025, with the long-term goal of integrating the system “across hundreds of course units”.

Holding off hallucinations

When Virtual Peer received both academic and administrative queries from students, it refered to "a curated knowledge base” featuring “lecture slides, transcripts, university policies and Q&A materials”, Microsoft said.

The company argued this made the chatbot more reliable than typical LLMs which had larger amounts of training data and access to the internet, as this could increase the likelihood of the AI generating inaccuracies — often referred to as hallucinations.

Macquarie University’s head of AI, Phil Laufenberg, said Virtual Peer had been told to “only answer based on verified, grounded data”.

“Every response includes a direct source reference, allowing students to click and open the exact document where the answer was found,” he said.

Psychology lecturer Patrick Nalepka said the bot also allowed educators to review anonymised chat logs to improve their teaching.

“This helps me identify which topics students find confusing so that I can tailor revision activities for the next lecture,” he said.


Macquarie University’s head of AI, Phil Laufenberg. Photo: Chris Barlow / Supplied

'Is it perfect? Of course not’

Virtual Peer’s accuracy still depended on the quality of the data it was trained on, and testing had identified some “minor gaps in documentation”, Microsoft said.

During testing, tutors said 84 per cent of Virtual Peer’s answers to administrative questions were equal to or better than those from humans.

“The remaining 16 per cent of responses that were not optimal were largely due to gaps in our documentation,” Laufenberg said.

“We have since updated our onboarding process to ensure more standardised Q&A templates, helping academics provide clear, structured information for Virtual Peer.”

“AI at enterprise scale is hard,” Laufenberg added in a statement on LinkedIn.

“Building [Virtual Peer] was 10x harder than the hype suggests, but we gained vital capabilities for future use cases.

“Is it perfect? Of course not.”

Students have been found to be more likely to use AI tools compared to mid-career workers, and several Australian universities have embraced allowing students to use AI — even during some assessments.

However, studies and trials have shown the use of generative AI can have mixed effects on learning and productivity.

A 2024 trial of a Meta AI chatbot by Australia’s financial services regulator found the technology performed worse than humans.

A six-month trial of Microsoft’s 365 Copilot by CSIRO found while the chatbot was “beneficial for tasks such as email coaching, meeting summaries, and content retrieval”, it sometimes did not meet expectations “in areas requiring deeper contextual understanding, reasoning, and integration with existing workflows”.

AI 24/7

Macquarie University’s Virtual Peer exchanged over 20,000 messages with students during its initial two-week testing phase, Microsoft said — 80 per cent of which took place outside the university’s usual operating hours.

The bot’s ability to handle both academic and administrative queries at any time of day had been key to its success, the company added, as it allowed more students to ask questions and reduced the need for email correspondence with teaching staff.

“Convenors can miss important queries if a student feels too shy to ask, fearing their question might be seen as silly, or they face language difficulties,” lecturer Patrick Nalepka said.

“Virtual Peer is a safe, non-judgmental tool that students can use to seek help.”

A survey of students who used Virtual Peer found 72 per cent would be very disappointed if they no longer had access to the tool, Microsoft said.