ChatGPT maker OpenAI has taken responsibility for a hack on open-source AI platform Hugging Face after an AI agent went rogue during internal testing.
Last week, Hugging Face detected an “intrusion” into part of its production infrastructure that was driven “end to end” by an autonomous AI agent system.
While Hugging Face staff were working hard to determine what partner and customer data was affected, investigators at OpenAI came to the bizarre realisation that its own AI models had performed the attack.
“We now know that this particular incident was driven by a combination of OpenAI models,” wrote OpenAI.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly.”
The incident occurred while OpenAI was testing its models against a prominent industry benchmark to essentially determine whether its AI agents can turn security vulnerabilities into real hacks.
Although this testing was intended to be confined to a “highly isolated” environment, OpenAI’s models found a way to break out and reach Hugging Face’s systems on an external network.
After harvesting a range of system credentials, OpenAI’s agent moved laterally across Hugging Face’s environment over the course of a weekend.
Notably, Hugging Face said its own AI was largely involved in successfully detecting and analysing the attack.
“Autonomous, AI-driven offensive tooling is no longer theoretical,” Hugging Face wrote last week, before knowing OpenAI’s models were behind the hack.
“Defending an online platform now means treating the data and model surface as a first-class attack surface and using AI on defence to keep pace.”
Testing gone wrong
In a Tuesday blog post, OpenAI explained how GPT-5.6 Sol and an “even more capable pre-release model” effectively identified and combined a series of vulnerabilities in OpenAI’s research environment and Hugging Face’s production infrastructure.
While attempting to solve a research problem, the agent opted to obtain existing test solutions from Hugging Face’s production database.
“The model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation,” wrote OpenAI.
Although the benchmarking exercise ran in a “sandbox” environment, OpenAI allowed limited network access so the AI could install packages via an internally hosted third-party software.
Hugging Face CEO Clem Delangue announced his company would be partnering with OpenAI following the hack. Photo: YouTube
By identifying and exploiting a now-disclosed zero-day vulnerability, the AI models effectively escalated their own privileges and gained access to outside networks.
“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem,” wrote OpenAI.
OpenAI announces collab after hack
By the time OpenAI and Hugging Face got in touch about the incident, Hugging Face’s security team and agents had already detected and stopped the unauthorised activity on its infrastructure.
Rather than pointing the blame, however, Hugging Face instead used the incident as an opportunity to collaborate with OpenAI.
The two companies are jointly investigating the AI-driven hack, while Hugging Face has gained entry to OpenAI’s cybersecurity-focused ‘trusted access’ program.
“We're grateful for the collaboration with OpenAI on this and other topics,” said Hugging Face chief executive Clem Delangue.
“AI safety won't be solved by any single company working in secret.
“It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
Advanced AI, or subpar safeguards?
Notably, OpenAI said it estimates “maximal cyber capabilities” by foregoing the production classifiers that typically prevent models from pursuing high-risk cyber activity.
Following the hack, the AI-maker said it would add stronger protections around future training and evaluations.
When asked whether the incident was made possible thanks to frontier AI capabilities, or a simple lapse in OpenAI’s testing safeguards, Troy Hunt, founder of Have I Been Pwned, said it may have been “a bit of a combination of both”.
Troy Hunt of Have I Been Pwned said this may be the first time an AI has gone rogue. Photo: Supplied
“Clearly, this ‘sandbox’ was not exactly sandboxed physically from the world wide web,” Hunt told Information Age.
“What may be unique about this is whether it’s genuinely the first instance of an AI effectively going rogue — going beyond its intended scope, finding novel exploits and independently chaining them together.”
Andrew Philp, field chief information security officer for ANZ at TrendAI – the cybersecurity business of Trend Micro – said the incident represented both a “capability milestone” and a “process failure”.
“With the release of frontier AI models, autonomous cyber capability is now a reality, reinforcing that test environments are now part of the attack surface,” Philp told Information Age.
“Any enterprise environment testing autonomous AI, whether it’s a lab, sandbox or attack-path simulation, must be treated as high-risk.”
The incident, which entrepreneur Elon Musk described as “troubling”, followed stark warnings from rival AI giant Anthropic that its frontier model Mythos is powerful enough to challenge the foundations of modern cybersecurity.
Such concerns drove the US government to ban and subsequently unban exports on Anthropic’s frontier models, before requesting that OpenAI stagger the public release of GPT-5.6.
Notably, OpenAI used the hack as an opportunity to share performance results of its models compared to those of Mythos.
Hunt observed that the broader narrative among AI giants is that frontier models are proving “very powerful, can do harm in the wrong hands, and can even have some level of self-sentient consciousness”.
“If AI companies demonstrate that they are not able to contain and control their models themselves, what does that say for the rest of us?” asked Hunt.
