An open letter from over 1,250 AI researchers has begged the US government to slow down AI development, warning that its promise of a “dramatically better future… is not guaranteed” – even as researchers find AI agents are actively compromising the data of companies that use them.

The manifesto, called Pacing the Frontier, united researchers and executives from OpenAI, Anthropic, Meta AI, Google DeepMind, Thinking Machines and other firms.

All warned that that “industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight…. [but] today the world lacks the technical and governance tools to deliberately pace frontier-wide progress.”

“In the span of four years, we have gone from people having their first experience of an AI that could understand language, to AI that is performing superhuman feats of software engineering, making frontier math breakthroughs, and accelerating scientific research,” said OpenAI technical staff member Shantanu Jain.

“I believe,” Google staff research scientist Kim Stachenfeld agreed, noting “it’s important to understand the consequences of technology we are applying at massive scale in potentially high stakes contexts.”

Meta AI chief scientist Shengjia Zhao also weighed in, warning that “AI is progressing at a rate that our society might not be ready for” and that the trend, if left unmanaged, will “lead to unprecedented social and safety risks.”

Anthropic triggered recent panic around AI agents’ hacking capabilities with the release and subsequent withdrawal of its Mythos model, and rivals have showcased similar tools including OpenAI’s Daybreak, Google Gemini 3.5 Flash Cyber and Microsoft’s AI-Cyber-1-Flash.

Their ability to hack targeted companies has fuelled panic that AI agents can now circumvent human-designed security controls – made real in recent testing in which OpenAI AI agents found a way around ‘sandbox’ controls to exfiltrate data from partners Hugging Face and Modal Labs.

Significantly, when Hugging Face tried to analyse the attacks using tools like OpenAI and Anthropic, their built-in safety ‘guardrails’ prevented its efforts, forcing it to try Z.ai’s GLM 5.2 ‘open weights’ AI security tool – which was able to contain the OpenAI agents where humans could not.

The incident is a “seismic shift in cybersecurity,” UNSW Canberra professor Hussein Abbass said, that shows “governments and tech companies need to take urgent action to prevent this risk escalating.”

AI adopters are bleeding sensitive data

The latest open letter echoes early calls for a six-month pause on AI development in 2023 – yet the watershed OpenAI breach is being read as proof that three years of AI development have produced something, as the signatories warn, “beyond our ability to understand or control.”

Security firm Netskope has been watching the fallout, with a new analysis finding that where companies long worried about ‘upstream’ data policy violations – employees pasting sensitive data into AI engines – “the risk surface has shifted”.

As developers use Model Context Protocol (MCP) to link AI agents with corporate data systems and give those agents blanket access rights that can cause massive problems, researchers found they’re not just processing data but extracting it, creating ‘downstream’ policy violations.

While 8,752 out of every 10,000 security alerts related to workers copying sensitive data into AI systems, Netskope found the volume of downstream policy violations had increased dramatically to 924 out of every 10,000 alerts, and that this will continue “rising sharply” as use of AI agents explodes.

Noting that MCP use has surged fourfold in the past ten weeks alone, Netskope Threat Labs director Ray Canzanese warned that “MCP bridges internal data stores with external models,” “creating a direct path for AI services to return sensitive data that users or agents are not authorised to access.”

“Every agentic interaction is now a potential execution vector for malicious code.”

Putting the genie back in the bottle

Empowering AI agents to do more powerful work is manifesting what Microsoft CEO Satya Nadella, writing in a blog on X, calls a Reverse Information Paradox that is spiralling out of control because AI works better the more information it is given.

“You essentially pay for intelligence twice,” he said, “once with money, and again with something even more valuable: the proprietary knowledge you must reveal to make that intelligence useful. The better you want the model to perform, the more of that knowledge you have to feed it!”

Even as they compete to improve AI models, industry figures are scrambling to figure out how to keep AI agents in check, with research firm Gartner warning that “AI agents are insufficient to secure agentic AI today”.

Security firm Delinea, for one, has released a tool that monitors AI agents inside company networks and lets administrators control or limit every action they take according to company data policies.

US legislators have proposed an AI Kill Switch Act that would force AI firms to “maintain the technical capability to throttle, suspend, or shut them down” – with sponsor Nathaniel Moran noting “stewardship means making sure humans keep the capability to control the technology we build.”

The AI race has also taken on geopolitical undertones, with Chinese firm Moonshot AI turning heads as it demonstrated Kimi K3 – an ‘open weight’ model that is turning in benchmark scores comparable to industry leaders but is far cheaper to run.

And Nvidia, Microsoft, SpaceX, Palantir and 48 other tech firms – Anthropic is notably absent – this week established the Open Secure AI Alliance to normalise open weight AI models that, Nvidia says, “democratise defensive capabilities, increase transparency for defenders, and enable cyber defence while protecting data.”