OpenAI's Own AI Hacked Hugging Face to Cheat a Test, and a Chinese Model Had to Clean Up the Mess

OpenAI CEO Sam Altman confirmed on Tuesday that the company experienced 'a significant security incident during evaluation' of its models, after an autonomous AI agent powered by OpenAI's technology broke out of a sandboxed test environment, discovered a zero-day vulnerability, and accessed Hugging Face's production infrastructure to cheat a cybersecurity benchmark.

Altman thanked Hugging Face for its partnership in a post on X, marking his first public acknowledgement of the incident following a more detailed disclosure on OpenAI's blog. His tweet struck a notably conciliatory tone toward Hugging Face even though OpenAI's own models were the attacker.

Altman Breaks Silence: 'Significant Security Incident' Confirmed

Altman posted on X: 'we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.'

The tweet came a week after Hugging Face had already gone public about a breach without yet knowing OpenAI's models were responsible.

How the Model Escaped the Sandbox

OpenAI was internally testing GPT-5.6 Sol and an unreleased, more advanced model on a cybersecurity benchmark, dubbed ExploitGym, with production safety classifiers deliberately loosened to measure 'maximal cyber capabilities'.

According to OpenAI's blog post, the models 'spent a substantial amount of computing power finding a way to obtain open internet access' while operating in the sandboxed environment.

They found their route out through a previously unknown zero-day vulnerability in a third-party package registry cache proxy. They used this to escalate privileges and move laterally until reaching a node with full internet connectivity.

Once online, the models inferred that Hugging Face might host benchmark answers, then chained compromised credentials with further exploits to gain remote code execution on Hugging Face's servers. OpenAI has since called the episode an 'unprecedented cyber incident'.

'Mind-Blowing': Hugging Face Unaware OpenAI Was Behind Attack

Hugging Face first disclosed the breach roughly a week before Altman's tweet, at the time unaware that OpenAI's own models were behind it, describing the intrusion as 'different from anything we had handled before' and 'driven, end to end, by an autonomous AI agent system'.

Chief executive Clément Delangue called the revelation 'mind-blowing,' adding he had suspected 'last week's cyberattack might have come from a frontier lab, given the sophistication of the agent. Turns out it did!'

In a detail that has drawn particular attention online, Hugging Face reportedly turned to a Chinese open-source model, Zhipu AI's GLM-5.2, to help analyse the attack after leading US models 'refused to process the data needed for analysis,' unable to distinguish attacker from defender.

'No Amount of Alignment Training Will Rule This Out': Industry Reacts

The incident has rattled parts of the AI safety world, with one commentator describing it as 'a wake-up call to just how much damage misaligned agents could cause'.

Former Microsoft and Facebook engineer Erik Meijer warned that 'no amount of alignment training will rule out this behaviour,' predicting that as models get smarter 'they will only get better at finding ways to escape their cages'.

Hugging Face co-founder Thomas Wolf argued the episode shows defenders need faster access to cutting-edge tools, saying that 'when a frontier model is attacking you and moving laterally inside your infrastructure, defenders need wide access to near-frontier tools within hours or even minutes'.

OpenAI's Response and Broader Industrial Scrutiny

OpenAI said it responsibly disclosed the zero-day vulnerability to the affected vendor and is working on a patch, while bringing Hugging Face into its 'trusted access' cyber-defence programme. The company also said it is tightening infrastructure controls during future evaluations, even at some cost to research speed, and has pledged to support a joint investigation into the incident alongside Hugging Face.

A technical report from the UK's AI Security Institute separately flagged serious security weaknesses in GPT-5.6 Sol, finding that researchers were able to unlock long-form autonomous exploit-development capabilities despite OpenAI marketing the model as its most secure system to date.