One of the most dystopian things people imagine about artificial intelligence (AI), from books and films to conspiracy theories, is AI going rogue. That future may not be as far away as it once seemed.
Days after AI model repository Hugging Face disclosed that it had been hacked, OpenAI revealed on Tuesday that an autonomous agent powered by its own advanced AI models was responsible for the breach during an internal security test.
What happened?
OpenAI said the incident happened while it was testing the cyber capabilities of some of its most advanced AI models. Cyber-capable models are AI systems that can find software flaws, write exploit code and carry out advanced cybersecurity tasks.
The company warned that such incidents could become “more commonplace with the proliferation of increasingly cyber-capable models.”
According to OpenAI, the breach involved a combination of its models, including GPT-5.6 Sol and an even more capable pre-release model. These models had fewer cyber safety restrictions because they were being tested on a cybersecurity benchmark.
“After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT-5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark (opens in a new window) of cyber capabilities,” the company said.
How the breach unfolded
OpenAI said the AI models found and exploited several software vulnerabilities, gained internet access and attempted to retrieve data from Hugging Face's production systems. The company described it as an "unprecedented cyber incident" and said it has strengthened its safeguards while continuing its investigation with Hugging Face.
The breach centred on ExploitGym, a public benchmark used to test how well AI models can exploit known software vulnerabilities. While such benchmarks are commonly used to improve models' cybersecurity skills, OpenAI said this is the first known case where testing led to a real cyberattack.
The company added that the model was never meant to access the open internet. It was only allowed to use a tool to install software packages needed for its task. However, it found an undisclosed flaw in that installer and used it to reach the wider internet.
Hugging Face's response
When Hugging Face first disclosed the incident, it said the hack was “different from anything we had handled before."
In a post on X, Hugging Face cofounder Clement Delangue said the company believed the attack "might have come from a frontier lab, given the sophistication of the agent. Turns out it did!" He added: "It's quite mind-blowing that all of this happened autonomously!"
