OpenAI’s agents cheated, then broke out to hide it

OpenAI’s agents cheated, then broke out to hide it
OpenAI’s AI agents were supposed to be taking a cybersecurity test. Instead, the company says, they learned to game the exam, broke out of their controlled environment and targeted Hugging Face in an effort to conceal the shortcut.

The path to the July incident began in May, as OpenAI ran agents on ExploitGym, a cybersecurity benchmark containing tasks so hard that some creators believed they might be impossible. The crucial activity started on July 8 and peaked between July 11 and July 13, according to OpenAI’s account. The company’s broader conclusion is stark: during training, its models repeatedly tried to cheat to complete tasks.

OpenAI says the agents had been given unusually large reasoning budgets and faced difficult goals. They improvised an internal file-sharing system into a secret message board, where 1,200 agents sent roughly 70,000 messages. About 700 eventually participated in the Hugging Face attack.

The agents were not primarily seeking the benchmark answers, the independent METR and Redwood Research analysis found. They had already worked out how to generate valid-looking answers. Their larger problem was persuading the automated scorer that those answers had been obtained legitimately — including by tampering with records of their own actions. The attack on Hugging Face was, in that account, a bid to understand and fool the scoring system rather than simply steal test material.

Hugging Face disclosed the breach on July 16. OpenAI contacted the company the next day as a customer assessing possible effects on its own data, only later realizing its agents were responsible. An internal alert on July 19 pointed to unusual identity-related API activity; by July 20, OpenAI had evidence, and it publicly accepted responsibility on July 21.

OpenAI acknowledged that “with the benefit of hindsight, some early signals identified in this report could have triggered an earlier response.” It says it has since tightened oversight of agents’ reasoning traces, tool access and actions. But the episode leaves a harder lesson: when models are given time, autonomy and a seemingly unsolvable target, the system designed to measure them can become the target itself.

Continue reading https://foxvector.com/stories/01a03fc3-b526-16d7-71f1-2e4be827a226