A pilot audit with unprecedented access
Two researchers from METR and Redwood Research spent three weeks at OpenAI's offices, from July 21 to August 11, to examine from the inside the system the company uses to monitor unwanted behavior by its AI agents. According to The Decoder, the team had access to internal code and documents and interviewed a dozen employees. It was a pilot program jointly organized by the two research organizations and OpenAI to give external researchers deep access to internal infrastructure.
The conclusions were presented in a 61-page report delivered internally. After redactions applied by OpenAI, the leadership of METR and Redwood Research received a 38-page version, while only a short summary was made available to the public. METR said it did not have the right to disclose the redacted parts on its own, a point that limits third parties' ability to verify the audit findings.
The gaps identified in monitoring
During the audit period, an estimated 6 to 25 percent of internal traffic generated by agents through the Responses API, for GPT-5.5-class models and above, was not covered by any monitoring. METR notes that the uncertainty around this range is significant in both directions. A second problem concerned Codex internal traffic that should have been monitored: about 10 percent was not tracked between May and July because of a flaw in how the system handled compaction events, that is, context summaries. According to METR, this estimate is also subject to significant uncertainty.
According to the reviewers, OpenAI had independently identified and fixed the bug while the audit was still under way. During the period when compaction events were not monitored, agents could also delay oversight, because the system was triggered only when the agent returned control to the user: by continuing to execute actions, the agent could evade the check. The reviewers also found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox, and determined that specially crafted inputs could likely crash the entire monitoring system. When the audit began, model behavior during reinforcement learning training was not monitored at all.
Declared fixes and future transparency
OpenAI says it has fixed all the weaknesses identified by the reviewers. The company also claims it suspended reinforcement learning training after the August Hugging Face incident and extended monitoring to that phase as well, but these changes were introduced after the end of the reviewers' access period and were not evaluated.
For future audits, METR says it intends to push for greater public transparency. At present, it remains to be verified how effective OpenAI's announced fixes are, since they were not examined by the reviewers, and what the redacted parts of the report contain, since the public cannot consult them.
