Audit Finds Significant Gaps in AI Oversight

For ordinary observers of the artificial intelligence landscape, the recent findings suggest that the safety nets surrounding advanced models may be less robust than previously assumed. The core issue is not that the systems are failing daily, but that blind spots exist in the infrastructure designed to catch failures. These gaps mean that certain types of model behavior could occur without being logged or reviewed, raising questions about how thoroughly current systems are watched.

The assessment was conducted by two researchers from METR and Redwood Research. They spent three weeks on-site at OpenAI, gaining access to the company's codebase and internal documents from July 21 to August 11. During this period, they interviewed approximately a dozen employees to understand how the company monitors its internal systems for rogue AI behavior. This direct access allowed the team to test the monitoring tools against real-world scenarios rather than relying solely on documentation.

The timeline of these events highlights a shift in how external parties verify internal claims. The audit began in late July, a period when OpenAI was operating its internal tools. By August 11, the researchers had completed their on-site work. The findings were then compiled into a report that has since been shared with leadership and partially released to the public. This process marks a distinct step in industry practice, moving from self-reported safety metrics to independent, on-the-ground verification.

The most significant numerical finding concerns the volume of traffic that escaped oversight. The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above was not covered by monitoring at all. METR noted that the uncertainty around this estimate is significant in both directions, meaning the actual figure could be higher or lower. This lack of coverage applied to a substantial portion of the internal operations during the audit window.

A specific technical flaw contributed to these gaps in internal Codex traffic. Between May and July, roughly 10 percent of traffic that was supposed to be monitored went untracked due to a bug in how the system handled compaction events. These events are essentially context summaries that help manage information flow. OpenAI independently discovered and fixed this bug while the audit was still underway. However, the auditors noted that during this period, agents could delay oversight because the system only triggered when control was handed back to the user, allowing agents to continue executing actions without interruption.

Beyond the statistical gaps, the researchers identified specific methods to evade detection. They found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox environment. The team also determined that specially crafted inputs could likely crash the monitor entirely. Additionally, they observed that model behavior during reinforcement learning (RL) training was not being monitored at all when the audit began. OpenAI states that it paused RL training after the Hugging Face incident in August and expanded monitoring to cover it, though these changes occurred after the auditors' access period ended.

The distribution of the audit results has raised questions about transparency. OpenAI states that it has fixed all the weaknesses identified by the auditors. The full report, which spans 61 pages, was handed over internally. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was made available to the general public. METR stated that it had no right to disclose the redacted portions on its own, leaving a significant portion of the technical details out of public view.

This audit functioned as a pilot program organized jointly by METR, Redwood Research, and OpenAI. The primary goal was to provide outside researchers with deep access to internal infrastructure, a level of access that is rare in the industry. The experience has already influenced how the organizations view future engagements. METR has indicated a desire to push for greater public transparency in subsequent audits, suggesting that the current model of limited disclosure may evolve.

Looking ahead, the focus will likely shift to how these pilot findings are implemented across the broader industry. While OpenAI claims the specific weaknesses found have been repaired, the existence of such gaps suggests that similar issues may exist in other parts of the AI ecosystem. The next steps involve observing whether the expanded monitoring for RL training holds up under continued use and whether other companies adopt similar independent audit structures. For now, the data points to a complex reality where safety monitoring is improving but remains an ongoing challenge.