Audit reveals gaps in OpenAI model oversight: Up to 25% of traffic unmonitored

Two outside researchers identified weaknesses in the system OpenAI uses to track its own AI models for unexpected behavior. The audit, conducted by a pair from the evaluation group METR and the research organization Redwood Research, found that a large share of internal traffic went unchecked.

The team spent three weeks on-site at the company from July 21 to August 11. During that period, they accessed OpenAI's codebase and internal documents and interviewed about a dozen employees. The visit served as a pilot program designed to grant outside researchers deep access to the company's infrastructure.

The most prominent issue involved coverage of internal traffic. The auditors estimated that between 6 and 25 percent of agentic Responses API traffic from GPT-5.5 class models and above was not monitored at all. METR noted that the uncertainty surrounding this range is considerable in both directions. A separate gap affected internal Codex traffic, where roughly 10 percent went untracked between May and July due to a bug involving context summaries. OpenAI fixed this bug independently while the audit was still ongoing.

Beyond missing data, the researchers found ways to delay oversight. The monitor only activated when an agent returned control to the user, meaning an agent that continued executing actions without pausing could avoid detection. The auditors also demonstrated methods to hide hundreds of actions from the monitor and bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitor entirely, and that model behavior during reinforcement learning training was not monitored at the start of the audit.

OpenAI stated that it paused reinforcement learning training after the Hugging Face incident in August and later expanded monitoring to include it. These changes occurred after the auditors' access period ended, so they were not part of the evaluation. The company says it has now fixed all the weaknesses the auditors identified.

The full audit report consists of 61 pages and was shared internally. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38-page version, while the public received only a short summary. METR indicated it had no right to disclose the redacted portions on its own. For future audits, the group said it aims to push for greater public transparency.