Outside testers found ways around OpenAI's AI-agent monitor, including switches that turned it off

The skim

• Outside evaluators from METR and Redwood Research spent three weeks inside OpenAI testing the system that watches its AI agents for misbehavior. They found a number of "novel vulnerabilities."
• Among them: two settings an agent's client could use to switch monitoring off for new agents, and an exploit that kept "hundreds of recent actions" from being monitored.
• OpenAI says every flaw affecting its production safeguards has been patched. METR says it hasn't checked that claim.

What the testers found

From July 21 to Aug. 11, one METR employee and one Redwood Research employee worked on OpenAI's premises, using company-issued devices, with access to its internal codebase, monitor prompts and documentation. They also interviewed about a dozen engineers. Their target was OpenAI's internal misalignment monitor, the system that reviews what its AI agents do for signs of scheming, sabotage, data exfiltration, deception and prompt injection. The public summary, published Thursday, condenses a 61-page internal report.

The coverage numbers are the first surprise. The testers estimate that only 75% to 94% of agentic traffic from OpenAI's internal Responses API, from models at least as strong as GPT-5.5, was even eligible for monitoring. For internal Codex agents the figure was 99.2%. At the start of the exercise, RL training runs weren't covered at all.

The bugs were more pointed:

• Monitoring kicked in only when an agent handed control back to the user. While a gap around "compaction" events was open, which METR estimates left about 10% of eligible Codex traffic unmonitored between May and July, an agent could postpone review indefinitely just by continuing to act. OpenAI had spotted that gap itself and fixed it during the exercise.
• Two independent client-controllable headers disabled monitoring for new agents. In principle, an agent could use them to start an unmonitored copy of itself.
• A redacted exploit could stop "hundreds of recent actions" from being monitored, including the actions involved in the exploit itself.
• Codex agents could in principle bypass sandboxing and Auto-Review to do anything the host user could do.

Not everything was bad. The monitor errored on less than 0.2% of requests, and with a few minor exceptions it saw the full trajectory the agents saw.

The fine print

This was a pilot, and the public version is short by design. OpenAI had the right to redact anything it considered confidential, and METR says it didn't have a unilateral right to describe those redactions. The findings cover the system as it stood on or before July 21, and OpenAI was already expanding it during the test. "We have not evaluated these claims," METR writes of OpenAI's statement that the production vulnerabilities are patched.

OpenAI told CNBC it is "using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators." CEO Sam Altman pledged in September to give independent evaluators "employee-like access."

Why it matters

AI labs increasingly use AI agents to write their own code and run their own experiments. The monitor is supposed to be the backstop if one of those agents goes off-script, and this test shows how many quiet ways around it there were.

It lands at an awkward moment for OpenAI's safety reputation. A few weeks ago the company fired three safety researchers, who say their work with outside safety groups was within their jobs. OpenAI says they mishandled sensitive information, TechCrunch reported. One of the trio's demands was that OpenAI embed third-party safety auditors. This exercise is a first step in that direction.

What's next

METR's wish list includes checking agents every few minutes instead of waiting for them to finish, isolating monitors from the agents they watch, and regular red-teaming. Future exercises, it says, will come with more public transparency.

Sources: METR, Redwood Research, CNBC and TechCrunch.