How an assessor could investigate control and monitoring of internal AI agents

What it is

METR's proposal (the post is dated 14 October 2026 in its URL) for how an outside assessor could test whether an AI developer's LLM-based monitors would catch serious misbehavior by internal agents. Footnote 1 says it describes the reasoning METR used to evaluate its own internal monitoring system. METR says it will update the five sub-questions as new capabilities and misaligned behaviors emerge.

How it works

The post gives concrete procedures beyond the list of sub-questions.

Coverage. The assessor estimates monitored traffic from inference logs cross-referenced with monitor logs, plus employee interviews to find inference sources nobody logged. Unmonitored inference is then sampled and classified as agentic or not. The post says to test this directly and not assume it is safe because it is "supposed to" have limited tool use. Further checks:

• Are trajectories silently dropped?
• Is multimodal data shown to the monitor?
• Do blocking monitors exist for the most safety-critical areas?

System vulnerabilities. The assessor needs all code on the critical path: the inference and monitor trigger system, internal APIs and serving code, and agent scaffolds. Exploits are tested against the production monitor with its full input and output visible. The post says bugs often sit in pre- and post-processing or in trajectory management such as compaction and memories.

Evasion. Three tests:

• Build attacks from "attack opportunities" found in real transcripts.
• Modify already-flagged incidents until they slip past the monitor.
• Search for severe incidents the monitor did not flag.

Large transcript pools also reveal cases where an agent could have attacked undetected and chose not to. For monitor-agent collusion, the assessor plants honeypots and searches the monitor's chains of thought for talk of collusion.

Human response. The assessor inserts its own attack trajectories into the pipeline, including ones styled like common false positives, and measures review latency and escalation. The post notes this needs the standard precautions of red-team and penetration-test engagements.

Preconditions / Affected

Access the assessor needs:

• Codebase access comparable to the developer's frontier security researchers. Git history is useful for dating coverage gaps.
• Inference and monitor logs, incident reports, cluster access, and inference on the most capable internal models.
• At least about 100k roughly representative transcripts, drawn from employee opt-in plus a similar order of magnitude from evals and training.
• At least about 100 transcripts of the most severe incidents, including ones reported through non-monitor channels.
• For impact analysis, infrastructure code (credentials, sandboxing, network config, devboxes) and limited runtime cluster access. Staff interviews can partly substitute, but the post calls that a meaningful limitation.

Caveats

• Everything rests on trusting the logs. Footnote 3 says the assessor must be convinced agents have not tampered with them, which is what the vulnerability and evasion work is meant to settle.
• Logs need not be complete if they are roughly representative. Compartmentalized work such as pretraining can be excluded if its trajectories and monitoring look similar.
• Transcripts hold sensitive data, so the assessor needs broad advance approval to view them incidentally. Having a company employee sample logs instead adds overhead and reduces independence.
• Removing sensitive code is acceptable but can be hard when dependency chains are complex.
• The post focuses on LLM-based monitoring. It says standard controls such as sandboxing and least privilege may suffice when agents need only a limited, known set of actions.