An update on independent evaluators

What it claims

Anthropic has embedded six METR staff with "employee-like access" and says METR will publish its own analysis of Anthropic's risk levels, incidents and practices. The post links the program to two prior steps: Anthropic's commitment to stronger third-party assessment and its signing of the White House Accord on Super Intelligence. Besides METR, it names SecureBio and Accenture as current evaluators, Faculty as a partner, and Hugging Face among organizations in early discussions. Anthropic argues every frontier AI company should embed evaluators the same way.

How it works

Anthropic lists four requirements for an evaluator. It needs expertise, including knowing how evaluations fail, such as through alignment faking. It must be independent, with no equity in the company and no financial or operational dependence on it. It needs access to training details and safeguard internals, not just models. It must be free to publish. The post compares the setup to resident bank examiners.

The METR team has company hardware, office space, and internal tool access "roughly on par" with Anthropic's own risk assessment teams. Staff are told to share information as with a colleague working on alignment risk. Three categories are excluded: customer data, attorney-client privileged information, and model architecture details. Other METR employees, including those working with Anthropic on other projects, are siloed from this team's work under enforceable agreements.

The three access tiers differ mainly in who controls publication:

1. Tier 1: narrowly scoped investigations. Anthropic chooses what access to give, and publication requires Anthropic's approval.
2. Tier 2: more information. Evaluators publish without approval, Anthropic can require removal of specific categories, and evaluators can publicly state whether they think the redactions were justified.
3. Tier 3: the METR arrangement. Employee-like access with publication rights similar to tier 2.

SecureBio published a pilot review of an earlier Anthropic risk report. It is now reviewing the chemical and biological weapons chapter of the August Risk Report. Anthropic will consider giving it more access after that review. The Accenture and Faculty work covers red-teaming, alignment assessments and safeguard testing.

Limits

Anthropic alone decides which information falls into the redactable categories, such as reproducing confidential IP. METR's counterweight is a reserved right to make statements about the level of access and information it received. The excluded categories mean outside evaluators do not see model architecture information. Every partner other than METR is described as early-stage, with more limited information and publication rights. The post contains no evaluation results, so no METR findings under this arrangement are available from this page. Anthropic describes the program as voluntary and still changing.