How an assessor could investigate evidence of acceleration from the automation of AI R&D

What it claims

METR's October 2026 proposal sets out how an outside assessor could test whether AI automation of AI R&D is speeding up frontier capabilities progress. It describes the evidence and access such an assessment would need. It reports no results and has not been run. It is a specification of data requests and analysis questions. The central difficulty it names is the "No AI" counterfactual, which no single line of evidence can settle.

How it works

Attribution runs in two stages. First, the assessor splits any capabilities acceleration into algorithmic progress and data, which AI automation can affect, versus inputs like training compute scaleup, which it affects less. Second, the assessor splits acceleration in algorithms and data into AI assistance versus experiment compute and human labor.

flowchart TD
    A[Capabilities acceleration] --> B[Algorithms and data]
    A --> C[Training compute scaleup]
    B --> D[AI assistance / automation]
    B --> E[Experiment compute]
    B --> F[Human labor]

The data requests are more specific than the summary's list:

• Benchmarks for all frontier models, including internal-only ones, so the assessor can compute ECI and similar aggregates. Each benchmark comes with example questions, scoring details and sample model answers.
• Algorithmic progress estimates, dated precisely and as disaggregated as possible. METR expects developers to measure these as the reduction in compute needed to reach a given performance level. The assessor does not need details of the research advances themselves.
• AI agent spend by team and over time, in tokens and inference cost. This sits alongside crude productivity metrics such as lines of code per researcher-day and self-reported uplift over time.
• Training compute per model, split into pretraining, midtraining and posttraining. Experiment compute is broken down by team and function.
• Data inputs, such as pretraining data or RL environments, plus team sizes over time.
• Existing internal forecasts of top-line metrics, with the reasoning behind them, and evidence on candidate bottlenecks such as training compute and human data.

All data should cover the longest available timeframe with exact observation dates, so changes can be measured against a historical baseline. It should be broken down by team or domain where relevant. Interviews must cover staff in pretraining, RL and infrastructure. They should also include managers and anyone who studies acceleration internally.

On security, METR states that analysis could run on developer hardware, possibly onsite, and that only minimal access to developer infrastructure is needed.

Limits

The page names failure modes in its own evidence sources. Benchmark scores may overstate capabilities because passing tests does not equal real-world usefulness. Researchers' self-reported speedup figures may overstate uplift because the counterfactual is hard to judge. An above-trend compute scaleup can produce above-trend capabilities growth that looks like AI-driven acceleration.

The assessor must also compare internal evidence against public evidence and explain any gaps. They must judge whether these methods would have caught a much larger acceleration if one had occurred. METR says forward-looking estimates are highly uncertain and should be reported as such.

The proposal assumes developers already keep algorithmic progress estimates, but METR notes that no developer has published its method. It does not name a developer that has agreed to this access.