How UK AISI and EvalEval Are Making Benchmark Results Reproducible

What it claims

AISI's verified results for five benchmarks and six models are published as Evaluation Cards in the Every Eval Ever (EEE) schema. Each card holds the score together with its run configuration and transcript-level data. The point is that two scores that look similar can be checked for whether they were produced under different conditions. The post shows two uses of this data:

• Humanity's Last Exam. Performance depends on evaluation protocol and inference compute. When models received correctness feedback from an oracle after each attempt, they kept solving additional tasks as token use grew.
• Terminal-Bench 2.0. AISI's results are plotted next to other published results for the same models, which were run under different evaluation setups. This is the "verified reference point" use case: comparing third-party claims against a run with a known configuration.

How it works

• Evaluation Cards combine three things into one record: benchmark metadata, evaluation-run data and model metadata.
• EEE is the shared schema and repository that evaluators report into. AISI's feedback, from collaboration that began at a joint workshop alongside NeurIPS 2025, shaped the schema.
• The HLE curves plot the cumulative share of attempted tasks solved within a given token count. Each task counts at its earliest observed success. This means the measure credits a model at its first success, so it rises with more attempts and more tokens rather than reflecting a fixed single-attempt budget.
• Related AISI work named alongside this release:
• OptStop, for more efficient evaluation
• HiBayES, for statistically rigorous analysis
• standardisation work on transcript analysis and capability elicitation

How to use it

The page gives these entry points but no URLs, commands or API details:

• Researchers can browse Evaluation Cards by benchmark or by model.
• Evaluation developers are invited to submit benchmarks and run data in the EEE schema.
• Model developers are invited to report verified results.

Limits

• The two cyber evaluations (Cyber CTFs and The Last Ones) use a different, only partially overlapping set of models. They are not directly comparable with the five-benchmark grid.
• AISI shares data "where appropriate". Coverage is selective, not every AISI evaluation.
• The post gives no scores, token budgets or effect sizes. Those are in the paper, How Inference Compute Shapes Frontier LLM Evaluation, and in the cards themselves.
• The post states no licence or terms of use for the released data.