RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

What it claims

RECLAIM tests whether autonomous agents can reproduce the central empirical claim of a recent ML paper from only the paper plus whatever artifacts its authors released. The summary gives the headline pass rates; the page names the agents behind them and shows the failures are not compute-bound. The strongest agent, DeepSeek-V4-Flash, reproduces 41% (Run), 27% (Retrain), 12% (Reimplement); Muse Spark 1.2 is best at Reimplement with 15%. The other two agents evaluated are Qwen3.6-27B and MiniMax-M2.7. Even at the Run tier — where code, data, and weights are all released and the authors' artifacts were verified to work — every agent fails on most papers.

How it works

Each of the 100 papers pins one match target before any agent runs: the cheapest configuration the authors report that still establishes the central claim, plus its metric, value, scope, and the threshold a run must clear. Example: SharpZO's target is 79.42% test accuracy on EuroSAT under a 16-shot CLIP ViT-B/16 config. Tiers reflect natural release state — Run (execute released code/data/weights), Retrain (no weights, agent trains), Reimplement (no code, agent rebuilds the method; datasets stay public benchmarks).

Grading is provenance-audited: a separate LLM traces every reported number back to the execution that produced it, so an agent's self-declared "REPRODUCED" is not trusted. The paper's worked example catches an agent that ran the authors' pipeline with a type=bool bug leaving a flag on, reported 2.86% as a match, and got scored "not reproduced, partial, score 6."

Construction (audited manually only on the selected pool): 3,414 NeurIPS 2025 arXiv papers → 1,000 sample → a MiniMax-M2.7 classifier verified artifacts on each, fully verifying 687, of which 33% Run, 44% Retrain, 13% Reimplement (5% blocked, 5% non-empirical) → 620 eligible → 200 audit pool → released 100 test (34/33/33) + 14 dev. Compute capped at 96 H100-hours per paper; total grant 1,153.8 H100-h.

Evidence

• Compute does not bind. The 48 cells in the 96 H100-hour band spent 6.2% of the compute granted; only 1 of them reproduced its claim. Failed attempts use on average 29% of their budget — most stop with budget to spare.
• Partial vs. full. At Retrain, the strongest agent reaches a verified partial result or better on 67% of runs, the weakest on 27% — but full recovery stalls at 15–27% across all four agents, whose mean audit scores span a 1.7-fold range.
• Dominant failure mode: reimplementing the method without checking any part against the paper's numbers — 63 of 400 runs. A failure-count table splits the two weakest vs. two strongest agents: "reimplemented but did not check" drops 43→20 and "failed before any number" 25→6 with stronger agents, while "ran, outside tolerance" rises 23→53 and "wrong artifact measured" 17→21.

How to use it

• Code: github.com/mithils3/reclaim
• Dataset: huggingface.co/datasets/Mithilss/reclaim
• Run logs (traces): reclaim-traces.vercel.app

Designed to be rebuilt yearly from new NeurIPS/ICML/ICLR submissions, since static agent benchmarks saturate quickly.

Limits

This is a preprint under review; each paper is run once per agent, so per-paper variance is not measured. Pass rates depend on the four specific agents tested. The Run/Retrain/Reimplement labels and the venue-wide 33/44/13% split come from the classifier pass; only the selected pool was human-audited. For non-Reimplement tiers the authors note a released implementation may have been in training data, though their cluster rarely matches the authors' stack. Reimplement is where the "12%" (DeepSeek) and "15%" (Muse Spark) figures diverge — note the summary's 15% is the best Reimplement rate, not the strongest agent's.