Tampered ML: Evaluating LLM Judgment of Machine Learning Research

What it claims

Tampered ML tests whether language models can tell a faithful ML experiment design from a near-identical copy with one injected methodological flaw. The 1,858 leaderboard items come from NeurIPS, ICML and ICLR oral and spotlight papers (2024 to 2026). The headline 77.9% for Opus 5 is both-orders accuracy, where a model gets credit only if it picks the original in both presentation orders. Chance on that metric is 25%; chance on per-order accuracy is 50%. The authors frame the benchmark as a check on whether models can critique, rank or filter experiments before they run, as part of automating alignment research. Code and data are at https://github.com/tamperedml/tamperedml.

How it works

Each item is a research context C, an original proposal O, a tampered proposal T and a flaw description f. The flaw description is never shown to the model under test. Construction runs through LLM-checked gates, and each gate allows one repair or retry before the candidate is discarded:

flowchart LR
  P[Source paper] --> E[Structured fact set]
  E --> G1{Faithfulness gates G1a, G1b, self-containment}
  G1 -- fail twice --> X[Discard]
  G1 -- pass --> O[Original O]
  O --> Pl[Plan flaw f] --> T[Apply edit: T = O + f]
  T --> G{Gates G2-G5}
  G -- fail twice --> X
  G -- pass --> D{Difficulty filter}
  D -- easy for panel --> R[Removed from leaderboard]
  D -- hard --> L[Leaderboard item]

The fact set marks unstated details as "unspecified" rather than inferring them. Faithfulness gates use majority vote across several independent checkers. The four admission gates are:

• G2: T must diverge from the paper, not match another ablation.
• G3: O and T must be experimentally non-equivalent.
• G4: the flaw must be detectable without the source paper.
• G5: no meta-text or artifact cues may reveal which proposal is tampered.

Flaw planning is separate from prose editing so that the change is methodological, not stylistic. The median proposal is about 1,050 words, the median tamper changes 2.0% of the text, and no tamper changes more than 10%.

The difficulty filter removes items that a subset of models already solve in both orders. The authors state this is not a validity filter, so the leaderboard is adversarially selected against the model panel.

Evidence

Fable 5 and Opus 5 lead with about 78 to 79% both-orders accuracy and 86 to 88% per-order accuracy. Fable 5 declines 7.8% of items (125 in both orders, 19 in one). Its plotted bar substitutes Opus 5's answers on those items. On the 1,714 items Fable 5 answers it scores 79.6%, and it scores 73.5% when declines count as errors.

Position sensitivity varies widely across models. Opus 4.6 scores about 66% per-order but only 43% both-orders, a gap of 23 points. Confidence intervals are 95% paper-cluster bootstrap. Twenty-two models are evaluated, including GPT-5.5, Gemini 3.1 Pro, Kimi K2.6, DeepSeek V4, GLM-5.1, Grok 4.3 and Qwen3.

The flaw taxonomy across the 1,858 items is:

Stronger models beat weaker ones in every category. The authors say the picture among frontier models is "less clean."

The human validation used CS/ML graduate and PhD validators on a stratified sample. It gives a post-stratified 86% valid estimate.

Limits

Ground truth only claims that T is less sound than O on the edited dimension. It does not claim that O is a sound experiment.

About 14% of items are estimated invalid. Accuracy differences of a few points between top models therefore sit near label noise.

The difficulty filter makes scores depend on the panel used to filter. Models in that panel may be disadvantaged.

Fable 5's headline number mixes in Opus 5 answers. Use the 79.6% and 73.5% figures for a clean comparison.

The failure modes in model rationales are:

• hallucinated quotes from the paper
• arguments that contradict themselves across the two orders
• treating convention as validity
• treating specificity or disclosure as rigor

The task covers pairwise comparison of short proposals only. It does not test localizing flaws or auditing full papers. The fetched text ends partway through the position-bias analysis in Section 3.1, so later results are not covered here.