AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

What it claims

SEER-Repro is a benchmark of 20 published papers built on the SEER cancer registry, from Stanford, Databricks AI Research and UMass Amherst (arXiv, October 2026). Using GPT-5.6 Sol, a coding agent reproduced 77.4% of 2,190 atomic claims. Another 3.5% were close and 19.2% were not reproduced. A robustness pipeline then flagged major effects in 55 of 579 fully reproduced findings (9.5%). When experts reviewed 10 of those flags, 9 did not hold up as a consequential change under a defensible alternative, and 1 needed further review. The authors read this as high robustness in the SEER papers tested. They contrast it with the sensitivity that Bertran et al. (2026) and Miao et al. (2026) reported for social science datasets.

How it works

Reproduction runs in three steps:

1. The agent extracts findings, leaving out related work, background and methods sections. It splits each finding into atomic claims, one numerical or qualitative target per claim.
2. A coding agent reads the paper, the data and the SEER documentation, then writes and runs analysis code for each finding.
3. A verification agent grades each claim. A numerical claim counts as reproduced at a relative error of 5% or less, and as close above 5% up to 10%. Qualitative claims are only reproduced or not.

Robustness uses three methods. In independent report generation, an agent answers a research question derived from the paper and its conclusions are checked for contradiction. In direct coding-agent analysis, an agent assesses robustness on its own. The structured method is the Perturbation Agent:

flowchart TD
  A[Reproduction code + evidence] --> C[Select up to 20 choices]
  B[Independent reports] --> C
  C --> D[Propose defensible alternatives]
  D --> E[Rerun baseline, then swap one choice]
  E --> F{Outcome}
  F -->|DirectChange| G[Score 1-9, GRADE-style]
  F -->|Qualification / StableUnderTest / Inconclusive / Unavailable / NotTested| H[Recorded, not scored]

Only findings whose atomic claims were all reproduced enter this stage. The reproduction code serves as the baseline. The pipeline selects analytical choices such as cohort definitions, endpoints and statistical models, and ranks them by importance and feasibility. Each alternative changes one choice and holds the rest of the analysis fixed. Only a DirectChange gets a consequentiality score, on a 1-9 scale adapted from the GRADE clinical evidence framework, where 7-9 means critical.

Evidence

Reproduction rates vary widely by paper. Bone sarcoma survival (Hu et al., 2021) reached 99.5% of 188 claims. Breast cancer incidence by age (Jemal et al., 2007) reached 58.2% of 67 claims. Most papers fall between 70% and 93%. The benchmark mixes papers from 1999 to 2021, with citation counts from 23 to 1,628. The 9.5% flag rate counts findings that scored 7-9, not confirmed errors.

Limits

The authors name unavailable data as the main barrier to reproduction, even after extensive recovery efforts. There is no gold standard for robustness, so the expert check covers only 10 of the 55 flags. Prior work cited in the paper shows that changing the analyst persona or the underlying LLM alters the distribution of outcomes. Results here come from a single model, GPT-5.6 Sol. The scope is 20 papers from one registry. The fetched text ends partway through Table 1 and does not state code, data release or licence terms.