PHYRD: A Living Benchmark for Reproducing and Extending Theoretical Physics Research

What it claims

PHYRD (arXiv 2610.32575v1, October 2026, EPIC Lab, Shanghai Jiao Tong University) scores five frontier models on 106 theoretical-physics problems. The problems split into 504 subquestions: 279 in Reproduce and 225 in Discover. The headline result is the gap between partial and complete solutions. On Reproduce, subquestion accuracy (SAcc) runs from 79.66% to 88.33%, but whole-problem accuracy (WAcc) runs only from 49.21% to 68.25%. On Discover, tool access mostly cuts calculation errors, while conceptual errors stay at a similar level.

How it works

FormulaForest turns each source paper into a relation graph. Nodes are formulas and premises. Edges are derivation or specialization relations with their applicability conditions, and each relation records where it appears in the source. Explicit results are kept separate from inferred dependencies.

flowchart TD
  A[Source paper] --> B[Relation graph]
  B --> C{Track}
  C -->|Reproduce| D[Published target plus premises]
  C -->|Discover| E[Transform operators, couplings, symmetries or boundaries]
  E --> F[Check modified model is well defined]
  D --> G[Verify derivations, blind solve]
  F --> G

Discover statements say which premises change and which stay fixed. The solver has to work out which source arguments still hold. Each subquestion is scored as correct or incorrect, and equivalent derivations are accepted. Reproduce is closed book. Discover is run both closed book and with symbolic-computation and retrieval tools, on the same 43 problems.

Evidence

Reproduce results (SAcc / WAcc, complete solutions out of 63):

• GPT-5.6 Sol: 88.33 / 68.25, 43 complete
• DeepSeek v4 Pro: 86.88 / 65.08, 41 complete
• GLM-5.2: 82.25 / 53.97, 34 complete
• Qwen3.7-Max: 84.30 / 52.38, 33 complete
• Kimi K2.6: 79.66 / 49.21, 31 complete

There were 315 responses in total. Of the 133 incomplete responses, 78 missed exactly one subquestion. Qwen had 21 one-part misses, compared with 13 for GPT. Ten problems were solved by all five models, six by none, and ten by exactly one model.

Discover WAcc, closed book compared with tools:

• GPT: 74.4 to 79.1
• DeepSeek: 55.8 to 76.7
• Qwen: 18.6 to 39.5
• Kimi: 23.3 to 37.2
• GLM: 32.6 to 30.2

The SAcc gains for DeepSeek, Kimi and Qwen are 8.96, 11.02 and 14.30 points. Their paired 95% bootstrap intervals all exclude zero.

GLM's SAcc rise (59.3% to 69.4%) comes from answer coverage. Closed book, it gave no final answer on six problems. On the 37 problems it answered in both settings, its SAcc was flat (68.96% to 68.85%), with an interval of [-10.52, 10.83].

A separate anonymous study used SciDeriv, a training corpus built by the Reproduce pipeline, with an ESCD cross-tokenizer distillation objective. That study is the source of the FrontierScience gain from 61% to 72%. The fetched text stops before Section 5.5, so the setup behind that number is not shown.

Limits

• Uneven subfields. HEP makes up 39 of 63 Reproduce problems and 20 of 43 Discover problems, so it dominates aggregate scores. Quantum and Stat have three Reproduce problems each, where one problem moves accuracy by 33.3 points. The authors say these subsets do not support domain comparisons.
• More than tool access differs. The authors note that the closed-book and tool-assisted settings also differ in interaction and computation. Gains cannot be attributed to tools alone.
• Aggregate rank hides per-task differences. Rankings change by subfield. GLM and Kimi each solve 8 of 10 Astro problems, and DeepSeek beats GPT on HEP (27 against 25).
• Missing adoption details. The fetched text gives no grader identity, data licence, release location or access instructions. Model settings and scoring rules are deferred to Appendix B, which was not fetched.
• Planned updates. Future versions are planned as timestamped additions that keep prior task and score versions. No release schedule is given.