NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents

What it claims

NeutronGym grades LLM agents on neutron-instrument design using the McStas ray tracer, with no LLM judge. Its headline training result comes with qualifiers the summary omits:

• Seeds: the 8B model's jump from 11.3% to 76.7% is one seed. A second seed reached 69%. The recipe was repeated on three further gated families at one seed each.
• Classical baseline: a classical optimizer reaches 81% at the agent's simulation budget, but only when it is handed the closed-form physics. The authors say the 4-point gap is not significant at this size. The trained model learned that physics from reward alone.
• Frontier models: they still solve 98–99% of these family instances, so the small model does not overtake them. RLVP, the nearest prior work, reported the opposite outcome.
• Untrained 32B: it is "no stronger than the 8B" on the headline family.
• Imitation fails: supervised fine-tuning on the model's own successes (rejection sampling) degraded it three times.

How it works

Tools. Agents never write .instr files or see a shell. They act through 22 MCP tools wrapping McStasScript, with access to 374 components. Every mutating call is validated on arrival, and a misspelled parameter fails at the call with the nearest valid alternative suggested.

Speed. The standard mcrun wrapper costs about 2.4 s per run in compilation. NeutronGym compiles each family once, keyed on a hash of its comment-stripped source. Each instance is then a parameter assignment run on the binary: 33 ms median at 10^5 rays, about 30 rollouts/s per CPU core.

Reward ladder.

1. L1 checks that only the free parameters are assigned, all finite and inside bounds, and that the geometry passes the family's checks. No simulation is needed.
2. L2 checks that a short run completes.
3. L3 runs the fixed-seed protocol and requires at least 500 events at the graded monitor. - On the two maximization families it also enforces band constraints and a phase-space bound, I ≤ ΦAΩΔλ. A result above 1.10× the bound fails. - The four matching families carry neither check. Every training number is measured on matching families.
4. L4 asks for r > 1. For maximization, r = FOM / (baseline × target fraction). For matching, r = tolerance / worst relative error.

Reward. 0.25 per level through L3, then 0.75 + 0.25·min(r, 2), capped at 1.25. Replacing this with pass/fail costs 60 points.

Targets. Each target is 0.85× the best result of a search (30 random valid samples, then local pattern search) scored at the agent's own seed. Matching instances whose targets the baseline already meets are discarded.

Held-out split. At least one context parameter is drawn from an interval disjoint from training, so held-out instances are new regimes, not fresh seeds.

Admission gate. Before any pass rate counts, a family must survive three no-model probes:

• the best fixed answer (≤20%),
• the best rule that reads the prompt,
• each instance's solution applied to every other instance.

These are decided on a one-sided confidence bound, plus a concentration diagnostic for class-level shortcuts. Four of the authors' own designs failed. Nine reward exploits are documented in §5.1. The probes are released.

Evidence

McStasBench has 16 scored tasks per model and arm, drawn from 12 published instruments in five classes:

• T1 (reproduce): 12 seen, 3 perturbed, 2 never-public held-out.
• T2 (improve): 2 tasks, target set 80% of the way to a classical optimum.
• T3 (open design): unscored.

Results:

• The best model, Claude Sonnet 5, rebuilt 7/16 tasks one-shot and 5/16 through the tool loop.
• Its memorization probe reproduced one reference, and it failed that task in both arms.
• No agent met a T2 improvement target.
• The open-weights Qwen models produced no parseable instrument without tools.
• Protocol: pass@1, temperature 0.

Calibration matters. An earlier version used an undersized baseline, and untrained models passed 80–93% of held-out guide-family instances.

Contamination. In a pilot audit, 4 of 7 agent episodes retrieved the reference file through permitted tools. Benchmark mode now blocks example retrieval and out-of-episode file loads. An automated transcript audit invalidates any episode that touched a reference.

Limits

• A T1 pass is narrower than it sounds. It certifies beam delivery (intensity, width, spectral centroid and width), not class-defining performance such as diffractometer resolution. In practice it is an exact reproduction: 32 of 39 seen-tier passes and all 12 held-out passes match the reference on every observable.
• Topology is never scored. Every scored task fixes the layout. The families vary numeric parameters only.
• The phase-space bound does not cover training. It applies only to the maximization families, which no training number uses.
• Training evidence is thin on seeds. There are two seeds on one family and one seed on each of three others.
• The page text is truncated mid-protocol. Later results, appendices, and any code, licence or install details were not visible, so none are reported here.