OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

What it claims

OpenGray tests whether LLM radiotherapy planning agents make the right submit-or-escalate decision, not only whether their dose is good. With Terra (openai/gpt-5.6-terra), two LLM systems scored 95.4% and 92.1% decision accuracy. They still missed most contradictions that had to be worked out from anatomy. Every system also submitted feasible plans that missed hard goals a stored reference plan had met. The authors (Henry Ford Health, October 2026) recommend scoring agents on decisions as well as dose and giving them an explicit stop-and-request-review action.

How it works

The benchmark uses the 100 head-and-neck cases of OpenKBP-Opt. Each case has a dose-influence matrix A, so the dose of any plan is d = Ax and can be recomputed exactly. The split is 10 cases for development, 30 for evaluation and 60 only for calibrating a plan-quality score. Systems act through a harness with nine planning functions, including set objectives, optimize, report metrics, normalize, and submit or escalate. Each task allows 15 actions and three optimizations of up to 500 iterations.

There are six task types per patient. The overlap contradiction is the hardest: if O voxels are shared between target and mandible, and the goals allow at most m underdosed target voxels and n overdosed mandible voxels, then O > m + n proves no plan can meet both. The feasible control tightens the spinal-cord D0.1cc limit to 33.6–39.4 Gy (24 of 25 tasks), and a stored plan from the same optimizer and budget meets every hard goal.

The four systems differ in how much the LLM decides:

flowchart LR
  T[Task] --> C1[Controller script]
  T --> AC[Arithmetic checks] -->|no conflict| C2[Controller]
  AC -->|conflict| E1[Escalate]
  T --> I[LLM interpreter, one review] -->|pass| C3[Controller]
  I -->|escalate| E2[Escalate]
  T --> F[Full LLM agent, every step]

Both LLM systems ran twice per task with Terra and with Ministral 3 8B, which ran locally on a desktop workstation. A submission on a contradiction got no credit, even with unmet goals listed. Confidence intervals come from 10,000 patient-level bootstrap resamples.

Evidence

• Decision accuracy (Terra): the controller alone scored 60.0%, arithmetic checks 99.2%, the interpreter 95.4% and the full agent 92.1%. Missing-structure tasks were escalated 100% of the time and coverage/cap contradictions 96.6%.
• Overlap contradictions (10 tasks): arithmetic checks escalated 9 of 10, the interpreter 50% (95% CI 25–75%) and the full agent 25% (10–40%). Even where shared voxels exceeded the allowance by at least 20, the full agent missed on all nine. In its 15 misses it used all three optimizations, then submitted a plan with unmet goals without calling the request infeasible.
• Feasible controls: all systems submitted every one, but only 76% met every hard goal. The main failures were PTV70 D99 short by 0.2–9.7 Gy and External max exceeded by 1.3–3.5 Gy. Raising the budget to 2,000 or 5,000 iterations left the same six controls deficient, with D99 changing by under 0.02 Gy. The authors attribute the shortfall to the controller's objective selection. Requiring goal attainment for credit would cut each system's score by 6.4 points.
• Ministral 3 8B: the interpreter scored 31.7% and the full agent 43.5%, escalating 6 and 4 of 20 overlap attempts. The interpreter broke the reply format in 113 of 188 attempts. The full agent lost 26 of 188 attempts to unreadable actions. The two systems wrongly escalated 8 and 7 of 50 feasible controls.
• Physicist review: a physicist agreed with automatic scoring on all 40 sampled scored-task responses.

SAGE planning-log audit

SAGE runs QwQ-32B inside Varian Eclipse. The audit covered 39 SRS runs (31 patients, 21 Gy in one fraction, July to September 2026). Of these, 35 met every goal, 28 after one dose calculation. In the other four, one goal failed on all 10 attempts. The agent named its failing goals correctly in all 88 replies, but it had no stop action.

For Patient 30, the instructions asked for a "not feasible with priorities" flag after four failed attempts, and the flag was never written. The LLM saw only the three most recent attempts and never the attempt number. In attempt 7 it counted three attempts and deferred the flag. In another run, an abbreviated reply sent only one of seven objectives to Eclipse, so no dose was calculated in attempts 9 and 10. Final goal values alone would not show this failure.

Limits

The benchmark covers head-and-neck cases only, with 30 evaluation patients. Only 10 overlap-contradiction tasks and 25 feasible controls could be built and verified. Two LLMs were tested. The SAGE audit is from one institution, and its logged clinical requests have no known correct decision. The fetched text ends partway through the Discussion, so any limitations the authors state later are not shown. Machine-readable results files are listed as arXiv ancillary files.