Benchmarking decision models against generative large language models on medical multiple-choice questions

What it claims

The study tests 13 "decision-native" models against 15 generative LLM configurations on all 1,273 English US four-option MedQA test questions, closed-book and text-only. Decision models return a selected label plus a probability for each option instead of generated text. The study is the first multi-model evaluation of this class in a medical setting. Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions). Clef Omni had the shortest median request duration (0.291 seconds). Decision models agreed with their own answers across three runs 92.30% to 100% of the time, while the repeatability of their probabilities varied. The authors' intended use is bounded checks in agentic or LLM-as-a-judge pipelines, where each extra evaluator call adds cost and latency.

How it works

The decision-model class is anchored on TypeSafe's JEV, which its developer describes as using a new architecture, parallel output computation and "Reinforcement Learning for Calibrated Decisions". TypeSafe claims low cost, short response times and "freedom from hallucinations", which here means only guaranteed matching of the output to the schema. To be included, a decision model needed a native selected-label interface ("Decisions" or "System One") that returns four label-indexed option probabilities. The returned label was kept as the answer even when it differed from the largest probability.

All 28 configurations ran through OpenRouter with a fixed serving provider and fallbacks disabled. Generative models had to return a JSON object {"answer": "A"-"D"} with max_completion_tokens=16384. Three matched reasoning-effort pairs varied only effort: GPT-6 Luna Low/High, DeepSeek V4.1 Flash Low/High and Qwen3.8-27B Low/Xhigh. Kimi K2.5 was run with reasoning disabled ("Instant") and with native reasoning.

Retry handling decides what counts as an error:

flowchart TD
  A[Initial attempt] --> B{Parser-valid?}
  B -- yes --> C[Final answer, even if wrong]
  B -- no --> D{Retries left? max 10}
  D -- yes --> A
  D -- no --> E[Exhausted: scored incorrect]

Refusals, truncations, invalid outputs and HTTP failures that persisted after up to 10 retries counted as incorrect. Cost and latency come from the selected final attempt only. Charges for earlier failed attempts are excluded, so real per-question spend for flaky routes is higher than reported.

Evidence

Run 1 gives the primary accuracy, cost and confidence results. Two more runs per decision model measure repeatability, for 68,742 question/configuration/run outcomes in total. Accuracy uses Wilson 95% intervals. The 273 primary pairwise comparisons (195 decision-vs-generative, 78 decision-vs-decision) use exact McNemar tests with Holm correction. Paired bootstrap intervals use 10,000 resamples. Probability quality is measured as AUROC for separating correct from incorrect answers, plus a binary Brier score. Probability vectors had to sum to between 0.98 and 1.02 to count. A sensitivity analysis used a common retained set of 1,165 questions. The fetched text ends before the per-model results tables and calibration results.

Limits

• MedQA was chosen for a controlled comparison, not as a measure of clinical reasoning.
• No images, retrieval, tools or search were used, so image capability was not tested.
• The study was not registered.
• 11 of the 13 decision models were added on 11 October 2026, after early outcomes were seen. The expanded roster, the comparison family and the resource-imputation rules were also set after that access.
• Missing charges and durations were imputed from per-configuration means.
• Only hosted routes were tested; local inference was excluded.
• No peer-reviewed studies of decision models existed according to the authors. The one earlier comparison (JEV 1.13 vs GPT-6 Sol) covered a single model pair.
• Answers given at least 0.90 probability were still wrong 1.71% to 12.77% of the time, depending on the model. Those probabilities are not usable as an error filter without per-model checks.
• Data were collected 6 to 12 October 2026 and are tied to the provider routes active at that time.