From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

What it claims

The study tests fourteen LLMs, released between July 2024 and July 2026, on medical exam questions and finds that multiple-choice scores overstate clinical recall. Two findings go beyond the summary. First, the two largest accuracy drops when answer choices are withheld belong to the medically specialised MedGemma models. Second, accuracy holds on questions the strongest models cannot reconstruct, so memorization does not appear to explain their scores. In the standard setting, the strongest models exceed the human-expert baseline on MedQA-USMLE and PubMedQA. Only two of the six smaller models pass that bar on the licensing exam. The authors frame this as the deciding question for hospitals that need small, on-premises models to stay inside a HIPAA boundary.

How it works

Each provider contributes a smaller and a larger model:

• OpenAI: GPT-OSS 20B and GPT-5.6 Luna
• DeepSeek: V4 Flash (284B total, 13B active) and V4 Pro (1.6T, 49B)
• Qwen: Qwen 3.5 9B and Qwen 3.7 Plus
• Meta: Llama 3.1 8B and Llama 4 Maverick
• Mistral: Ministral 8B and Mistral Large 3
• Google: MedGemma 4B and 27B

GPT-5.6 Luna and Qwen 3.7 Plus have undisclosed sizes. GPT-OSS 120B and Qwen 3.5 35B were added so size effects can be checked on models with reported parameter counts.

Twelve models ran through OpenRouter. The MedGemma models ran on a third-party endpoint (dr7.ai), because they were not on OpenRouter at the time. Every model used one identical prompt at temperature 0. Each model was evaluated with its built-in reasoning and with chain-of-thought prompting. Runs took place on 23 and 24 September 2026, and results apply to the checkpoints served on those dates.

The study has three settings:

1. Multiple choice: the standard format, scored against the answer key.
2. Generative: the answer choices are hidden, and a separate judge model grades the free-text answer against the gold answer.
3. Reconstruction: only the choices are shown, the model writes the question, and the judge decides whether it matches the original.

PubMedQA is excluded from settings 2 and 3 because its answers are only yes, no or maybe. Those two settings therefore use 2,576 questions.

Evidence

The evaluation sets are:

• MedQA-USMLE: the full 1,273-question test split.
• PubMedQA: the 500-question expert-annotated test split.
• MedMCQA: 1,000 questions from the validation split, because test labels are only scored through the leaderboard. This is the same sample as an earlier evaluation, for comparability.
• MedBullets: the Step 2/3-style questions posted publicly since April 2022.

Overall accuracy is pooled and weighted by benchmark size. Differences are tested with paired t-tests and χ² tests, with 95% confidence intervals. The paper also reports a cost-versus-accuracy analysis and an ensembling experiment, but the fetched text ends before those results.

How to use it

The authors state that they release several artifacts:

• the question-answering code
• the judge code and its verdicts
• prompt templates for all three settings
• both grading rubrics
• per-request manifests with API model IDs, timestamps, token counts and costs

The fetched text gives no repository URL or licence.

Limits

• The authors call reconstruction a proxy, not a measure of memorization. A model can infer a question from distinctive choices, and a failed reconstruction does not prove the question was unseen.
• The generative and reconstruction scores depend on an LLM judge. The fetched text does not name that judge.
• MedBullets has two different counts. The text says 308 questions, while Table 1 lists 303. Only 303 is consistent with the 3,076 total.
• The text says the judge is "not one of the twelve under evaluation", although fourteen models are evaluated.
• Provider endpoints change without notice, so results may not carry over to later checkpoints.
• The fetched page stops after the methods section. Per-model tables, the cost results and the ensembling results are not included here.