AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

What it claims

AI Consilium scores LLM clinical second opinions on eight separate criteria instead of diagnostic agreement alone: factual grounding, problem prioritization, data use, reasoning coherence, differential diagnosis, final-conclusion quality, management, and safety. It treats model choice, response format (free-form vs. structured clinical reasoning), synthesis stage, deliberation dynamics and inference requirements as separate comparison dimensions on the same cases. The abstract does not name the five criteria shared between primary-role opinions and final reports. It does say that synthesis gave its largest gains over the structured primary-role opinion in problem prioritization, factual grounding and reasoning coherence. Structured format helped most on management.

How it works

The cases are 300 published NEJM cases taken from the NEJM-MedQA diagnostic-reasoning dataset (Savage et al., 2024). The benchmark has two evaluation checkpoints:

1. Role opinions. Every case-specific clinical role writes an independent, standalone second opinion. Each one is scored as a product in its own right.
2. Consilium. All role opinions go to an interactive multidisciplinary consilium. It produces a case-level final report through iterative review and revision until it reaches consensus. The number of rounds is recorded.

flowchart LR
  C[NEJM case] --> R1[Role A opinion]
  C --> R2[Role B opinion]
  C --> R3[Role N opinion]
  R1 & R2 & R3 --> K[Consilium synthesis]
  K --> V{Consensus?}
  V -- no --> K
  V -- yes --> F[Final report]
  R1 -. checkpoint 1 .-> J[LLM Judge]
  F -. checkpoint 2 .-> J

The consilium protocol is fixed as a synthesis stage. The authors say this means benchmark users do not need to reproduce one fixed generation architecture to compare against it. Each pipeline uses a single model. The Judge is calibrated per criterion against physician reference ratings on a fixed 110-case split, and the remaining 190 cases are held out.

Why the framing matters for adopters

The paper's related work argues two things:

• Consensus is not a quality signal. Deliberation can suppress a correct minority view, so reaching consensus should not count as evidence that the report is good.
• Multi-agent gains are confounded by compute. Self-consistency and Self-Refine improve output without any agent interaction, and equal-compute studies show that much smaller advantages for debate remain once budgets are matched.

The benchmark records review rounds and inference requirements for this reason. Before crediting a council design for a gain, check whether its comparison is compute-matched.

Limits

• The fetched text ends partway through related work. Methods, results tables, judge–physician agreement figures, the role list, the identities of the eight models, and any code or data release or licence are not visible. Everything above comes from the abstract and introduction.
• Final reports are scored by an LLM Judge, not by physicians. The authors themselves say physician reference ratings, criterion-specific validation and analysis of judge bias are still required.
• The source cases are published NEJM cases, a curated literature series rather than real-world records.
• The headline comparison is final synthesis vs. the pre-consilium primary-role opinion. From the visible text, it is not shown whether this is compute-matched against self-consistency or self-refinement baselines.