What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

What it claims

Language adds narrow, measurable value to pathology vision-language models (VLMs), and most of the apparent gains elsewhere are not attributable to it. The paper, from the University of Virginia and Merck (arXiv, October 2026), runs four controlled probes on released checkpoints and public TCGA-derived data. It finds that stage-redacted report text improves survival prediction and that zero-shot prompts add nothing over a linear probe. SlideChat's VQA accuracy barely depends on the slide, and paraphrasing changes answers without changing accuracy. It closes with six minimum controls for future evaluations.

How it works

Every probe freezes the representation, fixes the predictor and split, and varies one input. Encoder probes use patient- and tissue-source-site-disjoint five-fold splits. CIs come from 1,000 bootstrap resamples, and an effect is significant when its CI excludes zero. No model is trained or fine-tuned.

• Probe 1 (survival): TCGA-BRCA, -LUAD and -COAD (n = 1,633; 394 PFI and 345 OS events). A frozen PubMedBERT report embedding is concatenated with a frozen TITAN slide embedding, then fed to an L2-penalised linear Cox model. Six arms: vision only, vision plus full report, vision plus stage-redacted report, structured stage and grade, and full or redacted report text alone.
• Probe 2 (subtyping): NSCLC (LUAD vs LUSC, N = 930) and RCC (KIRC/KIRP/KICH, N = 718). The encoders are CONCH v1, CONCH v1.5 and TITAN, against pure-vision Prov-GigaPath. A logistic head is scored by balanced accuracy. Within CONCH v1, zero-shot is compared with a linear probe under the same pooling.
• Probe 3 (grounding): SlideChat, WSI-LLaVA, Qwen2.5-7B-Instruct (SlideChat's backbone) and Qwen3-8B answer 1,227 SlideBench-VQA(TCGA) multiple-choice items. Perturbations: slide removed, features zeroed, mismatched slide, cyclic option shift, and the correct option replaced by "none of the above" (NOTA).
• Probe 4 (wording): SlideChat, Qwen3-VL, InternVL3 and text-only Qwen3 answer a matched SlideBench-VQA subset (TCGA and BCNB). The perturbations are paraphrase banks, reframings (a new framing sentence or persona line), and both together.

Evidence

• Survival: the redacted-report increment was significant in all six cohort-endpoint settings. Both leakage contrasts (full vs redacted report) were null, and structured stage and grade was the weakest arm. The redacted report alone was at least as prognostic as the slide embedding. The gain comes from a frozen text encoder, not a VLM.
• Subtyping: all three language-aligned encoders beat Prov-GigaPath under identical probes, but backbone, data and aggregation also differ. Zero-shot minus linear probe was -0.020 (NSCLC) and -0.019 (RCC) balanced accuracy, not significant. Top-k zero-shot's apparent edge vanished under shared pooling, so it is an instance-selection effect.
• Grounding: SlideChat's text-only and wrong-slide gaps were indistinguishable from zero, including on grading and staging questions. WSI-LLaVA lost a few points, significantly. The most-frequent-letter baseline is 40.7%, and Qwen2.5-7B alone scores 43.52%. That leaves about 31 points from instruction-tuning regularities rather than the slide. Option shift lowered accuracy significantly, and NOTA pushed SlideChat below the letter baseline.
• Wording: paraphrase gave no significant accuracy change for any system. Flip rates were 8-23% under paraphrase and 25% under paraphrase plus reframing, where consistency fell to 0.57. SlideChat changed a quarter of its answers when only the framing sentence varied. One persona line shifted Qwen3-VL accuracy significantly.
• Self-reported claims (not re-run): CPathAgent reaches 88.6% on PathMMU-HR², 15.4 points over single-pass Gemini-2.5-Pro. Gemini prompted to navigate the same way rises from 73.2% to 76.4%, so only part of the margin is navigation. At slide level CPathAgent leads ABMIL by 2.9 points and trails TITAN on CPTAC-Lung (88.1% vs 91.9%). Few-shot MIL language priors (ViLa-MIL, VLEER, GMAT) are graded weakly supported.

Compute: linear heads run on CPU, and inference uses up to two H200 NVL GPUs in bf16. The authors estimate that 7B-15B slide assistants need roughly 10^4 visual tokens and 40-80 GB of GPU memory per WSI. None of the 51 systems in their companion review reports calibrated confidence.

How to use it

The proposed minimum controls are a ready checklist for evaluating or procuring a pathology VLM:

1. Slide VQA: text-only, mismatched-slide, option-shift and NOTA arms, plus the majority-answer baseline.
2. Answer-flip rates and consistency under paraphrase and reframing.
3. Stage-redacted and text-only arms for prognosis claims.
4. Within-encoder comparisons for prompt claims.
5. CIs and seeds for headline results.
6. Compute: GPU type and count, GPU-hours, peak training and inference memory, visual tokens per WSI, and latency on a reference slide.

PathMMU, PathVQA and Quilt-VQA are flagged as contamination-exposed. TCGA slide benchmarks share their source cohort with assistant instruction-tuning data.

Limits

Only TCGA-derived data, two slide assistants, three language-aligned encoders and one vision encoder are probed. Reasoning-tuned, agentic and MIL systems are not tested directly. A null stage-leakage contrast could also mean PubMedBERT ignores stage statements, and redaction leaves stage-correlated findings such as tumour size in place. No survival arm combines vision with structured stage. Zero-shot vs supervised is compared under mean pooling only, without attention-based MIL. Probes 3 and 4 use multiple-choice items, so free-text generation may behave differently.