Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

What it claims

Reflective prompt optimizers such as GEPA optimize whatever binary event fills the rows of their scores matrix. In GEPA, DSPy and APO that event is per-instance correctness, so the search optimizes accuracy. On imbalanced clinical labels this can push AUROC down. Ranking-PE changes the row to "positive scored above negative", so the optimizer targets AUROC with no surrogate loss, no auxiliary loss and no extra rollouts.

On chest X-ray plus EHR diagnosis on MIMIC-IV (Atelectasis, Cardiomegaly, Consolidation), it beats accuracy-based prompt evolution by +5.8 AUROC pp on Qwen3-VL-8B with vision-encoder-tuned SFT and by +16.2 pp on MedGemma-4B. It also exceeds both Accuracy-PE and the base model on Balanced Accuracy.

How it works

• Continuous score at no extra cost. The score is s(x) = log p(Yes) − log p(No), read from the top-k logprobs at the answer token of a normal decoding pass. The scores matrix, the reflector feedback and the final selection all consume this one score. Tokenization and label-fallback details are in Appendix A.
• Stage 1, SFT (conditional). This stage trains LoRA adapters on the language backbone and the full vision-encoder weights. For Qwen, tuning the vision encoder gave higher post-evolution AUROC than frozen-encoder SFT on all three diseases. The stage is skipped for already medically pretrained checkpoints such as MedGemma.
• Stage 2, ranking-aware prompt evolution. This stage is training-free and changes three things: 1. Pair-level scores matrix. There is one row per (positive, negative) validation pair. A cell is 1 if the positive is ranked higher, 0.5 on a tie and 0 if reversed. By the Wilcoxon–Mann–Whitney identity, each column mean equals validation AUROC, so Pareto dominance and parent selection now operate on ranking events. 2. Ranking-shaped reflector feedback. The reflector receives the clinical error type, the confidence magnitude and the example's rank among the other cases. This is the only place the asymmetric cost of misses versus false alarms enters, because AUROC itself treats both error types the same. 3. Final selection. The returned prompt is the one with the highest validation AUROC.

flowchart TD
  A[Base MLLM] --> B{Medically pretrained?}
  B -- no --> C[SFT: LoRA + full vision encoder]
  B -- yes --> D[Skip SFT]
  C --> E[Pair-level Pareto pool]
  D --> E
  E --> F[Reflector with ranking feedback]
  F --> E
  E --> G[Select argmax AUROC]

Evidence

• Headline results: +5.8 AUROC pp on Qwen3-VL-8B + SFT (VE-tuned) and +16.2 pp on MedGemma-4B, both measured against Accuracy-PE. The page does not give the averaging details beyond "across three diseases".
• Atelectasis on Qwen3-VL-8B + SFT: Accuracy-PE stays near the base model. Ranking-PE adds +7.1 AUROC pp and +25.5 Balanced Accuracy pp over Accuracy-PE in the same pipeline.
• Ablation (Table 3): the pair-level matrix alone raises average AUROC over Accuracy-PE, and ranking-shaped feedback adds further gains.
• Probes: a non-medical Gemma3-4B checkpoint and closed-source GPT reflectors are used only for context and zero-shot baselines.
• Other metrics: AUPRC is reported in Appendix Table 10.

Limits

• Visual backbone is a prerequisite. Prompt search did not substitute for a medical-grade visual backbone. To get the gains, you need either vision-encoder-tuned SFT or a medically pretrained model.
• Narrow evaluation. Only one dataset (MIMIC-IV), three binary diagnoses and two open model families were tested.
• Threshold. Balanced Accuracy is reported only at the default decoder threshold. The operating point is left to deployment.
• Matrix size. The scores matrix has |P|·|N| rows, which grows quadratically with the validation set. The visible text does not discuss the cost of this.
• Incomplete page. The fetched text cuts off in the middle of the method section. Full results tables, hyperparameters, dataset sizes, and any mention of code or licence were not visible.