Enhancing RAG-based Medical VQA with Gated Multimodal Representations

What it claims

A Gated Multimodal Unit (GMU) fuses image and question embeddings into one retrieval query for RAG in closed-ended medical VQA. The generator, LLaVA-Med-1.5 7B, stays frozen. On three binary VQA sets (MIMIC-CXR, IU-Xray, Harvard-FairVLMed), the best BiomedCLIP variants beat LLaVA-Med without retrieval by up to 27.7 accuracy points and 26.5 AUROC points on FairVLMed. Against RULE and MMed-RAG the authors call the results "mixed" and "not conclusive". On IU-Xray the method trails both prior methods and plain image-only retrieval.

How it works

Frozen encoders produce image and text embeddings. Two setups are tested: MobileNetV2 with PubMedBERT, or BiomedCLIP. The GMU projects each modality with tanh layers into a shared space of dimension d. A sigmoid gate computed from the concatenated inputs then mixes them per dimension: h = z⊙h_v + (1−z)⊙h_t.

The GMU is trained with InfoNCE under four choices of positive. Baseline uses no extra anchor. F-I uses the matching image as positive, F-Q the question, and F-R the paired report.

The knowledge base is the training-set reports. Each report is encoded with the text encoder and passed through the same trained GMU with a zero visual vector, so queries and contexts share one space. There is one FAISS index per configuration. Retrieval uses cosine similarity, with top-k = 5 by default. The retrieved report text goes into a simple in-context prompt alongside the image and question.

flowchart LR
  I[Test image] --> GMU[Trained GMU]
  Q[Question] --> GMU
  R[Training reports] --> Z[Text encoder + zero visual vector]
  Z --> GMU2[Same GMU]
  GMU2 --> IDX[FAISS index]
  GMU --> S[Cosine top-k]
  IDX --> S
  S --> L[Frozen LLaVA-Med-1.5]
  I --> L
  Q --> L

Evidence

The binary splits are heavily imbalanced (train positives / negatives, then test size):

• MIMIC-CXR: 962 / 6,532, test 3,468
• IU-Xray: 1,868 / 6,022, test 2,569
• FairVLMed (glaucoma): 851 / 6,457, test 4,266

Each result below gives Accuracy / F1 / AUROC for the BiomedCLIP Baseline at k = 5:

• MIMIC-CXR: 82.61 / 86.41 / 88.32. No-RAG LLaVA-Med scores 75.79 / 80.49 / 68.84, and text-only retrieval scores 81.90 / 85.77 / 87.24. MMed-RAG reaches 83.57 / 88.49 / 85.08. The GMU has the best AUROC here.
• FairVLMed: 90.72 / 94.80 / 71.26. These are the best accuracy and F1, against RULE's 87.12 / 92.89 / 77.08. The F-Q variant reaches 89.50 AUROC, against MMed-RAG's 80.81.
• IU-Xray: 76.06 / 65.31 / 85.56. Image-only retrieval scores 86.83 / 74.12 / 91.05, and MMed-RAG scores 89.54 / 80.72 / 87.13.

BiomedCLIP beats the dual encoder in every configuration, by an average of +3.9 to +14.1 points. F-I is the weakest configuration. With BiomedCLIP, the anchored F-R and F-Q variants lift FairVLMed AUROC from 71.26 to about 89, while accuracy is similar.

The ablation over k shows k = 5 is generally best and k = 1 often hurts. For example, Baseline FairVLMed AUROC falls to 57.48 at k = 1.

How to use it

The page lists no code, weights or licence. The required parts are named: BiomedCLIP or MobileNetV2 with PubMedBERT, FAISS, and LLaVA-Med-1.5 7B. The experiments ran on Google Colab Pro with T4, L4 and A100 GPUs.

Limits

• Prior-work numbers are not reproduced. RULE and MMed-RAG figures are retrieval-only results copied from their original papers, not re-run in this setup.
• Narrow task scope. Evaluation covers only closed-ended binary questions on three datasets.
• In-distribution knowledge base. The knowledge base is built from the same datasets' training reports.
• Inconsistent gains, by the authors' own account. They state that further work is needed to identify when improvements are consistent.