CLIMB: Confidence-Guided Complementary Evidence for Multimodal Retrieval-Augmented Generation

What it claims

CLIMB wraps a frozen MLLM and a standard CLIP retriever with two inference-time controls. The first selects a 5-passage evidence pool by MMR. The second is a refinement loop of at most 3 rounds. In each round the same MLLM scores the passages, answers, and rates its own confidence, and a new answer is kept only if that confidence strictly rises. Nothing is retrained. The pool is never re-retrieved, so cost per query is bounded. The authors credit the critic as the main driver of answer quality, and the confidence rule mainly with stability and early stopping.

How it works

1. Candidates. A CLIP-based search returns a candidate set C_N, filtered by a relevance threshold τ. The retriever's similarity score is reused as the relevance score.
2. Micro-pool (K=5). Starting from the most relevant passage, each next passage maximises λ·rel − (1−λ)·max cosine similarity to passages already chosen, with λ=0.6. Embeddings are cached, so this costs O(NK). The pool stays fixed for every later step.
3. R/E/C critic. A constrained prompt to the frozen MLLM scores each passage from 1 to 5 on three axes: - relevance - evidence specificity (named entities, dates, numbers) - cross-modal alignment with the image

The three scores are normalised and combined with equal weights. The top M=⌈√K⌉ passages (3 when K=5) form the first evidence subset.

1. Confidence U ∈ [0,1]. U is the mean of three signals: coverage of the answer's claims by the passages, consistency between passages and image, and answer specificity.
2. Facet refresh. The MLLM names at most two short "missing facet" phrases. Each passage in the pool is rescored with φ' = φ + γ·ψ, where ψ combines BM25 and sentence-embedding matching against those phrases. The top M passages under φ' feed the next answer.

flowchart TD
  A[Pool P, K=5] --> B[R/E/C score, top-M]
  B --> C[Answer + confidence U]
  C --> D{U ≥ U_STAR or T_max reached?}
  D -- yes --> Z[Return last accepted answer]
  D -- no --> E{Missing facets found?}
  E -- none --> Z
  E -- yes --> F[Rescore within P, new top-M]
  F --> G[New answer + U']
  G --> H{U' > U?}
  H -- yes --> D
  H -- no --> Z

The loop runs at most T_max=3 rounds. The value of the stopping threshold U_STAR is not given in the fetched text.

Evidence

The fetched text cuts off partway through the main results table, before any CLIMB rows appear. The baselines it does show, with the metric unnamed in the visible text, are:

The page also says CLIMB:

• beats Top-K retrieval and iterative RAG variants
• keeps performance on questions that need no extra evidence

The ablations credit complementary pooling, critic scoring and the confidence loop each with part of the gain. No numbers for any of these claims are in the fetched text.

Limits

• No CLIMB numbers available. The fetched text ends before CLIMB's own scores. Check the full PDF before relying on the size of the gain.
• Self-assessment. The confidence score and the critic both come from the same frozen MLLM. The authors say U is not a calibrated probability of correctness and the critic is not an independent oracle. "Monotone improvement" therefore means monotone in self-assessed verifiability, not in accuracy.
• No extra retrieval. If the 5-passage pool misses the needed fact, refinement cannot recover it.
• Latency. Each query makes several MLLM calls: the R/E/C judgments for the pool, plus up to 4 rounds of answer, confidence and facet calls.
• Unstated details. The fetched text does not give a code release, licence, the value of γ, or the value of U_STAR.