When Is Additional Retrieval Worthwhile? A Cost-Aware Study of Evidence Gain and Answer Reliability in RAG

What it claims

A ridge regression on six features from the baseline retrieval decides per question whether to run a second, hybrid retriever. On a held-out set of 60 QASPER articles, it calls hybrid on 46 of 188 questions and matches always-hybrid Hit@10 (78.70% vs 76.92% for always-keep). The author, Zhouting Zhao, states that paper-level bootstrap intervals do not establish a reliable advantage. Answer correctness is identical to keep-only.

How it works

Both arms search only the evidence units of the question's own article: paragraphs, section headings and figure/table captions.

• Keep embeds with all-MiniLM-L6-v2, takes the top 30 by cosine similarity, reranks them with the ms-marco-MiniLM-L6-v2 cross-encoder, and returns 10.
• Hybrid fuses dense and BM25 rankings with reciprocal rank fusion (k = 60), uses a candidate pool of 30, and applies the same reranker.

The training target is Δ = hit(Hybrid) − hit(Keep), which takes values in {−1, 0, 1}. The six features come from the Keep arm alone:

• top-1 dense similarity
• top-2 dense margin
• mean top-5 dense similarity
• overlap between the dense and reranked top-10
• dense rank of the reranked top-1 (11 if absent)
• fraction of the top 10 that are captions

The model is median imputation, then standardization, then ridge regression with penalty 20. Hybrid runs when predicted Δ ≥ τ = 0.0310. τ was chosen by document-grouped cross-validation to maximize utility, defined as Hit@10 minus 0.02 per hybrid call. The model and threshold were frozen, and file hashes were saved, before held-out gold labels were opened.

flowchart LR
  Q[Question] --> K[Dense + rerank]
  K --> F[6 baseline features]
  F --> R[Ridge predicts gain]
  R -->|gain >= tau| H[Run hybrid BM25+dense]
  R -->|gain < tau| KEEP[Keep baseline top-10]
  H --> G[4-bit Qwen3-4B]
  KEEP --> G

Evidence

Development used 240 articles (690 questions). It had only 16 potential hybrid rescues and 7 harms, so the target is highly imbalanced.

On the held-out set, always-hybrid gives 5 rescues and 2 harms relative to keep. The adaptive policy gets 4 rescues and 1 harm, for the same net gain, while avoiding 142 hybrid calls. Unitized utility is 0.7819 for adaptive, 0.7670 for always-hybrid and 0.7692 for always-keep. The 95% intervals both include zero: [−0.0114, 0.0385] versus keep and [−0.0021, 0.0328] versus hybrid.

Generation used mlx-community/Qwen3-4B-Instruct-2507-4bit with at most 12,000 evidence characters and 256 output tokens. The prompt required citations and the exact abstention string "Insufficient evidence."

Scores are on a 0 to 2 scale. The adaptive-minus-keep correctness difference is 0.000, with interval [−0.029, 0.024]. An earlier judge-only pass had shown a small adaptive gain, which disappeared after partial review.

Limits

• Abstract cost. The 0.02 penalty is a unit cost, not a measured latency, token or monetary figure. The author warns that fewer hybrid calls is not a 75% runtime reduction.
• Data split. Both partitions come from the official QASPER training split. The test split was not used.
• Not a clean test for generation. The generation analysis reuses the held-out articles and is labeled exploratory.
• Judge. Correctness and faithfulness were scored by a Qwen-family judge with partial assistant review and no independent human validation. The bootstrap intervals do not account for judge error.
• Caption proxy. Captions stand in for figures and tables, so numeric answers stored in table bodies may not be retrievable.
• Narrow scope. Retrieval is within a single article, and the policy chooses between only two actions.