Re:CAP: Auditing Retrieval Coverage in Production RAG Pipelines

What it claims

Re:CAP audits retrieval coverage — the gap between what a RAG retriever surfaces and what the corpus contains — without per-query relevance labels, which don't exist for the multi-million-passage, continuously re-indexed corpora in production (JPMorgan Chase authors, enterprise setting). It reframes the intractable problem (enumerate every relevant doc) as a tractable one (probe for positive evidence that something relevant was missed). It's positioned explicitly as an audit layer, not a competing retriever: it runs on top of whatever retriever a deployment already uses, and none of its comparisons claim it retrieves better.

How it works

A six-step loop over an unchanged host pipeline: run the pipeline (S1); extract topics from the answer and reconcile against retrieved docs so only retriever-side gaps count, plus build an entity ledger (S2); generate five gap-questions per iteration (S3); expand retrieval with those (S4); an LLM judge labels each new candidate NEWTOPIC / SUBTOPIC / REDUNDANT / IRRELEVANT / CONTRADICTORY (S5); terminate on convergence, cap, or budget (S6). Gaps are defined at the topic level (paragraph test, type-not-instance rule, cap 50/query), giving built-in dedup and a normalised gap rate.

The generator has one recall lever and four guards, a distinction the paper insists on: the entity ledger (every probe must contain a verbatim anchor) accounts for nearly all the average-recall lift; the four guards — probe-role diversity (a five-role taxonomy), coverage-aware topic serialisation, anti-collapse termination, failure memory — are not recall levers but suppress worst-case failure modes (topic skew, low diversity, cold-start collapse, wasted iterations) that leave-one-out on average recall doesn't surface.

Default config: hybrid BM25+dense, k=10, m=50, 5 gap-Qs/iter, MAX_ITER=3, MAX_TOPICS=50. Reader fixed at GPT-5.2 to remove reader quality as a confound; other components (extractor, judge, generator) default to GPT-4.1, temp 0 (generator temp 0.3).

Evidence

Datasets: MuSiQue (21K corpus, 100q), HotPotQA (5.2M, 98q), MultiHop-RAG (609 articles, 98q), TREC-COVID (171K, 50q), plus MS MARCO (8.8M, BM25-only) as sensitivity ladder.

• MuSiQue: recall 0.901 vs. flat hybrid top-500's 0.772 (+12.9pp) at 238 docs vs. 500; +4.7pp over flat OpenAI dense top-500.
• HotPotQA: +11.5pp over flat MiniLM dense top-500.
• TREC-COVID: 48% of recovered gold absent from flat BM25 top-500; 21.2% still absent from an ensemble of BM25+dense+hybrid top-500 each (a 1,500-doc budget), appearing in 49/50 queries.
• Human validation on the ensemble-missed stratum: 78.9% [71.5–86.2%] of n=123 docs add info beyond the baseline answer (Fleiss κ=0.79); 73.9% on n=180 live production traffic.
• End-to-end recall reproducible to ±1% across three runs.

Confidence intervals are disjoint on the three largest margins (+11.5 to +29.1pp) but overlap on the three smallest (+2.4 to +6.1pp) — a stronger baseline narrows the recall margin, which is why unreachable-gold share, not recall, is the retriever-independent signal.

How to use it

This is a periodic sampling auditor, not an inline monitor: at 250–500 LLM calls per audited query, run it over a representative sample on an audit cycle (typically before promoting an index, embedder, or ranker change). Cost is operator-tunable — an all-mini pipeline is 4.6× cheaper for −0.51pp paired recall, at the edge of the noise floor.

Limits

• On TREC-COVID recall it actually trails flat BM25 top-500 by −2.0pp (0.221 vs. 0.241); the authors include this dataset to delimit where Re:CAP does not apply. Pooled-graded gold (~493 relevant docs/query) makes binary-novelty judging at a 378-doc budget structurally mismatched.
• MultiHop-RAG's 609-article corpus saturates at flat top-500 (0.88–1.00), so it's only a ceiling sanity check, not discriminative.
• Gaps reflect the host retriever's limitations, since the same retriever issues the probes — it surfaces what deeper retrieval on that system could reach via reformulation, not documents fundamentally outside the retriever's reach.
• This is arXiv text (v2, Sep 2026); the fetched page cuts off mid-results, so appendix ablations (guards, cost) are referenced but not shown here.