TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

What it claims

TriageRAG changes how SimRAG-style self-generated QA data is used when adapting a RAG reader to a domain. Each pair is probed closed-book and open-book, triaged by failure type, and routed to a different SFT signal. The authors trace the gain to one ingredient, a "parametric" route. It trains the reader to give the verified answer even when the retrieved context lacks the evidence. SimRAG's round-trip filter discards exactly those examples, so its reader learns to over-trust noisy retrieval. A variant that also fine-tunes the retriever on retrieval failures made both retrieval and accuracy worse, so the main setup keeps the retriever frozen.

How it works

Stage-I fine-tunes the backbone on a 16,000-example scaled-down version of SimRAG's general retrieval-QA blend. Stage-II runs up to T=3 rounds over a synthetic pool built once per domain. The pool has span, multiple-choice (three distractor spans from other passages) and claim questions. A pair is kept only if the student recovers the answer from the gold passage. Round-trip retrieval is recorded, not used as a gate.

Probing draws n=4 samples at temperature 0.7, both with no context and with the top-5 retrieved passages. "Evidence hit" means the gold passage, or the answer text, appears in the retrieved set. The threshold τ is 0.75.

flowchart TD
  P[Probe pair] --> K{closed-book ≥ τ?}
  K -->|yes| KN[KNOWN: 5% consolidation]
  K -->|no| H{evidence retrieved?}
  H -->|no| RF[RETRIEVAL-FAILURE: parametric SFT, 20%]
  H -->|yes| O{open-book ≥ τ?}
  O -->|yes| S[SOLVED: 15% consolidation]
  O -->|no| CF[CONTEXT-FAILURE: contextual SFT, 60%]
  KN & RF & S & CF --> L[LoRA SFT + 2,000 general replay]
  L -->|re-probe pool| P

Quotas are 60/20/15/5% for context-failure, retrieval-failure, solved and known. Context failures were smaller than their quota in every domain, so all were used and the shortfall was filled with SOLVED items. The method uses SFT only, with no reward model, judge or labeled data.

Evidence

All methods share the Stage-I model, the pool, the filter, the budget and the recipe. They differ only in selection and context construction. Headline comparisons use three seeds with paired bootstrap intervals.

• Medicine (Qwen3-4B, six datasets, average accuracy): TriageRAG scores 64.84. Baselines are SimRAG 63.67, Stage-I+RAG 63.96, random selection 64.15, RAFT-style 64.10, backbone+RAG 61.65 and closed-book 52.33. The gain over Stage-I (+0.9) comes from exam-style sets, for example MedQA 63.08 vs 61.12.
• Science and CS: retrieval hurts the base model. Science drops from 87.4 closed-book to 85.3 with RAG, and CS-Bench from 68.0 to 63.6. TriageRAG reaches 87.6 and 68.2. On CS-Bench with Qwen3-8B it reaches 70.1 against 69.8 closed-book.
• Margin over SimRAG: +1.2 (medicine), +1.9 (science), +6.4 (CS, 4B) and +4.4 (CS, 8B).

How to use it

The setup uses Qwen3-4B-Instruct-2507 or Qwen3-8B in non-thinking mode, with bge-base-en-v1.5 on a flat FAISS index at k=5. LoRA settings are r=64, α=128 on all linear layers, one epoch, learning rate 1e-4 with cosine decay, batch 64 and max length 3,072.

Pools were 40,540 / 32,686 / 20,669 tuples for medicine / science / CS. Per-round budgets were 12,000 / 6,000 / 6,000. One medical round at 4B takes about 15 minutes of vLLM probing, 80 minutes of LoRA training on two A100s and 10 minutes of evaluation. The fetched text gives no code link or licence.

Limits

More rounds did not help in medicine: the averages are 64.84 for one round, 64.74 for two and 64.50 for three. The medical margin over random selection is 0.7 points. On MedXpertQA, TriageRAG (13.39) stays below closed-book (14.53).

All benchmarks are multiple-choice or yes/no and scored by accuracy, so open-ended generation is untested. The science and CS corpora are small (10,863 and 6,237 passages). There is no comparison with GPT-4-distilled readers. The fetched text ends after Table 1, so the science and CS tables and the ablation numbers were not visible.