How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions

What it claims

RAG-MAMA is a retrieval-augmented question-answering system over 18 breast-lesion management documents. Its hybrid retriever (BM25 plus dense, fused with RRF, then cross-encoder reranked) put accepted evidence in the top three for 17 of 18 queries. Recall@3, R-Precision, MRR@3 and nDCG@3 were each 0.94. The authors find that cross-encoder reranking gave the highest scores of four retriever configurations. Ground truth is scored as evidence spans mapped to every chunk that contains them, so overlapping chunks carrying the same fact are not penalised as misses. Curation grew the spans from 25 to 37 and the accepted chunk targets from 36 to 51, and retrieval rankings did not change.

How it works

flowchart LR
  Q[Query + pathology label] --> G{Scope gate}
  G -->|malignant or desmoid| D[Fixed deferral]
  G -->|in scope| B[BM25]
  G -->|in scope| E[bge-large dense]
  B --> R[RRF k=50]
  E --> R
  R --> X[bge-reranker top 20]
  X --> K[Top 3 chunks]
  K --> L[Qwen 3.6 27B]

• Scope gate. It runs before retrieval and works on the normalised pathology label. Malignant categories and fibromatosis/desmoid tumour get a fixed deferral. Desmoid is gated because its text stays in the index, while sarcoma guidance favours surveillance.
• Preprocessing. Text is extracted with PyMuPDF. Header and footer lines that recur three or more times are dropped. Reference sections are removed when citation density exceeds 25% of lines.
• Chunking. Text units are packed up to 384 tokens using the bge-large tokenizer, with 77 tokens of overlap built from trailing units. This gives 757 chunks.
• Noise filter. A seven-feature noise classifier sorts chunks into five categories. Only 558 of the 757 chunks are eligible for retrieval.
• Dense retrieval. It uses BAAI/bge-large-en-v1.5 in ChromaDB, with a semantic cap of 30 and a fetch multiplier of 2.
• Lexical retrieval. It uses BM25 (rank-bm25) with a biomedical tokenizer. The tokenizer keeps forms such as HER2+, ER-, BI-RADS, decimals, percentages and comparison operators.
• Reranking. Up to 20 fused candidates are re-scored with BAAI/bge-reranker-v2-m3.
• Generation. The generator is qwen/qwen3.6-27b at temperature 0.3 with 2,048 maximum output tokens. Each chunk is truncated to 3,500 characters and passed with its chunk ID, document ID, source file, DOI and rank.

Evidence

The ablation varied one factor at a time. Chunk size ranged from 128 to 1024 tokens at about 20% overlap. Then, at 384 tokens, overlap was set to 0, 38, 58 or 77 tokens. All four retrievers (BM25, dense, hybrid RRF, and hybrid plus reranker) used the same 18 queries and annotations. Per-configuration scores are not in the fetched text; only the selected configuration's 0.94 is given.

Generation was evaluated separately on a 512-token index, not the selected 384-token one. It used DeepEval 4.0.7 claim-level metrics with deepseek-v4-pro as judge (threshold 0.5), plus BLEU and ROUGE across K = 1 to 5. Those results fall outside the fetched text.

Limits

• Benchmark size and origin. The benchmark has one query per document, 18 in total, all Yes/No or WH questions. Codex (gpt-5.6-terra) drafted the queries, reference answers and evidence, and the authors reviewed them.
• Curation and validation. Curation was informed by retrieval output. The authors call for independent validation, including radiologist review of evidence sufficiency. They frame the work as a methodological baseline that does not test multi-evidence questions or clinical effectiveness.
• Corpus scope. The corpus reflects North American practice. The EUSOMA B3 guideline, which defaults to vacuum-assisted excision, was excluded. NCCN treatment guidelines (over 820k characters) were also excluded because they would dominate retrieval.
• Missing from the page. The fetched page gives no code, weights or licence.