DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge

What it claims

DeclineCraft is a 1,000-question benchmark (700 answerable, 300 unanswerable) built from the Minecraft Wiki with a corpus cut-off of 31 March 2025. The unanswerable questions ask about game features documented only after that date, so neither the retrieval corpus nor, ideally, the model's weights contain the answer. It scores three outcomes separately: acknowledged uncertainty (correct abstention), premise denial ("that item does not exist"), and confabulation. Context-only benchmarks such as RefusalBench and UAEval4RAG restrict models to the supplied documents. DeclineCraft lets models use both retrieved and parametric knowledge. The authors report that RefusalBench's context-only prompt makes models decline many answerable DeclineCraft questions.

How it works

The retrieval corpus, Cpre, holds 2,642 wiki pages (11,612 chunks) as they stood on 31 March 2025. Evaluated systems can only retrieve from Cpre. Cpost holds 170 pages (712 chunks) created after the cut-off and scraped on 8 September 2026. Pages that existed before the cut-off are excluded from Cpost even if they were edited later. Cpost is used only to generate questions.

flowchart TD
  S[Scenario from 3 random Cpre chunks] --> T{Question type}
  T -->|answerable| A[Target chunk from Cpre]
  T -->|unanswerable| U[Target chunk from Cpost]
  A --> F[Fact absent from comparison text]
  U --> F
  F --> Q[Question appended to scenario]
  Q --> V[Verifier checks retrieved Cpre chunks]

Qwen3.5 122B-A10B writes every generation step. The target chunk is sampled from the top five Qwen3-Embedding-8B neighbours of the scenario. Gemma 4 31B verifies each question against 10 Cpre chunks retrieved with hybrid search plus reranking. The authors manually reviewed all unanswerable questions. They dropped 21 malformed ones and found none answerable from Cpre.

The scenario prefix is deliberate. For 71.3% of unanswerable questions, the hybrid retriever returns at least one of the scenario's own source chunks. Those chunks are related to the question but contain nothing about the missing fact. Removing the scenario raised CAR for all 10 models tested with single-shot RAG.

Parametric leakage is audited by asking each model the 300 unanswerable questions without retrieval. Answers are scored against references written by an oracle model from Cpost, and published knowledge cut-offs are reported per model.

Evidence

Evaluation covers 27 LLMs under no retrieval, single-shot RAG and iterative RAG, with thinking on and off where the model supports it (40 configurations).

• Single-shot RAG: five chunks from hybrid BM25 plus embedding retrieval. It significantly raises CAR over no retrieval in 38 of 40 configurations.
• Iterative RAG: the model self-assesses sufficiency and may request up to two more rounds. It significantly lowers CAR versus single-shot in 30 of 40 configurations, with a mean CAR of 46.4%. Answer Accuracy barely moves, from 68.5% to 69.3%.
• Other pipeline changes: retrieving ten chunks instead of five also lowers CAR. Adding query rewriting and reranking to hybrid retrieval lowers CAR for all 12 models tested.
• Scale and recency: accuracy correlates with total parameters (Spearman ρ = +0.60). CAR shows no significant association with parameter count or release date. Qwen3.5 and Qwen3-Next often deny that the asked-about item exists.
• Thinking: without retrieval, enabling thinking significantly lowers CAR for 8 of 13 switchable models, by 14 points on average, and raises it for none. The lost abstentions mostly become premise denials. With retrieval, the effect of thinking varies by model.

How to use it

Code and data are at https://github.com/Thakur-Atharva/declinecraft-benchmark. The benchmark is described as dynamic: it can be regenerated with a later corpus cut-off to evaluate newer models. The fetched text gives no install or run commands.

Limits

Temporal separation holds only when the fact also postdates the model's knowledge cut-off. Models trained after mid-2025 may have seen Cpost content, and the leakage audit detects this only after the fact. Some unanswerable questions could be guessed correctly by assuming an older mechanic carries over to a new item, such as copper tools accepting an iron-tool enchantment. These questions are still scored as requiring abstention. A single generator and a single verifier LLM produce and filter all questions. The fetched text cuts off during the assembly section, so per-model tables and appendices were not available.