Configuring RAG Systems: What Breaks, Where Originates, and How to Fix

What it claims

Configuration problems in RAG systems frequently show up in a different pipeline stage from the one that caused them, and that gap makes them more expensive to fix. The study covers 654 closed GitHub issues and 345 linked pull requests from 10 RAG systems. It builds a taxonomy of 5 symptom types, 9 root causes and 6 fix patterns. Findings the authors highlight:

• Most common symptom: "core pipeline fault", meaning a stage fails with an exception.
• Configuration bugs most often come from how the system invokes external libraries or systems.
• User misconfigurations most often come from resource provision that does not meet the system's needs.
• Where things show up: symptoms cluster at initialization, indexing and generation. Root causes mostly sit in early stages.
• Stage mismatch: in about one third of cases, the symptom appears in a different stage from the root cause.
• Cost of distance: the further apart the two stages are (for example, cause at initialization, symptom at generation), the longer real bugs take to resolve, by up to 2.83×.
• Most common fix for configuration bugs is patching logic or algorithms.

How it works

The paper splits the RAG pipeline into six stages and assigns every issue to a symptom stage and a root-cause stage:

1. Initialization: environment, validation, resources
2. Indexing: chunking, embedding
3. Query transformation
4. Retrieval: vector or graph database
5. Post-retrieval processing: re-ranking, compression
6. Generation

It counts two kinds of issue:

• Configuration bugs: the system mishandles a valid configuration.
• Misconfigurations: a user picks incompatible or improper settings.

To qualify, an issue had to cause an observable problem (a crash, a functional error, or worse latency or answer quality) that traces to a user-adjustable option in YAML/TOML, environment variables or API arguments.

The data was collected and coded as follows:

1. Systems were found by searching GitHub, keeping repos with at least 2,000 stars, at least 20 issues and accessible configuration, then taking the top 10 by stars as of 1 Sep 2025.
2. Issues up to 1 Aug 2025 were found with keyword pairs (config terms × RAG component terms), plus bug labels where the repo maintained them. This gave 4,652 candidates, of which 3,258 were closed.
3. Manual filtering for validity, relevance and traceability left 654.
4. Four authors open-coded a stratified 10% sample, then labelled the full set. Agreement was measured with Cohen's κ, targeting κ ≥ 0.7.

Evidence

Issues per system (filtered total / with PR):

The 2.83× figure measures resolution time on real configuration bugs. The paper also cites prior work in which a RAG misconfiguration caused a 10× latency regression and two orders of magnitude higher cost.

How to use it

The practical takeaway for debugging: when a RAG stack fails at generation or indexing, check initialization-time and early-stage configuration before patching the stage that threw the error.

The dataset and code are released at https://github.com/ideas-labo/rag-config-bugs. They can serve as a labelled corpus for configuration-testing or fault-localization tools.

Limits

• Truncated page: the fetched text cuts off in the methodology section. Per-category counts, the per-stage distributions and the nine actionable insights are not in it, so this document cannot report them.
• Sample skew: RAGFlow and DB-GPT together supply 337 of the 654 issues, and 237 of the 345 PR-linked issues used for fix patterns. The fix-pattern results lean heavily on two projects.
• Selection rules shape the sample:
• Only closed issues were included.
• The search was keyword-based.
• Each multi-category issue was given its single "most fundamental" label.
• The 2,000-star cutoff was chosen "pragmatically".
• Status: the paper is a v1 manuscript submitted to ACM TOSEM and has not yet been peer reviewed.