RAG Failures After Retrieval

When a retrieval-augmented generation (RAG) system gives a wrong answer, the correct document is already in the search results in a substantial number of cases. Measure separately whether the system found the correct document, whether it retrieved the evidence inside it, and whether the model read and used that evidence. Only then does the failure location become visible.

This post divides the failure location in RAG into three stages. Document discovery puts the correct document among the candidates. Evidence retrieval finds the passage that holds the answer inside that document. Evidence use applies the received evidence to the answer. Measurement studies from 2026 point to large losses in the last two stages.

Each stage needs a different fix, so this division is necessary. If the failure occurs in evidence use, a new embedding model or new chunking will not correct the answer. The sections below use four studies that measured each stage separately, and derive criteria for which metric to check first when RAG quality is low.

Missing the Evidence Page After Finding the Document

Kobeissi and Langlais measured document discovery and evidence retrieval separately (arXiv:2602.17981, 2026-02). The data is the 150-question open-source subset of FinanceBench. FinanceBench marks the correct document and the correct pages for each question about U.S. Securities and Exchange Commission (SEC) filings.

This post uses two of the paper's metrics, both computed on the top 5 chunks. DocRec@5 is 1 when at least one chunk from the correct document is present. PageRec@5 is the share of correct pages that the top 5 chunks retrieve. Chunks are 1024 tokens long with a 128-token overlap.

BGE-M3 dense retrieval put the correct document among the candidates for 88% of questions, but its average page recall was 0.34. Multi-HyDE generates hypothetical answer passages and searches with their embeddings. With Multi-HyDE and a cross-encoder reranker, DocRec@5 rose to 0.93, but PageRec@5 stopped at 0.46. The gap between the two metrics did not disappear in any configuration.

The most direct evidence is the condition that restricts candidates to chunks inside the correct document. This condition removes document discovery failures, and BGE-M3 still had a PageRec@5 of 0.60. So a loss remains in evidence retrieval even when the correct document is certainly found.

The paper raised PageRec@5 to 0.55 with a page scorer that selects pages first and then searches chunks only inside those pages. On metric questions it reached 0.81, above the 0.68 of the correct-document condition, so evidence retrieval can improve independently of document discovery. But on 14 questions about earnings call transcripts it scored 0.10, below the 0.36 of the strongest baseline.

Retrieving the evidence does not guarantee a correct answer. With candidates restricted to the correct pages, Qwen-2.5-7B-Instruct had a numeric match of 0.34 on 50 metric questions. Numeric match counts a success when any one pair of numbers from the answer and the reference falls within tolerance (absolute 0.03, relative 3%).

How Much Chunking Changes the Results

Śmigielski et al. compared eight chunking methods under the same conditions (arXiv:2606.00881, 2026-05). The data is eight public question answering (QA) datasets plus three merged variants, in which each dataset's documents are joined into one. Retrieval used BGE-M3 and bge-reranker-v2-m3, and GPT-OSS-20B did both answer generation and the 1-to-5 scoring.

The table shows only the four methods that finished on all datasets and also have answer scores. Accuracy@5 and chunking time are means over 11 variants, and the answer score is a mean over the 8 variants that could be scored.

Recursive Semantic, first in mean Accuracy@5, was 1.65 percentage points above fixed size, and its answer score was 0.10 higher on the 5-point scale. Sequential HAC was 7.62 percentage points below fixed size. The paper's overall reading is that advanced methods give marginal gains and that simpler approaches remain highly competitive.

Lumberchunker and DenseX, two methods based on a large language model (LLM), took 8.37 and 15.05 hours on average. Both exceeded the 48-hour limit on several datasets. Their averages use only the datasets on which they finished, and a recalculation from the per-dataset values confirms this. So these averages cannot be compared with the table above. For the same reason, the paper asks for caution with Lumberchunker's top answer score.

The part of this paper most relevant to this post's claim is the metric definition. A chunk counts as relevant when it overlaps the answer span. When no extractive answer is available, every chunk of the correct document counts as relevant. In that case, Accuracy@5, which counts relevant chunks in the top k, measures something closer to document discovery than to evidence retrieval.

The LiteraryQA results fit this case. For fixed-size chunking, Accuracy@5 was 85.91%, but Recall@10 was 4.53% and the answer score was 1.93. The paper does not state which definition applies to which question, and it only says that the two metrics follow the PIRB benchmark. So this reading is an estimate.

Models That Receive Evidence but Do Not Read It

Retrieved but Not Read, by Walters, Chen, and Toni, measured the evidence-use stage directly (arXiv:2610.21731, 2026-10). The test set has 225 facts whose values changed after the models' training cutoffs. The facts cover six domains: U.S. Food and Drug Administration (FDA) drug labels, finance, sports, people, SEC filings, and chemical classification. Each fact has an outdated value and a current value, so outdated answers can be identified.

The key device is frozen retrieval. For 223 of the 225 facts, every model receives the same 10-result list, built once with Brave Search, and every query returns that same list. If the correct page was not in the top 10, it was inserted into the list. Failures in this condition therefore do not come from document discovery.

The same list was used to evaluate GPT-5.6 Luna and six open-weight model configurations. The share of episodes answered from search-result snippets alone, with no page opened, ranged from 2.6% for gpt-oss-120b to 76.0% for qwen3-8b. In the deployed condition, Luna ran with its own web search. It searched in all 1,747 episodes, but it opened no page in 53.6% of them.

The table pools the seven frozen-retrieval configurations and all prompt types. A citation is unfaithful when the cited page contradicts the value that the answer asserts. Snippet-only answers had the highest rate on both metrics, and the paper reports this relation as an association.

Answers that cited an opened page still had unfaithful citations in 2.3% (29/1258) of cases, so opening a page alone does not guarantee evidence use. The difference from the 1.6% of answers that opened a page without citing it has no significance test, so this post does not interpret it. Search itself helped: enabling search raised claim-check accuracy by 27 to 42 percentage points, depending on the configuration.

The study also measured which values the snippets carry. Of the 47 sports facts, 70% had the outdated value in a snippet and 72% had the current value. When both values appear, a model that opens no page may not find in the snippets what it needs to decide which value is current.

Failures That Remain with the Page in the Prompt

The authors changed how evidence was delivered for 50 sports transfer claims. These 50 include 3 facts that the paper later withdrew after reviewing their ground truth. Each claim names the former club, so the correct verdict is always "inaccurate". The table below gives endorsements of the outdated claim divided by graded answers.

In this experiment, gemma-4-31b never opened a page, and with the same page in the prompt its endorsements fell from 14 to 0. Taken together, these results indicate that this model's failures came from the decision not to open the page. qwen3-8b still had 8 failures with the page in the prompt, so it failed while processing evidence that it had already received.

The authors also examined the remaining failures inside the models. They used sports transfer cases in three open models. With the logit lens and the Jacobian lens, they read the scores of the two club names at each layer. In failed cases, the page still raised the score of the new club, but it did not lower the score of the outdated club enough. In gemma-4-31b, it did not lower it at all.

This analysis is correlational, and the paper states that it did not run causal tests such as activation patching. Adding a request for sources gave no clear improvement. For qwen3-8b with the page in the prompt, endorsements rose from 8/49 to 12/49.

One mitigation showed an effect: restating the original question after each opened page. In a test that asks the model to find one outdated line among true lines, this recovered 41 of 157 failed checks (26.1%). Recovery rates ranged from 4.5% to 57.1% across configurations.

Attributing Failures to Stages

TREK, by Hu, Singh, Tandon, and Arora, proposed TrekTrace, which assigns each failure to a stage automatically (arXiv:2610.25323, 2026-10). TrekTrace first checks whether the gold evidence that holds the answer is in the search results. It then separates post-retrieval failures with two re-runs: one with the gold evidence only, and one with the actual search results plus the gold evidence.

If the model fails with the gold evidence only, the failure is an interpretation failure. If it succeeds with the gold evidence only but fails when other results are mixed in, the failure is a distraction failure. One case can receive both a pre-retrieval or retrieval label and a post-retrieval label. On 94 human-reviewed system-related errors, TrekTrace agreed with human judgment at a Cohen's κ of 0.930.

In TREK, pre-retrieval and retrieval failures both mean that the answer-bearing evidence is absent from the results. They differ in whether the query targeted that evidence. Together they correspond to document discovery and evidence retrieval in this post. On 144 real questions about rapidly changing knowledge (TrekBench-Wild), the results leaned toward the post-retrieval stages.

For GPT-5.4 with search and high reasoning effort, 88.9% of the reviewed failures included a post-retrieval failure. The share that included a pre-retrieval or retrieval failure was 27.8%. For Gemini-3.1-Pro, the values were 100.0% and 23.1%.

The controlled experiment (TrekBench-Ctrl) measured the loss while changing the type of distractor documents. GPT-5.4-mini was correct 70.8% of the time with gold evidence only. With only outdated documents mixed in, it reached 64.6%. With irrelevant, republished, and fabricated documents added and source URLs hidden, it reached 34.0%.

A baseline that reranked documents by semantic and temporal scores reached 62.5% and 33.3% in the same two conditions, no better than answering without reranking. The shift of the reranker training objective toward generator usefulness is covered in Reranker Objective Shift. Even with gold evidence only, about 29% of answers were wrong, and reranking, which changes candidate order, is unlikely to reduce this loss. That last point is a judgment.

Conditions Under Which This Claim Does Not Hold

With a weak retriever, the bottleneck is document discovery. In the FinanceBench experiment, BM25 had a DocRec@5 of 0.32, so document discovery failure was the largest in that configuration. In TREK as well, 27.8% of the reviewed GPT-5.4 failures included a pre-retrieval or retrieval failure.

Both studies that measured evidence-use failures directly deal with rapidly changing facts. In that condition, the outdated value stays in the model's prior knowledge and in snippets, and it competes with the new value. In internal-document RAG where values rarely change, this competition can be weaker, and the evidence-use failure rate can be lower. This is an estimate.

On FinanceBench, which uses static SEC filings, the numeric match in the correct-page condition was also 0.34. But this value does not separate calculation errors from evidence-use failures, so it does not show how large the evidence-use loss is for static documents.

The samples and conditions are also limited. The only proprietary model in Retrieved but Not Read is the lightest tier of GPT-5.6, and the severe failures concentrate in the sports domain. In a pre-registered check without search, the answer-line instruction lowered Luna's false endorsement by 17.5 percentage points. On this basis, the paper states that the deployed failure rates are likely lower bounds.

In the FinanceBench experiment, 74.7% of the 150 questions concern 10-K filings, and the generator is a single 7B model. The chunking comparison used the same GPT-OSS-20B for answer generation and scoring. The TREK attribution results rest on 144 English questions and the failure reviews of two models.

Some questions the sources leave open. No study assigned each failure to one of the three stages in this post. So the share of loss per stage can vary with the data and the model.

Mitigations for evidence-use failure also lack verification. Re-asking recovered 26.1% of the failed checks. In TREK, a filtering baseline uses a reasoning model to filter documents. With source URLs hidden, it raised GPT-5.4-mini from 34.0% to 43.1%, but it lowered GLM-4.7-Flash from 28.6% to 18.9%.

Criteria for Choosing the First Metric

Interpret the results in this order: document discovery, evidence retrieval, evidence use. One evaluation run can record all three metrics, so measure them together and interpret them in order. Calculate evidence recall only on queries where the correct document was found, so that it does not mix with document discovery failures. The FinanceBench values above are averages over all 150 questions, so they differ from values calculated this way.

flowchart TD
  Q["Evaluation query<br/>labels for correct document and evidence location"] --> D{"Is the correct document<br/>in the top k?"}
  D -- "No" --> F1["Document discovery failure<br/>retriever, query rewriting"]
  D -- "Yes" --> E{"Is the evidence passage<br/>in the top k?"}
  E -- "No" --> F2["Evidence retrieval failure<br/>page- or passage-level search"]
  E -- "Yes" --> G{"Correct with<br/>gold evidence only?"}
  G -- "No" --> F3["Interpretation failure<br/>generator, evidence format"]
  G -- "Yes" --> N{"Correct with the<br/>actual candidates too?"}
  N -- "No" --> F4["Distraction or reading failure<br/>check page opens, re-ask"]
  N -- "Yes" --> OK["Pass"]

Make the decision by answering these questions in order.

• Is the retrieval metric document-level or evidence-level? If the definition counts every chunk of the correct document as relevant, it measures only document discovery.
• Is DocRec@k high while answers are wrong? Check evidence recall. If the gap is large, page- or passage-level search is the candidate. Methods that compute document structure at index time are covered in RAG Moves From Retriever to Compiler.
• Is evidence recall high while answers are wrong? Measure accuracy with gold evidence only. If that value is also low, a new retriever will not correct the answer, so change the generator or the evidence format.
• Is the system correct with gold evidence only but wrong in the actual pipeline? Examine distractor documents and reading behavior. For an agent that uses a search tool, record the share of opened pages and whether each cited page supports the answer.

If DocRec@k itself is low, fix the first-stage retriever at that point. First-stage retriever design is covered in DPR-Based Dense Retrieval for RAG. The basic structure of second-stage reranking is in RAG Cross-Encoder Reranking.

References

• Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering, arXiv:2602.17981 (2026-02)
• Chunking Methods on Retrieval-Augmented Generation - Effectiveness Evaluation Against Computational Cost and Limitations, arXiv:2606.00881 (2026-05)
• Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence, arXiv:2610.21731 (2026-10)
• Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge, arXiv:2610.25323 (2026-10)

Every figure was checked against each original source as of 2026-10.

Summary

RAG failures can be divided into three stages: document discovery, evidence retrieval, and evidence use. Measurement studies from 2026 report large losses in the last two stages. On FinanceBench, BGE-M3 had a page recall of 0.60 even with candidates restricted to the correct document. In the frozen-retrieval experiment, snippet-only answers gave outdated values more often. TREK found that most reviewed failures of two frontier models included a post-retrieval failure.

These results come from rapidly changing facts, financial filings, and specific model generations, and with a weak retriever, document discovery remains the bottleneck. So when RAG quality is low, measure DocRec@k, evidence recall, and accuracy with gold evidence only before you change the retriever. The differences among the three values show which stage to fix.