RAG Moves From Retriever to Compiler

What you precompute while building the index sets the ceiling on RAG quality more than which retriever you attach at query time.

Most RAG improvements cluster at query time. Teams add a reranker, blend sparse and dense scores, and rewrite the query. Yet recent research keeps producing cases where the ceiling on improvement was already fixed when the index was built.

This article makes one claim. The center of gravity in RAG design is shifting from query-time retrieval to index-time precomputation. The wall that no retriever swap can climb is built into the index structure.

Basic retrieval structure is covered in RAG and DPR Dense Retrieval. This article looks only at the design decisions above that layer. The arXiv identifiers and figures below were confirmed against each paper's abstract as of 2026-09.

The Division of Labor Between Index Time and Query Time

Separating what each phase owns makes the argument concrete. Index time can read the whole corpus repeatedly and works under a loose time budget. Query time keeps a user waiting, so it is squeezed into milliseconds.

The direction of the arrow is what matters. Every query-time stage moves inside the candidate space the index handed it. Information the index dropped cannot be recovered by any reranker, however good it is.

Repeated-Call Costs No Retriever Reduces

While an agent runs dozens of loop steps, each step re-embeds the query and injects raw text fragments into the context again. This repeated overhead accumulates independently of retrieval quality. The name compile-time RAG, used in Pinecone-adjacent discussion, is a coinage with no academic definition, so the term itself is safer treated as a signal rather than a result.

WiCER studies the same idea in a verifiable form (arXiv:2605.07068). The LLM Wiki pattern compiles source documents into one persistent artifact in advance and serves it through a KV cache. WiCER measured that pattern across 17 RepLiQA domains and 6,800 questions. On curated knowledge the full-context KV cache approach scored 4.38 out of 5 against 4.08 for RAG.

The compilation step itself is the problem. A wiki compiled blindly in one pass fell to between 2.14 and 2.32, with a catastrophic failure rate in the 53 to 60 percent range. WiCER proposed an iterative procedure that uses diagnostic probes to find dropped facts and force them into the next compilation. One or two iterations recovered 80 percent of the lost quality.

This result adds two things to the argument. Precomputation does pay, and precomputation without a verification loop produces an artifact worse than the source. Moving work to index time does not remove the cost, it relocates it.

Two Ways to Build Structure in Advance

PageIndex uses a document's own hierarchy as the index. An LLM reads the PDF, reconstructs the table of contents, and builds a tree whose nodes carry page ranges and summaries. When a query arrives, an agent descends from root to leaf and chooses which nodes to open.

The performance claim needs care. The vendor reports 98.7 percent accuracy on FinanceBench in its own announcement. That is a self-measurement, and the grading is done by an LLM judge.

The original FinanceBench paper had humans grade a 150-case sample (arXiv:2311.11944). Under those conditions even GPT-4-Turbo with a retrieval system answered 81 percent of questions incorrectly or refused them. The grading axes differ, so the two numbers cannot be placed side by side.

Strip the numbers away and the design idea stands. The substance is not discarding embeddings but using the document's native hierarchy as the index structure and keeping page numbers in citations. The official documentation states that the hybrid tree search mode initializes its value function with vector search, which points the same way.

A knowledge graph does the same job across documents. Extracting entities, merging surface variants into canonical nodes, and attaching provenance to relation edges all finish at index time. What remains at query time is cutting a k-hop subgraph from a seed entity and serializing it. The basic behavior of GraphRAG is covered in the GraphRAG section of RAG.

Filter-Blind Indexes Beyond Query-Time Repair

Filtered vector search supports the claim most directly. When a metadata predicate and vector similarity must both hold, the induced subgraph of predicate-satisfying nodes loses the connectivity of the original graph. Once an intermediate node linking two valid nodes drops out, the path breaks and the search stalls in a local minimum.

ACORN does not build a separate index per predicate (arXiv:2403.04871, SIGMOD 2024). It emulates predicate subgraph traversal on top of HNSW instead. The paper reports 2x to 1,000x higher throughput than prior methods at a fixed recall. The key point is that it changed the query-time traversal rule and the build-time neighbor selection rule together.

For numeric range predicates, RNSG goes further (arXiv:2603.12913, 2026-03). It defines a range-aware relative neighborhood graph that considers spatial and attribute proximity jointly. The paper proves that any range-induced subgraph remains a valid graph of the same kind. This structural heredity removes the need to build a separate graph per range.

Both papers say the same thing. A filter is query-time information, but the ability to withstand a filter comes from index-time edge rules. The complexity of graph traversal itself is covered separately in Complexity of Graph-Based Approximate Nearest Neighbor Search.

Embedding Choice Evaluated After Reranking

The most common index-time decision is which embedding model to use. Making that choice from a standalone MTEB score can put it out of step with the reranking stage that follows.

An experiment by Rao and colleagues shows this (arXiv:2506.00049, 2025-05). In hybrid retrieval fusing dense, sparse, and graph signals, BGE-Large led before reranking with nDCG@10 of 0.6608 against 0.6505 for MiniLM-v6. Adding GPT-4o reranking reversed the order, and on SciFact MiniLM-v6 reached 0.6681 while BGE-Large fell to 0.6170.

The paper attributes the win of a model with 93 percent fewer parameters to alignment between the embedding space and how the LLM judges relevance. The decision belongs to index time, but the unit of evaluation has to be the whole pipeline.

SPLATE shows the same logic on the cost side (arXiv:2404.13950, SIGIR 2024). A lightweight MLM adapter maps the frozen token embeddings of ColBERTv2 into a sparse vocabulary space in advance. A standard inverted index can then handle candidate generation.

The paper reports matching PLAID ColBERTv2 by reranking 50 documents retrieved in under 10 ms. The latency budget was bought with index-time adapter training. A comparison of reranker families is in Late Interaction Reranking.

What Does Not Move to Index Time

There is a range where the claim does not hold. Knowledge conflict is the largest part of it. When a retrieved document contradicts the model's parametric knowledge, which one to trust is decided only after the candidate set exists.

Chen and colleagues measured the weakness of standard RAG (arXiv:2605.14473). Under a worst-case TruthfulQA probe with misleading retrieval injected, standard RAG accuracy reached only 15.0 percent. An intervention that elicits contextual and prior answers separately raised Entity Swap results from 79.3 to 88.0 percent, but the gain varied by model.

ConflictRAG is a pipeline that filters conflicts before generation (arXiv:2605.17301). A lightweight classifier detects first and an LLM reconfirms only ambiguous cases, cutting API cost by 62 percent. The paper reports that detection F1 held at 88.7 percent under that arrangement.

Both papers intervene at query time. My judgment is that index time can go only as far as attaching the metadata, validity window, and schema version that a source credibility score would rest on. The sources do not offer an index-time technique for resolving conflicts.

The remaining limits share that character. A precomputed artifact is a snapshot and goes quietly stale without a recompilation schedule. Access permissions that must be enforced per query cannot be baked into an index. Candidate reduction rules for a knowledge graph are domain specific, so that part is rebuilt and evaluated each time even when the extraction prompt is general.

Deciding What to Move First

The decision starts from the nature of the corpus. The table below orders index-time investments by priority.

Three checks are enough. First, look at whether per-query cost is growing faster than user count. Second, confirm whether the current index knows the filter conditions attached to queries. Third, confirm that a precomputed artifact has a regeneration cycle and a verification set.

Summary

More room for RAG improvement remains at index time than at query time. Connectivity under filtered search comes from edge rules, and document structure and entity relations cannot be restored once index time passes. The ranking of embedding models only becomes visible with reranking included.

Judgments that require a candidate set, such as conflict adjudication and access control, stay at query time. Precomputation pays only when a verification loop and a regeneration schedule are designed alongside it.

The criterion fits in one line. Move work to index time when the corpus is stable, query shapes repeat, and filters are always attached. Leave it at query time when the data keeps changing or permissions differ per query.