Retrieving the same document again does not make its cached KV reusable when the preceding context has changed.
The Boundary of Prefix Reuse
Retrieval-Augmented Generation (RAG) supplies retrieved documents to a model. During prefill, a Large Language Model (LLM) processes the input and computes Keys and Values. The KV cache stores these values so generation does not repeat the same computation.
Prefix caching in vLLM reuses the initial token sequence shared with an earlier request. Changing the retrieved documents or their order shortens that shared prefix. Identical documents can therefore yield less reusable computation.
The example below assumes document boundaries align with token block boundaries. S is the system instruction, A, B, and C are document chunks, and Q is the question.
| Previous request | New request | Shared prefix |
|---|---|---|
| S → A → B → Q | S → A → C → Q | S → A |
| S → A → B → Q | S → B → A → Q | S |
The cache key reflects this constraint. According to vLLM's design documentation, each block hash incorporates the preceding block hash as well as its own tokens. A document identifier alone does not establish a cache match.
Position and Context Mismatches
The non-prefix reuse studied in CacheBlend assembles separately precomputed chunk caches for each request. Encoding chunk B alone cannot capture the representation formed by reading chunk A before it. Differences persist particularly in later layers, which consume the preceding layers' attention outputs.
flowchart TD
subgraph FULL["Full prefill"]
direction TB
A["Chunk A"] --> B["Chunk B attending to A"]
B --> F["KV from connected context"]
end
subgraph CACHED["Independent precomputation"]
direction TB
C["Encode B alone"] --> D["KV from context within B"]
D --> E["Place after A in the request<br/>context from A remains missing"]
end
Positions also need alignment. Models using Rotary Position Embedding (RoPE) require cached Keys to be transformed for their new positions. Position alignment and missing cross-chunk attention are separate problems; CacheBlend recomputation addresses context differences.
Moving positions alone therefore does not reconstruct the representations produced by full prefill. If A names a person and B refers to that person with a pronoun, evaluation must also test that connection. Successful position alignment and answer quality require separate checks.
Selective Recomputation
CacheBlend loads the cache and recomputes KV for selected important tokens. Those tokens attend to the current request context while others retain cached values, reducing computation. This does not guarantee that every token representation matches full prefill.
CacheBlend was published at EuroSys 2025, and the conference artifact evaluation confirmed availability and functionality. This does not mean every quality or speedup result was independently reproduced. Here it supports the selective-recomputation design.
Selecting and updating tokens also takes time. Compare quality and complete request latency against full prefill with the same model, questions, and retrieved documents. Reused-token percentage alone establishes neither speed nor accuracy.
Conditions for performance comparisons
Time to first token (TTFT) runs from request submission to the first output token. Prefill time that omits retrieval, queuing, cache loading, and correction differs from TTFT. Matching the following execution boundaries is the starting point for a comparison.
| Item | Comparison condition |
|---|---|
| Baseline | Full prefill and exact prefix caching with the same model |
| Input | Identical questions, documents, and document order |
| Quality | Full-set accuracy and tasks combining evidence across documents |
| Time | Retrieval, queuing, loading, correction, and first output |
| Cost | Precomputation, storage, and cache updates |
When full prefill also fails, errors in both paths can hide differences in aggregate scores. Inspect the full set and a baseline-success subset separately, with explicit denominators. This is a proposed evaluation method, not a theorem or guarantee from CacheBlend.
Measurements before adoption
The following table proposes a service evaluation. Separate cache hits from misses and track quality differences caused by approximate reuse.
| Observation | Decision it informs |
|---|---|
| Chunk recurrence and prefix-hit rates | Whether non-prefix reuse merits an experiment |
| Retrieval, queuing, KV loading, and correction time | Whether the targeted computation is a bottleneck |
| Accuracy across multiple documents | Effects of lost context |
| Rebuilding after source or model changes | Whether precomputation costs are recovered |
Include source, tokenizer, model-weight, and adapter versions in cache-validity conditions. Prevent mixing KV values from incompatible versions. This article does not reproduce CacheBlend serving performance, so deployment requires separate measurement.
Summary
Retrieving the same document and reusing the same KV values require different conditions. Non-prefix reuse must address preceding context as well as position; CacheBlend uses selective recomputation for this purpose. It does not guarantee identical results to full prefill in every case. Evaluate chunk recurrence, complete request latency, and answer quality together before adoption.