LLM Serving Seen Through the KV Cache

Instead of picking serving techniques one at a time, it is better to pick along four axes: where the KV cache sits, how far it shrinks, who holds it, and how its output is verified.

Optimizations That Converge on One Cache

Prefill-decode disaggregation, KV cache compression, prefix-aware routing, and speculative decoding are usually introduced as techniques at different layers. Yet they act on a single resource. A KV cache management survey from Texas Tech researchers (arXiv:2607.02574) classifies more than thirty serving systems along four axes. Its opening declares that the KV cache has shifted in status from a temporary per-request tensor to a first-order memory object.

That declaration rests on arithmetic. The same survey shows that cumulative KV read traffic over T decode steps grows as O(TS + T²) with respect to prompt length S. As context grows and concurrency rises, the decode phase becomes steadily more sensitive to KV bandwidth and capacity.

The survey itself classifies systems by locality, lifetime, ownership, and substrate. The four axes used in this article differ from that classification. I re-cut them to match the order in which a practitioner picks techniques. The first two are where the cache goes (placement) and how many bytes it occupies (size). The other two are which instance holds it (ownership) and how the output of a reduced cache is guaranteed (verification).

Where the Cache Sits

Prefill is compute-bound and decode is memory-bandwidth-bound. Placed on one GPU, the two phases push each other aside. DistServe (arXiv:2401.09670, OSDI 2024) assigned the two phases to separate GPU pools. It reports serving 7.4x more requests, or meeting a 12.6x tighter SLO, compared with prior systems.

What matters is that disaggregation changes the physical location of the cache. The cache built on a prefill node must move to a decode node. The KV cache therefore turns from a buffer inside GPU memory into an explicit payload crossing network and storage boundaries. KVServe (arXiv:2605.13734, 2026-05) describes this shift as KV becoming the dominant end-to-end bottleneck.

Measurements say how much that move costs. A prefill deflection paper from 2026-07 (arXiv:2607.02043) measured an A100 cluster built from two prefill nodes and two decode nodes. Prefill execution itself accounted for only 2 to 23 percent of P95 TTFT, and queuing plus inter-node KV cache transfer took the rest. The same paper let decode nodes absorb prefill in chunks when they had room, removing the transfer and cutting P95 TTFT by up to 81 percent.

These numbers mean that shaving kernels is a low priority. Look first at where the cache is created and where it goes.

The Bytes the Cache Occupies

There are three ways to reduce those bytes. Change the attention architecture so fewer are produced, encode the tensors that were produced, or discard the less important tokens.

The representative architectural case is MLA. Folding Key and Value into a low-dimensional latent vector before caching also reduces transfer volume in disaggregated serving. Multi-head Latent Attention and KV Cache Compression covers this in detail. An architectural change is fixed at training time, however, so it cannot be applied to a model that is already deployed.

Deployed models get encoding instead. KVTC (arXiv:2511.01815, ICLR 2026) chains PCA-based decorrelation, adaptive quantization, and entropy coding. It reports up to 20x compression, and 40x or higher for specific use cases, while leaving model parameters unchanged and preserving reasoning and long-context accuracy.

For reasoning models the compression target is not the prompt. These models generate a long chain of thought before answering, so the cache grows during decoding. ThinKV (arXiv:2510.01290, ICLR 2026) varies token precision by thought type, cutting the cache below 5 percent of the original while raising throughput up to 5.8x. R-KV (arXiv:2505.24133, 2025) drops redundant tokens and reaches near-original performance with 10 percent of the cache, and 105 percent at 16 percent.

The trap here is that compression ratio is a poor proxy for serving quality. The benchmark by Agrawal and Mayer (arXiv:2607.05399, 2026) compared KIVI, TurboQuant, SnapKV, and CaM on the same models and the same stack. The conclusion was that compression ratio alone does not predict end-to-end performance. KIVI4 held the most stable quality, and SnapKV gave the highest long-context throughput. CaM won large margins on some QA tasks but swung in both quality and realized compression ratio when the workload changed.

The Instance That Holds the Cache

Routing requests that share a system prompt to the same instance reuses the cache and lowers TTFT. But when requests pile onto a popular prefix, only that instance becomes overloaded. Cache affinity and load balancing cannot be satisfied at once by a single routing policy.

The survey weighs this axis heavily. Once workload and hardware are fixed, it concludes, ownership accounts for much of the remaining design variance among distributed systems (arXiv:2607.02574).

Solutions diverge by layer. Mooncake (arXiv:2407.00079, 2024) pools the underutilized CPU, DRAM, and SSD of a GPU cluster into a disaggregated cache. On top of it sits a KVCache-centric scheduler that maximizes overall effective throughput while meeting latency SLOs. It reports up to a 525 percent throughput increase in simulation, and 75 percent more requests handled for Kimi under real workloads.

DualMap (arXiv:2602.06502, 2026-02) rewrites the routing algorithm itself. It maps each request to two candidates through two independent hash functions, then picks whichever looks better under current state. This power of two choices raised effective request capacity by up to 2.25x under the same TTFT SLO.

PEEK (arXiv:2607.02525, 2026) looks inside one instance's pending queue rather than between instances. It maintains an incremental radix tree over waiting requests to expose prefix-sharing groups, then admits each group's pioneer first so siblings inherit the freshly built cache. It improved TTFT by up to 7.9x over SGLang and 7.1x over vLLM, and matched the baselines on workloads with no shared prefix structure.

BanaServe (arXiv:2510.13223, 2025) solves it from the opposite direction. It makes weights movable at layer granularity and the KV cache movable at attention granularity, and it shares a global KV store. The router therefore schedules on load alone, without being tied to cache placement. It reports 1.2x to 3.9x throughput over vLLM. Reading the difference among these four systems as whether ownership is a fixed constraint or a movable variable is my own judgment.

The Price of Using a Cache That Can Be Wrong

The first three axes all give something up. Compression gives up accuracy, routing gives up either locality or balance, and disaggregation accepts transfer cost. The fourth axis is the mechanism that buys those concessions back.

Speculative decoding has a small draft model propose several tokens, and a large target model verifies them in one forward pass. This cuts latency without changing the output distribution. VeriCache (arXiv:2605.17613, 2026-05) applies that structure to the cache rather than the model. It drafts tokens with the compressed cache and verifies them against the full cache. That guarantees output identical to full-cache decoding while using lossy compression, and raises throughput up to 4x.

The failure mode the VeriCache authors point to is the reason this axis exists. Token dropping and quantization lose little accuracy on short outputs, but they diverge further from the original output as decoding runs longer. The result is catastrophic failure in code generation and tool calling. That is why output length has to be considered when choosing a compression ratio.

The verification step is not free either. FASER (arXiv:2604.20503, 2026-04) identifies the scheduling granularity of existing speculative decoding systems as the problem. They set one speculative length for the whole batch and run draft and verify serially. Under low load the draft phase blocks verification and leaves compute idle, and under high load the system wastes work computing tokens that will be rejected.

FASER adjusts speculative length per request and splits verification into chunks that overlap with drafting. It raised throughput by up to 53 percent and reduced latency by up to 1.92x.

Four Axes in Tension

The four axes cannot be optimized separately because a gain on one axis moves as a cost onto another. The table below restates the evidence from the preceding sections as relations between axes.

Two directions show up in this table. One direction hands the cost downward, and the other erases that cost on the axis that received it. Compression reduces transfer volume and helps the placement axis, but hands loss to the verification axis, and VeriCache erases that loss again. Cache-affinity routing pins ownership and creates hotspots on the placement axis, and BanaServe unpins it by making the cache movable, then carries the migration cost.

Where This Lens Does Not Reach

The KV cache does not explain every serving optimization. In MoE models, expert placement and all-to-all communication form a resource problem separate from the cache. The routing skew covered in MoE Load Balancing and Gradient Interference is a matter of expert load rather than cache. The survey also leaves MoE serving on its list of open problems.

The lens also weakens where context is short and batches are small. The O(TS + T²) scaling matters when T and S grow, and for short requests kernel launch overhead and network round trips outweigh cache size. Machinery such as a global cache pool or prefill deflection only adds complexity in that regime. The groundwork on attention itself and on batching is covered in Transformer Architecture and LLM Serving Optimization.

The provenance of the numbers is a limit too. The multipliers in this article were checked against each paper's abstract as of 2026-09. Most of them, however, were measured by the authors on their own prototypes, with baselines and hardware differing across papers. Whether the same multipliers reproduce on other frameworks has not been confirmed outside each paper.

The survey points at the same gap. It audits current evaluation practice and identifies seven missing KV-specific measurements. It then links that gap to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.

Measurement Order

The four axes have an order in which to touch them. The order below is one I assembled from the intervention cost of each axis and the size of the numbers cited above, and it may shift with the workload.

The reason to put item 1 first is the measurement that prefill execution stops at 2 to 23 percent of P95 TTFT. If prefill is under a quarter of tail latency, touching kernels or compression leaves the tail almost unchanged. Item 3 comes later because compression hands cost to the verification axis, and item 4 is needed only for workloads with long outputs and tight accuracy requirements.

Summary

LLM serving papers from 2025 and 2026 look like separate techniques, yet most of them address the placement, size, ownership, and verification of one resource, the KV cache. These four axes are not independent, so a gain taken on one axis moves as a cost onto another. Compression reduces transfer but leaves loss behind, and cache-affinity routing raises hit rate but creates hotspots. So before choosing a technique, measure the share of TTFT taken by queuing and transfer. If that share is large, the fix is to move the cache, not to shave the kernel.