Multi-head Latent Attention and KV Cache Compression

The road to a smaller KV cache forks into erasing heads and folding dimensions, and MLA takes the second fork.

The wall that head-count compression hits

Decoding produces one token at a time and re-reads the Key and Value of every preceding token. The KV cache therefore has to stay in GPU memory until the request finishes. Its size grows linearly with batch size and context length, so a 128K context at batch 32 leaves tens of gigabytes occupied by the cache alone.

The cache footprint of a single token is $2 L n_h d_h$. Here $L$ is the layer count, $n_h$ is the number of attention heads, and $d_h$ is the head dimension. The leading 2 stands for the two copies, Key and Value. Only two terms in that expression are open to change: head count and head dimension.

GQA (Grouped-Query Attention) and MQA (Multi-Query Attention) work on the head-count term. Several query heads share one set of Key and Value, which lowers $n_h$ to a group count $n_g$. The cache shrinks in proportion, but inside a group no head can capture a feature different from its neighbors. Compression ratio and quality end up tied to the same knob.

MLA (Multi-head Latent Attention) folds the remaining axis, the dimension. Head count stays untouched while the dimension of the cached vector goes down. The MLA row is the only one without a leading 2 because Key and Value are not stored separately.

Caching one latent vector

MLA is the attention variant introduced by DeepSeek-V2 (arXiv:2405.04434, 2024). Instead of keeping Key and Value as they are, it squeezes the input hidden state into a low-dimensional latent vector $c$ and caches only that. Key and Value are reconstructed from $c$ when they are needed.

During prefill, a down-projection matrix $W_d$ compresses the hidden state to $d_c$ dimensions. The Key and Value used in the attention computation come from an up-projection matrix $W_u$ that expands $c$ back out. Both matrices are low-rank structures obtained through training, so compression and reconstruction follow a fixed rule.

• Only two pieces enter the cache: the latent vector $c$ and the positional key $k_r$.
• What differs per head is the up-projection matrix, not the cache, so head count never multiplies the cache size.
• The DeepSeek-V2 configuration uses $d_c = 512$ and $d_r = 64$, and cache per token dropped 93.3% against MHA.
• The reason positional information sits in a separate stream is covered in the next section.

The compression comes from a single dimensionality reduction. Key and Value that used to be scattered across heads collapse into one latent vector, and the cache size falls from $2 n_h d_h$ to $d_c + d_r$.

The absorption trick and decoupled RoPE

Caching only the latent vector still repays the saved memory in compute if Key and Value are reconstructed at every step. The absorption trick removes that repayment. An attention score is the inner product of a query and a key, and since the key is $k_j = W_u c_j$, the score can be rewritten as follows.

$q_t^\top k_j = q_t^\top W_u c_j = (W_u^\top q_t)^\top c_j$

$W_u$ is a fixed matrix once training ends. It can therefore be multiplied into the query projection matrix ahead of time, and inference takes the inner product of the absorbed query with $c_j$ read straight from the cache. The step of materializing the Key in memory disappears entirely. The Value-side up-projection is absorbed into the output projection matrix the same way.

The trick holds only at inference. Training has to materialize Key and Value explicitly for backpropagation, so absorption is unavailable and MLA training memory stays close to MHA. A larger query-side matrix after absorption can also raise the compute of the prefill stage, which belongs in the same accounting.

RoPE (Rotary Position Embedding) collides with this directly. RoPE multiplies the key by a rotation matrix $R_j$ that varies with token position $j$, which makes the key $R_j W_u c_j$. Because $R_j$ changes per position, $W_u$ cannot be peeled off and moved to the query side.

MLA avoids the collision by splitting the key into two streams.

• NoPE stream: position-independent semantic information. It is compressed into the latent vector $c$ and cached, and absorption applies only here.
• RoPE stream: a $d_r$-dimensional key carrying position alone. It sits in a small separate cache with the rotation already applied.

The final score is the sum of the two inner products. Folding position into the latent vector would require a different latent vector per position, which erases the compression gain. The latent representation would also become dependent on position, which makes training unstable. Decoupled RoPE isolates the position dependence in a small cache and leaves everything else compressible.

What separates expressiveness at an equal cache budget

The question worth asking when MLA and GQA sit side by side is not cache size but what a given cache can express. TransMLA (arXiv:2502.07864, 2025) answers with Theorem 1. At an equal KV cache budget, the expressiveness of MLA is strictly greater than that of GQA. Every GQA can be converted into an equivalent MLA, while the reverse is generally impossible.

The structure explains why. GQA shares Key and Value across heads verbatim, so heads bound into one group read the same cache in the same way. MLA expands a low-rank structure learned in latent space through a different up-projection per head. Shared information and per-head information live together inside the same cache.

Measurements point the same direction. The same paper reports that converting Qwen2.5 7B and 14B from GQA to MLA raised accuracy on math and coding tasks with a parameter increase below 1.3%. The gain arrived without any increase in the cache budget.

The price sits on the compute side. MLA carries a slightly higher up-projection cost than GQA during training, and as noted above a larger absorbed query matrix raises prefill compute. It trades saved memory for spent compute, and that trade favors MLA in serving environments where decoding is bound by memory bandwidth.

Moving a trained MHA model over

Using MLA does not require training a model from scratch. MHA2MLA (arXiv:2502.14837, 2025) decomposes already-trained MHA weights with SVD and moves them into the MLA structure. The recommended SVDjoint approach concatenates the Key and Value projection matrices and factors them in one pass, outperforming separate factorization by 0.91 to 1.38%.

Joint factorization works better because the two matrices have to share the same latent vector. Separate factorization finds the subspace that is optimal for each matrix alone, and that optimum does not survive reconstruction from a shared cache.

Adding quantization pushes the ratio further. The same work applied Int4HQQ quantization to reach 92.19% KV cache compression. It reports 70 KB per token on DeepSeek-V3 against 192 to 328 KB for GQA models, a factor of 2.7 to 4.7.

Quantizing the latent vector lets the error pass through the up-projection and amplify across the whole of Key and Value. Per-head scale correction and calibration on a proper calibration dataset are required, and the high compression ratio makes per-task downstream validation necessary as well.

Shrinking the transfer in disaggregated serving

In disaggregated serving, where prefill and decode run on separate GPU pools, the KV cache becomes a network payload. Cache size is transfer volume, so the choice of attention variant translates directly into a bandwidth requirement.

Measurements from KVServe (arXiv:2605.13734, SIGCOMM 2026) give the scale. On Qwen3-235B, one 32K-token request produces several gigabytes of KV cache, and moving it in time calls for 2.1 Tbps of egress bandwidth. Cross-cluster bandwidth in a typical cloud sits at or below 100 Gbps. In the same experiments, KV communication accounted for up to 60% of job completion time (JCT).

There are two responses to that gap: compress the cache after it exists, or make it small in the first place. KVServe takes the first route. It selects a combination of transform, quantization, and entropy coding per request based on bandwidth, latency SLO, and accuracy requirements. That cut TTFT by up to 32.8x and JCT by up to 9.13x. MLA takes the second, and because the compression lives inside the model structure, it adds no extra delay on the transfer path.

MLA does open one new choice in disaggregated serving: where the up-projection runs. Reconstructing Key and Value on the prefill node before sending returns the transfer volume to MHA levels. Reconstructing on the decode node keeps the transfer small at the cost of extra decode-side compute. TPLA (arXiv:2508.15881, 2025) reports that combining MLA with tensor-parallel disaggregation greatly reduces KV transfer cost against MHA-based disaggregated serving. The same work points out that a latency-versus-bandwidth trade appears in its place.

Summary

MLA shrinks the KV cache by folding the dimension of the cached vector instead of cutting head count. It exploits the fact that the up-projection matrix is fixed after training and absorbs that matrix into the query side. Inference then reads only the latent vector and never reconstructs Key and Value. RoPE, which varies with position, does not fit that absorption and was split into a small separate stream, which leaves everything else compressible.

Cache per token fell 93.3% against MHA in the DeepSeek-V2 configuration, and TransMLA proved that MLA is strictly more expressive than GQA at an equal cache budget. In disaggregated serving that compression carries straight over into lower transfer volume. Placing the up-projection on the prefill or the decode side then becomes a new decision between latency and bandwidth.