Transformer Architecture and LLM Serving Optimization

The attention architecture determines inference cost, and serving optimization is the work of cutting that cost with KV caching, batching, and quantization.

Serving a large language model (LLM) quickly and cheaply starts with the model's internal architecture. Most of the inference cost comes from attention piling keys and values into memory and rereading them at every step. Once the transformer block is clear, KV caching, batching, and quantization all reveal themselves as different angles on cutting that same memory cost.

The two halves of a transformer block

Models in wide use today stack many transformer blocks. Text is split into tokens, the embedding layer turns them into vectors, and those vectors pick up context as they pass through the blocks. One block consists of two parts: self-attention and a feed-forward network.

Self-attention is the stage that reads relationships between tokens and mixes information across them. The feed-forward network (FFN) applies a nonlinear transformation to each token's vector independently. It processes the context that self-attention gathered, inside the token, without referring to any other token.

In real implementations, self-attention takes the form of multi-head attention, computed in parallel across several heads. Layer normalization sits before and after both parts to stabilize training. Each block's output becomes the next block's input, and the last layer's output is used to predict the next token.

From encoder-decoder to decoder-only

The original transformer splits into an encoder and a decoder. Inside a decoder block there is one more component alongside self-attention: cross-attention. Cross-attention lets decoder tokens refer to the encoder's output vectors, modeling the output probability p(y|x) conditioned on the input.

GPT-style generative models drop the encoder and stack decoders only. This structure generates tokens one at a time, autoregressively. The fact that producing the next token requires looking again at every token so far is where the serving cost begins.

The asymmetry between prefill and decode

LLM inference splits into two stages with different characteristics. Prefill processes the entire prompt in one pass, and decode generates tokens one at a time. That the two stages bottleneck on different resources is the central premise of every optimization here.

Prefill receives all tokens at once and saturates the GPU compute units. Decode reads billions of parameters and the accumulated cache from memory at every step while doing very little arithmetic. So decode leaves compute units idle and stalls on memory bandwidth.

The cache that accumulates here is the key-value (KV) cache. Autoregressive generation must keep the keys and values of previous tokens in GPU memory (VRAM). Rereading that entire cache from high bandwidth memory (HBM) at every step is what makes decode memory-bound.

KV cache and PagedAttention

KV cache size scales linearly with context length and batch size. In multi-head attention (MHA), cache size is roughly proportional to 2 × layers × heads × head_dim × sequence_length. So as concurrent requests grow, the KV cache is the first thing to eat memory, and it becomes the ceiling on concurrency.

Earlier serving stacks preallocated a contiguous block of memory per request sized for the maximum length. When actual generation turned out short, most of that reserved space sat empty as internal fragmentation. PagedAttention borrows the idea of virtual memory paging from operating systems to eliminate that waste.

PagedAttention maps the logical KV cache onto non-contiguous physical blocks, allocating them dynamically as needed. Any free block can be used immediately, so fragmentation is close to zero. The vLLM paper (Kwon et al., SOSP 2023) reported 2 to 4 times higher throughput on the same hardware with this approach.

Those numbers come from reducing memory waste and therefore fitting more requests on one GPU. In other words, the direct cause of the throughput gain is better memory management, not a faster model. The principle that saving memory raises concurrency applies identically to the techniques that follow.

Filling the GPU with continuous batching

Batching means processing several requests together in one pass. Static batching holds the GPU until the longest request in the batch finishes. Short requests finish early and leave their slots empty, which lowers utilization.

Continuous batching schedules at the granularity of a single token generation step (an iteration). Finished requests are removed from the batch immediately and waiting requests take their slots. This token-level scheduling was proposed in Orca (OSDI 2022), and vLLM adopted it as standard alongside PagedAttention.

With continuous batching, GPU utilization can rise from the 30-40% range to the 75-90% range depending on the workload (estimate). Since the gain comes from filling the empty slots of static batching, the more KV cache headroom there is, the larger the gain. PagedAttention saves the memory and continuous batching fills it.

Attention variants and quantization that shrink memory

There is also a direction that shrinks the KV cache itself. It splits two ways: reducing the number of KV heads in attention, and lowering the storage precision. Attention variants cut cache size directly by sharing or compressing KV heads.

Grouped-Query Attention (GQA) has several query heads share one KV head, shrinking the cache while holding quality. Multi-Query Attention (MQA) reduces KV heads to one, which is smaller still but can degrade quality. Multi-head Latent Attention (MLA) compresses KV into a low-rank latent vector; by the DeepSeek-V2 technical report it cut the KV cache by about 93%.

The second branch is quantization. Weight quantization converts 16-bit weights to a lower precision such as 4-bit. Going from 16-bit to 4-bit arithmetically reduces weight memory to about a quarter. Activation-aware Weight Quantization (AWQ) and GPTQ are the representative techniques for limiting the accuracy loss.

The KV cache itself can also be quantized. fp8 is half of 16 bits, so the KV cache memory halves. The same memory can then support roughly twice the concurrency (estimate).

# vLLM OpenAI-compatible server: store the KV cache in fp8, cap the context length
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --max-model-len 8192

The --kv-cache-dtype fp8 flag above is the vLLM option that stores the KV cache at 8 bits to save memory. When serving an already quantized checkpoint, the --quantization flag selects the AWQ or GPTQ format. The attention computation itself is restructured for the memory hierarchy by FlashAttention (Dao et al., 2022), which cuts HBM traffic.

Managing latency and throughput together

Serving optimization has to watch throughput and latency at the same time. In speculative decoding, a small draft model predicts several tokens ahead and the large model verifies them in one parallel pass. Verified tokens are used as is, so quality stays at the large model's level while latency drops.

Prefill and decode, which bottleneck differently, can also be split across separate GPU pools. This prefill-decode (PD) disaggregation removes resource contention between the two stages and improves both latency metrics. DistServe (OSDI 2024) reported up to 7.4 times higher throughput on the same hardware budget. Measured against latency targets, that translates to up to 12.6 times tighter service level objectives (SLOs).

Splitting a model across GPUs serves different goals depending on how it is split. Tensor parallelism divides a single layer across GPUs to lower the latency of a single request, but it requires a fast interconnect between GPUs. Pipeline parallelism assigns groups of layers to each GPU and raises throughput.

All of these techniques are measured with the metrics below. Which metric takes priority changes which technique to reach for.

Time To First Token (TTFT) matters where the first response has to be fast, as in chat. Time Between Tokens (TBT) governs whether a long streamed answer arrives without stutter. Prioritizing throughput points to batching and quantization first; prioritizing latency points to speculative decoding and PD disaggregation.

Summary

The transformer's attention architecture and autoregressive generation determine LLM inference cost. Prefill is compute-bound, decode is memory-bound, and the KV cache sets the ceiling on concurrency. So the main thread of serving optimization converges on saving KV cache memory and keeping the GPU fully occupied.

PagedAttention eliminates memory fragmentation, and continuous batching fills the freed memory to raise utilization. Attention variants such as GQA and MLA, along with weight and KV quantization, shrink the cache itself. Apply speculative decoding and PD disaggregation first when latency matters, and batching and quantization first when throughput matters. Whichever technique is used, the safe order is to confirm the effect with the TTFT, TBT, and throughput metrics.