Transformer Architecture and LLM Serving Optimization

The attention architecture determines inference cost, and serving optimization is the work of cutting that cost with KV caching, batching, and quantization.

Serving a large language model (LLM) quickly and cheaply starts with the model's internal architecture. Inference costs include KV access, weight reads, feed-forward computation, and scheduling. The transformer block explains how KV caching, batching, and quantization reduce that memory cost in different ways.

The two halves of a transformer block

Models in wide use today stack many transformer blocks. Text is split into tokens, which the embedding layer converts into vectors. Those vectors incorporate context as they pass through the blocks. One block consists of two parts: self-attention and a feed-forward network.

Self-attention is the stage that reads relationships between tokens and mixes information across them. The feed-forward network (FFN) applies a nonlinear transformation to each token's vector independently. It processes the context that self-attention gathered, inside the token, without referring to any other token.

flowchart LR
    IN["Token vectors<br/>input embeddings"] --> SA["Self-attention<br/>mixes information across tokens"]
    SA --> FFN["Feed-forward network<br/>per-token independent transform"]
    FFN --> OUT["Next block<br/>or final output"]

In real implementations, self-attention takes the form of multi-head attention, computed in parallel across several heads. Layer normalization sits before and after both parts to stabilize training. Each block's output becomes the next block's input, and the last layer's output is used to predict the next token.

From encoder-decoder to decoder-only

The original transformer splits into an encoder and a decoder. Inside a decoder block there is one more component alongside self-attention: cross-attention. Cross-attention lets decoder tokens refer to the encoder's output vectors, modeling the output probability p(y|x) conditioned on the input.

GPT-style generative models drop the encoder and stack decoders only. This structure generates tokens one at a time, autoregressively. The fact that producing the next token requires looking again at every token so far is where the serving cost begins.

The asymmetry between prefill and decode

LLM inference splits into two stages with different characteristics. Prefill processes the entire prompt in one pass, and decode generates tokens one at a time. That the two stages bottleneck on different resources is the central premise of every optimization here.

Large prefills can use GPU compute efficiently, while small-batch decodes are often limited by weight and cache reads. Larger batches and longer contexts can change the bottleneck, so confirm it in the actual execution.

The cache that accumulates here is the key-value (KV) cache. Autoregressive generation must keep the keys and values of previous tokens in GPU memory (VRAM). Dense attention accesses the accumulated cache at every step. Reading from high bandwidth memory (HBM) can limit small-batch decode performance.

flowchart LR
    P["Prompt tokens"] --> PRE["Prefill<br/>input tokens in parallel"]
    PRE --> KV["KV cache<br/>keys and values accumulate"]
    KV --> DEC["Decode<br/>one token at a time"]
    DEC --> KV
    DEC --> Y["Generated tokens"]

KV cache and PagedAttention

KV cache size scales linearly with context length and batch size. In multi-head attention (MHA), cache size is roughly proportional to 2 × layers × heads × head_dim × sequence_length. As concurrent requests grow, KV cache memory use grows first and limits concurrency.

Earlier serving stacks preallocated a contiguous block of memory per request sized for the maximum length. When actual generation turned out short, most of that reserved space sat empty as internal fragmentation. PagedAttention borrows the idea of virtual memory paging from operating systems to eliminate that waste.

PagedAttention maps the logical KV cache onto non-contiguous physical blocks, allocating them dynamically as needed. Any free block can be used immediately, so fragmentation is close to zero. The vLLM paper (Kwon et al., SOSP 2023) reports 2–4× higher throughput at the same latency level than FasterTransformer and Orca on its evaluated models and workloads.

Those numbers come from reducing memory waste and therefore fitting more requests on one GPU. In other words, the direct cause of the throughput gain is better memory management, not a faster model. Whether free memory becomes higher throughput also depends on computation and scheduling costs.

Filling the GPU with continuous batching

Batching means processing several requests together in one pass. Static batching holds the GPU until the longest request in the batch finishes. Short requests finish early and leave their slots empty, which lowers utilization.

Continuous batching schedules at the granularity of a single token generation step (an iteration). Finished requests are removed from the batch immediately and waiting requests take their slots. This token-level scheduling was proposed in Orca (OSDI 2022), and vLLM adopted it as standard alongside PagedAttention.

Continuous batching reduces empty slots by replacing completed requests with new ones. The gain comes from filling the empty slots of static batching. More KV cache headroom therefore allows a larger gain. PagedAttention saves the memory and continuous batching fills it.

Attention variants and quantization that shrink memory

There is also a direction that shrinks the KV cache itself. It splits two ways: reducing the number of KV heads in attention, and lowering the storage precision. Attention variants cut cache size directly by sharing or compressing KV heads.

Grouped-Query Attention (GQA) shares KV heads among query heads. The paper reports quality close to MHA in its tested configurations. Multi-Query Attention (MQA) reduces KV heads to one, which is smaller still but can degrade quality. Multi-head Latent Attention (MLA) compresses KV into a low-rank latent vector. The DeepSeek-V2 report describes this architecture; compute cache size from actual dimensions and the comparison model.

The second branch is quantization. Weight quantization converts 16-bit weights to a lower precision such as 4-bit. Going from 16-bit to 4-bit arithmetically reduces weight memory to about a quarter. Activation-aware Weight Quantization (AWQ) and GPTQ are the representative techniques for limiting the accuracy loss.

The KV cache itself can also be quantized. fp8 is half of 16 bits, so the KV cache memory halves. This is arithmetic for element storage, excluding scales and metadata; it does not establish proportional improvements in total memory or throughput.

# vLLM OpenAI-compatible server: store the KV cache in fp8, cap the context length
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --max-model-len 8192

The --kv-cache-dtype fp8 flag above is the vLLM option that stores the KV cache at 8 bits to save memory. When serving an already quantized checkpoint, the --quantization flag selects the AWQ or GPTQ format. The attention computation itself is restructured for the memory hierarchy by FlashAttention (Dao et al., NeurIPS 2022), which cuts HBM traffic.

Managing latency and throughput together

Serving optimization has to watch throughput and latency at the same time. In speculative decoding, a small draft model predicts several tokens ahead and the large model verifies them in one parallel pass. The original acceptance and correction rules preserve the target distribution. Latency benefit depends on drafting and verification costs.

Prefill and decode, which bottleneck differently, can also be split across separate GPU pools. This prefill-decode (PD) disaggregation manages contention and resource allocation separately for each stage. DistServe (OSDI 2024) reports up to 7.4× more requests or up to 12.6× tighter service-level objectives (SLOs) in its evaluated conditions. These are separate maxima, not simultaneous gains in every configuration. Include KV transfer and pool queues in the evaluation.

Splitting a model across GPUs serves different goals depending on how it is split. Tensor parallelism divides a single layer across GPUs to lower the latency of a single request. It requires a fast interconnect between GPUs. Pipeline parallelism assigns groups of layers to each GPU and raises throughput.

All of these techniques are measured with the metrics below. Which metric takes priority changes which technique to reach for.

Time To First Token (TTFT) matters where the first response has to be fast, as in chat. Time Between Tokens (TBT) governs whether a long streamed answer arrives without stutter. Prioritizing throughput points to batching and quantization first; prioritizing latency points to speculative decoding and PD disaggregation.

Summary

The transformer's attention architecture and autoregressive generation determine LLM inference cost. Prefill and decode bottlenecks vary with batching and context, while KV capacity also limits concurrency. Serving optimization therefore focuses on reducing KV cache memory use and keeping the GPU fully occupied.

PagedAttention reduces memory waste, and continuous batching fills the freed memory to raise utilization. GQA, MLA, and KV quantization reduce cache representation; weight quantization reduces weight memory. Apply speculative decoding and PD disaggregation first when latency matters, and batching and quantization first when throughput matters. Whichever technique is used, the safe order is to confirm the effect with the TTFT, TBT, and throughput metrics.