Specializing LLM Inference Stages

The assumption that all inference stages share one weight precision, one context, and one execution order is breaking down. The gain from a stage-specific setting depends on where that stage is limited and on which component holds the global context.

LLM serving usually loads one checkpoint at one precision. All stages then process the same context in a fixed order. Three studies from 2026 each remove one of these assumptions at a different stage. Disaggregated quantization gives prefill and decode different precisions, DLite reduces the context that the drafter reads, and Saguaro overlaps draft and verify.

All three studies report gains, but each split has a cost: extra weight copies, extra training, or extra GPUs. To decide which stage to split, you must look at the conditions where the gain appeared and the conditions where it disappeared. This post sorts those conditions along two axes: the bottleneck type and the context dependency.

Where Each Stage Is Limited

Prefill processes the full prompt in one pass, and decode generates one token at a time. The asymmetry between the two stages is covered in Transformer Architecture and LLM Serving Optimization. The disaggregated quantization paper (arXiv:2609.26333) explains the difference with the arithmetic of one linear layer.

When an H×H weight processes T tokens, the layer transfers O(H²+TH) elements and does O(TH²) arithmetic. In a sufficiently long prefill, T is large, so the weight transfer is spread over much arithmetic and the linear layers are generally compute-bound. In decode at batch one (T=1), each weight is used for only one multiply-add, so weight loading sets the latency.

Speculative decoding adds two more stages. In the draft stage, a small drafter proposes K candidate tokens. In the verify stage, the target model checks all candidates in one forward pass. The average number of tokens accepted per round is the acceptance length (τ). The speedup increases when τ is large and when draft and verify are short.

Choosing Precision per Stage

Disaggregated quantization by Panferov et al. (arXiv:2609.26333, 2026-09) gives the two stages different formats. Prefill quantizes both weights and activations to NVFP4, a 4-bit floating-point format that the hardware computes directly (W4A4). Decode uses a low-bit weight-only format that reduces the number of bits to move.

The starting point is an asymmetry in sensitivity. The authors quantized only one stage to NVFP4 at a time and compared the losses of the two stages.

On decode-heavy tasks such as GSM8K, MATH-500, and MMLU-Pro, the decode-only loss was 2 to 4 times the prefill-only loss. This range applies to most models, and the ratio was up to 7 times on Gemma3-1B. On RULER, which has long prompts and short answers, the prefill-only loss was 1.1 to 4.1 times the decode-only loss on seven of eight models.

The cheapest split is format disaggregation. It shares one set of weights and turns off activation quantization only in decode. The paper says this makes memory-bound decode slightly faster because it skips activation quantization. A larger difference comes from full disaggregation, which keeps separate prefill weights. The difference is clearest at 2 bits, where the accuracy of weight-only decode drops sharply.

Full disaggregation keeps two sets of weights, so device memory increases. The paper's ODP (Offloaded Disaggregated Prefill) reads the prefill weights block by block from SSD storage and brings this memory back down.

The values come from Table 1 of the paper. Speed and memory are for Qwen3-8B, and all of them were measured on DGX Spark. Prefill is measured at 16K context, and decode as per-token latency in vLLM at batch one. Accuracy is the average over Qwen 3 (0.6B to 8B) / Gemma 3 (1B to 12B) model sizes. LUT2 is a 2-bit lookup table format.

Full disaggregation has higher accuracy than 2-bit weight-only on both task types. But the resident weights increase from 4.66 GB to 8.57 GB. With ODP, the memory goes back to 4.66 GB, and the prefill speedup decreases slightly, from 1.49x to 1.47x.

The largest difference comes from an experiment that keeps the decode checkpoint fixed. The authors kept a public Qwen3.8-27B GGUF IQ1_S decoder unchanged and trained only an NVFP4 prefiller for it.

As a result, MMLU-Pro increased from 29.04 to 61.54, and MMMU-Pro increased from 24.39 to 59.65. The BF16 references are 84.62 and 75.26. The evaluation used vLLM on GB300 at temperature 1.0, with one run per cell.

The paper separates part of this gain. An untrained RTN (round-to-nearest) NVFP4 prefill alone gave 48.39 on MMLU-Pro and 30.29 on MMMU-Pro. Comparison with 61.54 and 59.65 for the trained prefiller separates two gains: the gain from a separate prefill by itself and the gain added by training.

Response length also changed. On MMMU-Pro with IQ1_S, the mean response length fell from 14,579 to 6,582 tokens, a 54.8% reduction. The rate of responses truncated at the maximum length fell from 24.57% to 4.16%. But the path that produces the accuracy gain is not isolated.

When the Drafter Does Not Read Its History

DLite (arXiv:2610.23787, 2026-10) reports that the acceptance length of existing drafters drops by up to 80% on agentic workloads. In the same paper, the DFlash drafter for a Qwen3-4B target had τ of 5.97 on GSM8K. On AgentLongBench, with inputs of 55.5K to 74.1K tokens, τ was 1.14. The speed was 0.95x that of autoregressive decoding, so speculation made decoding slower.

The authors diagnose that the drafter repeatedly reconstructs history that the target has already processed. This load also appears as a system cost.

The authors measured the MTP (multi-token prediction) module of DeepSeek-V4 in SGLang. KV cache updates and upkeep took 10% to 20% of end-to-end latency. The hardware and batch for this measurement are not given.

DLite limits each round's input to three items: the target's contextual hidden states, the current anchor token, and K draft placeholders. It uses local positions inside the block, not accumulated global positions. It keeps no draft KV cache between rounds. The authors describe this as a split: the target models the global context, and the drafter makes only local predictions.

The conditions are H800 GPUs and a block size of K=7. The baseline DSpark is the semi-autoregressive drafter that the paper names as the prior state of the art. It combines a parallel backbone with a lightweight sequential module.

On GSM8K, the τ difference is about 2%. On agentic workloads with inputs of tens of thousands of tokens, the difference becomes larger. The authors explain that the drafter can drop its history because the target has already processed that history and passes it on through hidden states.

The Opposite Direction Adds Context

ReaDy (arXiv:2610.20503, 2026-10) adds context to the same DSpark drafter. It keeps the parallel backbone and the sequential head of DSpark. It reads the recurrent states of the target and adds them as supplementary context.

The target, Qwen3.8-27B, mixes Gated DeltaNet linear attention layers with full attention layers. Each linear layer summarizes the history in a recurrent state of 128×128 per head, and this size does not change with prefix length. ReaDy reads the states of layers 30 and 62 with a query for each draft position.

On the average of seven benchmarks, with one B200, SGLang, concurrency 1, and greedy decoding, throughput increased by 7.97%. It went from 280.42 tokens per second (TPS) for DSpark to 302.78 TPS. A control that used the same fine-tuning procedure without state access gained 2.17%. When the state of the shallow layer 2 was also added, the gain decreased to 6.22%.

DLite removes the drafter's history. ReaDy adds a fixed-size summary, computed by the target, to the existing history path of DSpark. The DLite paper classifies DSpark as a drafter that still depends explicitly on a history whose length grows. In both studies, the drafter gets history information from representations that the target computed. This is a judgment.

The two studies use different targets, hardware, and workloads, so their numbers cannot be compared directly. No experiment compares them on the same workload.

Context in Target Decode

Target decode holds the global context itself, so its case is different. SAS (arXiv:2609.13141, 2026-09) keeps dense attention in prefill and makes only decode attention sparse. Each query reads the current block and a subset of 64-token blocks that a selector picks.

On Qwen3-4B, the GPQA-Diamond result changed with the budget. At a budget of 1024 tokens, SAS reached 50.41%, lower than 56.19% for full attention. At the same budget, the distillation-based SeerAttention-R reached 39.84%, StreamingLLM 13.86%, and a sliding window 1.52%.

When the budget increases to 2048 and 4096 tokens, SAS reaches 54.86% and 55.11%. This difference is within the ±2.8 to 2.9 range reported for SAS.

The speed gain comes at long contexts. At batch one, the speed at 8K context was almost the same as full attention. It was 2.4x faster at 64K and 5.6x faster at 512K.

At long contexts, the accuracy loss also becomes large. On RULER in the appendix, at a budget of 4096, Qwen3-4B scored 35.53 with SAS against 73.17 with full attention at 64K. At 128K, the scores were 21.87 against 63.81.

The range where speed improves overlaps the range where accuracy drops sharply. Thus the context of target decode is harder to reduce than the drafter's history. This is a judgment.

Drafting Without Waiting for Verification

In standard speculative decoding, the drafter must know the verification result before it makes the next candidates. The accepted length decides the context that the next candidates continue from. Thus draft and verify run in turns.

Speculative Speculative Decoding (arXiv:2603.03251, ICLR 2026) by Kumar, Dao, and May overlaps these two stages. It treats a verification result as a pair: the accepted length k and the bonus token. While the target verifies, a drafter on a separate GPU prepares the next candidates for each likely result. Saguaro is the optimized algorithm for this method, and this post calls the method Saguaro.

flowchart LR
  V["verify: target checks current candidates"] --> O["Actual result<br/>accepted length k, bonus token"]
  P["draft GPU: next candidates for each expected result"] --> C["speculation cache"]
  O --> H{"Is the result in the cache?"}
  C --> H
  H -->|"hit"| R["Use prepared candidates in the next verify"]
  H -->|"miss"| B["Backup drafter makes candidates"]
  B --> R2["Next verify after drafting ends"]

The devices do not move the KV cache between them. They exchange only verification results, candidate tokens, and logits. If the actual result is in the cache, the prepared candidates are used at once. If not, the backup drafter makes the candidates.

Appendix B.3 of the paper reports the average decode speed over four datasets for a Llama-3.1-70B target with a Llama-3.2-1B drafter. Autoregressive decoding reached 54.7 tok/s, speculative decoding 161.8 tok/s, and Saguaro (SSD in the paper) 255.8 tok/s. The conditions were batch one and 512 output tokens, with prefill excluded. Autoregressive and speculative decoding used four H100 GPUs, and Saguaro used five, with one more for the drafter.

In the batch model of the paper, the gain from the overlap decreases as the batch becomes larger. This model, Corollary 16, assumes that cache hits are independent across requests. It also assumes that one miss makes the whole batch wait for the backup.

p      : probability that one request hits the cache
b      : batch size
T_p    : primary drafter time (verification time = 1)
T_b    : backup drafter time
E_hit  : expected generated tokens on a hit
E_miss : expected generated tokens on a miss
 
speedup(b) = (p*E_hit + (1-p)*E_miss) / (p^b * max(1, T_p) + (1 - p^b) * (1 + T_b))
As b increases, p^b -> 0, so speedup -> (p*E_hit + (1-p)*E_miss) / (1 + T_b)

The probability that the whole batch hits, p^b, goes toward zero as the batch becomes larger. Then the max(1, T_p) term, which comes from the overlap of verify and draft, disappears. The speed is then set by the backup time T_b and the yield on a miss. The paper switches to a fast backup at large batch sizes, and the backup in the public implementation returns random tokens.

Conditions Where Specialization Does Not Pay

The first condition is how much accuracy the decoder lost. The prefiller gain followed the accuracy drop of the weight-only decoder more closely than the nominal bit count. Even among 2-bit formats, IQ2_S gained 2.80 on MMLU-Pro and 0.87 on MMMU-Pro, while Q2_K_XL changed by +0.18 and −0.64.

With 3-bit decoders, the gain was almost zero, and MMMU-Pro was 1.97 to 2.77 lower for all three formats. But Table 5 uses one run per cell at temperature 1.0, with no significance test. Small differences such as −0.62 and −0.63 on MMLU-Pro cannot be called a decrease with confidence.

The second condition is batch size. The decode speed of disaggregated quantization was measured only at batch one. The authors state that they did not evaluate highly batched serving, multi-turn use, or agentic behavior.

By the arithmetic above, decode linear layers reuse weights over more tokens as the batch grows, so the reason for a precision split can become weaker. This is an estimate.

DLite's speedup also decreased as concurrency increased. On Qwen3-8B with LiveCodeBench, 3.95x at concurrency 1 became 2.13x at concurrency 64. Under the same conditions, DSpark also decreased from 3.60x to 2.08x. This concurrency test covers only general tasks such as Alpaca, MT-Bench, and LiveCodeBench, with no results for agentic workloads.

Between concurrency 1 and 64, the margin of DLite over DSpark differed by model. It grew from 2.1% to 6.3% on Qwen3-4B with MT-Bench, but fell from 9.8% to 2.4% on Qwen3-8B with LiveCodeBench. On Qwen3-4B with MT-Bench at concurrency 16, DLite's 8162 TPS was slightly lower than DSpark's 8177 TPS.

The third condition is the coupling between stages. The disaggregated quantization authors note that, in multi-turn use, cached assistant tokens keep the KV entries that decode produced. If the same history is rebuilt through prefill, a different cache results. This dependency is not tested.

The fourth condition is the strength of the evidence. The direct evidence that DLite does not need history KV is Fig. 5(b). It compares DLite with sliding-window variants of 128 and 512 tokens, trained with the same data and setup.

The text says only that the models reach comparable acceptance lengths. It gives no numbers or benchmarks, and it does not state the evaluation temperature.

The fifth condition is cost. Full disaggregation increases the resident weights by 1.7 to 2.1 times. ODP, which removes this increase, is for dense models and does not transfer directly to mixture-of-experts (MoE) models. With short prompts, loading from SSD storage stalls the computation.

DLite needs two-stage training through an isomorphic teacher, and the Saguaro comparison uses different GPU counts.

The sources leave some questions open. None of the papers above tests several splits together.

For example, verify processes K candidates at once, so its T is larger than 1 and much smaller than in prefill. Whether verify should use the prefill format or the decode format is an open question. This is a judgment.

Criteria for Splitting a Stage

Two axes decide whether to split a stage. Is the bottleneck of the stage different from that of the adjacent stages? Does the stage hold the global context itself? Answer four questions in order.

• Are the bottlenecks different? Measure first whether prefill is compute-bound and decode is load-bound at your batch size and prompt length. If the batch is large enough that decode also becomes compute-bound, the reason for a precision split can become weaker. This criterion depends on the estimate stated above.
• Which stage sets quality? Quantize only one stage and measure the loss. The disaggregated quantization paper used this method, and decode-heavy and prefill-heavy tasks gave opposite results.
• Which component holds the global context? If the target holds it, the drafter can drop its history (DLite) or add a fixed-size summary (ReaDy). If target decode holds it, only learned selection can reduce it, and the accuracy loss is large at long contexts.
• Can the result of a serial dependency be predicted? At batch one with a spare GPU, result prediction can overlap draft and verify. At large batch sizes, p^b reduces the gain.

If prefill and decode run on different GPUs, moving the KV cache between the stages adds a cost. That cost and the question of cache placement are covered in LLM Serving Seen Through the KV Cache. A structure that reduces the cache to move is in Multi-head Latent Attention and KV Cache Compression.

References

• Disaggregated Quantization: Specializing LLM Prefill and Decode, arXiv:2609.26333 (2026-09)
• DLite: History-free Speculative Decoding for Agentic AI, arXiv:2610.23787 (2026-10)
• ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models, arXiv:2610.20503 (2026-10)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking, arXiv:2609.13141 (2026-09)
• Speculative Speculative Decoding, arXiv:2603.03251 (2026-03, v3 2026-05, ICLR 2026)

Every figure was checked against its original source as of 2026-10.

Summary

The four stages of LLM inference are limited in different places, and studies from 2026 gained by splitting precision, context, and execution order per stage. When the accuracy of weight-only decode dropped sharply, disaggregated quantization used a separate prefill to recover part of the lost accuracy.

DLite increased the speedup on agentic workloads with inputs of tens of thousands of tokens by dropping the drafter's history. Saguaro overlapped draft and verify at batch one with one more GPU.

The conditions for the gain are the bottleneck type and the context dependency. This is a judgment. If the target holds the global context, the drafter can drop its history or add a fixed-size summary. Target decode holds the context itself, so even learned selection loses much accuracy at long contexts.

The speed of disaggregated quantization was measured only at batch one. On general tasks, the speedups of DLite and DSpark both decreased as concurrency increased, and in the paper's model, Saguaro's gain decreases as the batch becomes larger. The effect of combining several splits is not yet measured.