Measurement Boundaries in LLM Serving Optimization

A serving improvement holds only inside the boundary where it was measured. When costs outside that boundary take the time, gains on a kernel, on one GPU, or on one component disappear or reverse end to end.

Serving optimization results are usually reported as a speedup for one segment of the work. Examples are the run time of one kernel, the inter-token latency on one GPU setup, or the compute time on the GPU side. For that speedup to carry over to per-request performance, the measured segment must be the real bottleneck. The improvement must also not add cost outside the segment.

Intermediate metrics also cannot stand in for outcome metrics. In one case, routing agreement improved but quality measured by negative log-likelihood (NLL) became worse. In another case, the direction was the opposite: more bytes were read, but decode speed increased.

Three main papers published in August and September 2026 show cases where these conditions fail, each at a different place. The places are a kernel-level speedup, a latency signal that a scheduler uses for control, and an execution time measured only on the GPU.

Two supporting papers serve as points of comparison. SLOWeave uses a different control signal, and in Edge0 an intermediate metric moved against the outcome. Together, the five papers give a list of items to check first when you read serving results from papers, benchmarks, or your own experiments.

Kernel Speedup and End-to-End Time

Launch-Bound and Substitutable (arXiv:2608.26612, 2026-08) is a paper on Mixture-of-Experts (MoE) inference optimization. It applied three standard inference optimizations to MoE models and measured the results.

The models are OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B, and the hardware is an A100 80GB. The end-to-end latency was measured on OLMoE-1B-7B with sequence lengths 128, 512, and 1024 and batch sizes 1 and 4.

The first optimization is two kernels fused in Triton, one for RMSNorm and one for the router softmax. In an isolated benchmark, the RMSNorm kernel was 5.62 times faster at hidden size 512 and 8.98 times faster at 4096. The softmax kernel was 2.01 times faster.

With both kernels applied, the end-to-end speed was 0.999x at sequence 512 and batch 4. At sequence 128 and batch 1, it was 0.785x, which is slower than the baseline.

The paper explains this gap with Amdahl's law. Profiler ranges show that the operations of the two kernels take only 7.87% of the forward-pass time (RMSNorm 7.70%, softmax 0.17%). With an effective kernel speedup of 6.88x, the end-to-end ceiling is 1.072x. The measured values did not reach even this ceiling.

def amdahl_ceiling(f: float, s: float) -> float:
    # f: fraction of total time spent in the improved segment
    # s: speedup of that segment measured in isolation
    return 1 / ((1 - f) + f / s)
 
print(round(amdahl_ceiling(0.0787, 6.88), 3))  # 1.072

The paper concludes that the model is bound by kernel launch wait (launch-bound), not by arithmetic. The evidence is that a 32-fold increase in work increased latency only 1.55 times, from 246.8 ms to 383.4 ms. The work went from sequence 128 and batch 1 to sequence 1024 and batch 4.

The count of about 1,000 kernel launches per forward pass is a scale estimate from the loop structure of 64 experts and 16 layers. The paper reports no direct count.

The removal of all 23 graph breaks in torch.compile points in the same direction. At sequence 1024 and batch 4, the model ran at 0.327x of eager speed, about three times slower. At sequence 512 and batch 4, the speed was 0.613x. This sub-study ran with the cuDNN attention backend disabled, so you cannot compare it directly with the end-to-end numbers above.

The authors identify dynamic shapes as the cause. Of the 23 breaks, 16 came from the data-dependent torch.nonzero operation in expert dispatch. When this operation goes into the graph, the compiler generates code and guards that handle any size.

Intermediate Metrics and Output Quality

The INT4 experiment shows the same problem on the quality side. When OLMoE was quantized to 4 bits (NF4) with bitsandbytes, an average of 0.53 of the 8 selected experts changed at each token position. But when only the INT4 routing was applied to the FP16 weights, it reproduced only 2.7% of the NLL increase.

An experiment that kept only the router in FP16 points to the same conclusion. Under NF4, the routing drift measured by Jaccard fell from 0.1140 to 0.0910, about 20% less. But the NLL increase grew from +0.0872 to +0.0979. An improvement in an intermediate metric, routing agreement, can come with worse output quality.

The paper reads this as evidence that the experts can substitute for each other. The causal replay used one model, OLMoE, and one quantization library, so other MoE models need more measurement. The basic structure of expert routing is covered in MoE Load Balancing and Gradient Interference.

Where the Control Signal Is Measured

Decode-Latency Feedback Prefill (DLFP, arXiv:2609.38386, 2026-09) is a controller that sets a cap on the prefill chunk size. It acts only when ongoing decode work and the prefill of a new request overlap.

If the observed interval is longer than the target, the controller decreases the next cap. If it is shorter, the controller increases the cap. The observed value L_t is the time between scheduler calls, and the controller does not measure GPU completion time. In the formula below, floor_128 rounds down to a multiple of 128 tokens.

c_next = clamp(floor_128(c_t * T / L_t), c_min, c_max)
 
T      = 80 ms        target time for a mixed iteration
c_0    = 3072 tokens  initial cap
c_min  = 1024 tokens
c_max  = 6144 tokens
L_t    = observed scheduler-call interval

The primary experiment ran Qwen3-0.6B (BF16) on one A100 80GB with vLLM 0.17.1. Asynchronous scheduling and CUDA graphs were on. A synthetic Agent workload with 16,384 input tokens and 256 output tokens arrived as a Poisson process at 1 req/s. Over 3 seeds with 100 requests each, the P99 inter-token latency (ITL) fell by 27.7% on average (95% confidence interval 21.0 to 34.3%).

In the same experiment, the P99 time to first token (TTFT) increased by 34.8% on average (95% confidence interval 3.6 to 65.9%). The change in service level objective (SLO) goodput was -0.10%, and its confidence interval (-0.50 to +0.29%) included zero. All 300 of 300 requests passed the declared SLO. The SLO was TTFT of 5 s or less, mean ITL of 35 ms or less, and total latency of 14 s or less.

The generalization experiments changed the model and the execution setup. Qwen3-8B and Qwen3-32B ran on two A100 80GB GPUs connected through PCIe, without NVLink. Each row compares a baseline and a candidate under a separate condition. The 8B and 32B rows are small samples of 20 or 12 requests.

In the 8B row at 0.25 req/s, P99 ITL and the service result go in different directions. P99 ITL fell from 1.763 s to 707 ms. But P99 TTFT increased from 12.33 s to 14.81 s, and SLO passes fell from 2/20 to 0/20. A report with P99 ITL alone would show this row as an improvement.

The authors find the cause in where the signal is measured. In their analysis, the scheduler-call interval correlated with GPU iteration completion time on the small model. Larger kernels, asynchronous queueing, and tensor-parallel synchronization break this correlation. When the host sees a short interval, the controller increases the cap, but device work can still be pending at that time.

sequenceDiagram
    participant S as Scheduler (host)
    participant G as GPU
    S->>G: Submit iteration t
    S->>G: Submit iteration t+1
    Note over S: Observe short call interval L_t, increase cap
    S->>G: Submit iteration t+2 with a larger chunk
    G-->>S: Iteration t completes

This diagram is a conceptual reconstruction of the authors' analysis, not a measured timeline. The paper also states in its limitations that it did not measure device completion time directly.

A 32B variant updated the controller at completion time and bounded the size of each control step. Under synchronous scheduling, this variant also had worse P99 ITL and fewer SLO passes than its baseline. This result comes from 12 requests, so it is not evidence that a change of signal alone solves the problem.

SLOWeave (arXiv:2609.07883, 2026-09) selects chunks based on deadlines. It sets each deadline as the last token completion time plus a time per output token (TPOT) target. Then it selects the largest chunk whose predicted cost plus a margin fits in the time left before the earliest deadline.

The guarantee of this method also has a boundary. The paper's proposition makes three assumptions. The cost prediction is exact, the cost does not decrease as chunk size increases, and a decode-only iteration meets the deadline. Under these assumptions, it guarantees the deadline for one iteration only, and TTFT, throughput, and freedom from starvation under overload are outside the guarantee.

Component Metrics and Time per Token

SSD-LLaMA (arXiv:2609.18110, 2026-09) runs inference on a consumer PC with MoE weights stored on an SSD. The test system has an Intel Ultra5 230F, an RTX 5090 (32GB), 16GB of host DRAM, and a 4TB NVMe SSD (up to 9 GiB/s). The batch size is 1. The system makes two decisions: the delivery path that brings weights from the SSD, and whether the CPU or the GPU runs each selected expert.

Section 4.3 of the paper compares the two decisions separately on a DeepSeek model. It uses a 77 MiB layer buffer, and every placement setup performs the same 16,770 SSD reads, 224.204 GB in total. The 22 GiB VRAM cache setup in the same section is a separate comparison with a different cache condition.

The authors state that CPU placement decreased GPU kernel time. But as more experts went to the CPU, the remaining CPU work increased from 9.42 ms to 64.52 ms per token. This work delayed layer completion. A measurement of GPU kernel time alone would show CPU placement as an improvement.

Both the first and third rows of the table run on the GPU. So the gap between 513.95 ms and 148.87 ms is a difference in cache condition. Between the CPU placement row (529.52 to 573.73 ms) and the third row, both placement and cache condition change. Thus you cannot read that gap as an effect of placement.

A placement comparison must match cache capacity and bytes read. The controlled comparison in Section 4.3 does this.

The hit rate also cannot replace the outcome metric. In a cache policy comparison on the Alpaca dataset, some policies cached only gate tensors, or only gate and up tensors. These policies gave a higher VRAM hit rate than full-tensor caching, because each expert uses less space.

But repeated misses and extra I/O for the uncached tensors lowered their throughput. Full-tensor caching had the highest throughput, 2.78 times the no-cache baseline.

The bytes read can also move against the result. Edge0 (arXiv:2609.18063, 2026-09) aims to decrease load wait. It uses a prerouter prediction computed at the previous token as the actual expert selection for the next token.

The test ran an int4 model based on Qwen3.6-35B-A3B with top-4 routing on a MacBook M2 16GB. Reads per step increased by 16%, from 50.9 MiB to 58.9 MiB. But decode speed increased from 3.5 tok/s to 6.4 tok/s. In the same measurement, the sum of per-layer load waits fell from 244.0 ms to 101.9 ms per step.

The sum of load waits adds layer waits that overlap, so it is not the same as the actual stall time. On a Mac mini M4 Pro 24GB in the warm condition, the speed was 19.9 tok/s with the prerouter off. With the prerouter on, it was 20.4 tok/s, a small difference.

The Edge0 prerouter replaces the selection of the original router, so the output can change. The quality comparison in the paper combines int4 quantization, routing replacement, and a recovery LoRA, so it does not isolate the prerouter. Thus Edge0 is not a same-condition comparison with SSD-LLaMA, which keeps the selected experts.

Factors Outside the Boundary

In the cases above, nothing guaranteed that the measured value and the outcome metric would move in the same direction. Factors outside the measured segment set the outcome. The table below shows the measured value, the factor that set the outcome, and the outcome metric for each case.

The type of boundary is different in each case. The fused kernel case is a boundary of time share, and DLFP is a boundary of measurement point, host or device. Graph breaks, routing drift, hit rate, and bytes read are boundaries of intermediate metrics, and CPU placement is a boundary of the measured device.

When the Claim Does Not Hold

If the measured segment is the real bottleneck and the change adds no cost outside it, the local gain carries over end to end. The Amdahl ceiling of 1.072x comes from a time share of 7.87%, and a larger share gives a higher ceiling for the same kernel speedup. Local measurement is still necessary, and with its time share reported, you can calculate the end-to-end ceiling.

The single-GPU result of DLFP is valid under its conditions. To move it to another setup, you need evidence that the call interval also stands in for GPU completion time there. The authors also state that the result must not be generalized by model size or device count.

The strength of the evidence is also limited. The primary DLFP experiment is 3 runs of 100 requests, and the 8B and 32B generalization experiments use 20 or 12 requests per condition. The causal replay for expert substitutability used one model, OLMoE, and SSD-LLaMA was measured only at batch 1.

The launch-bound conclusion is indirect evidence from latency scaling, and the paper reports no direct count of kernel launches. In SSD-LLaMA v1, the speedup ranges in the abstract do not match the per-baseline ranges in Section 5.2. So this post does not quote a combined speedup. Section 4.3 of SSD-LLaMA does not name the exact DeepSeek variant, and Edge0 warns of a ±40% run-to-run spread in single-shot benchmarks.

Some questions remain open in the sources. All five papers measured on a single node. They do not tell whether the same pattern occurs with production traffic or across multiple nodes.

The claim that the seven cases in the table are a recurring pattern in serving optimization in general is a judgment. This post did not find independent reproductions of these results.

Measurement Boundaries to Check

When you read a serving optimization result, check the measurement boundary before the size of the improvement. If a result cannot answer the questions below, it is weak evidence for an end-to-end improvement.

• Does the result report the time share of the improved segment? Use the share and the isolated speedup to calculate the Amdahl ceiling, and compare it with the reported end-to-end value.
• Was the bottleneck measured directly? Find out whether kernel launch wait, CPU work, and transfer wait came from a profiler or from an estimate based on the structure.
• Where is the signal for control or decisions measured? Under asynchronous execution, the host call time can differ from the device completion time.
• Does the result report the metrics that trade off against the main one? For a result that decreases ITL, check TTFT, total latency, SLO passes, goodput, and each confidence interval.
• Was the time measured on one device only? GPU kernel time can fall while CPU work or transfers delay token completion.
• Were the other variables matched between the compared setups? If cache capacity, bytes read, batch size, or synchronous and asynchronous scheduling differ, you cannot assign the difference to one cause.
• Does an intermediate metric stand in for the outcome metric? Hit rate, bytes read, routing agreement, and graph break count have moved against the outcome metric in these cases.
• Was the result measured again after the execution setup changed? A change in model size, GPU count, tensor parallelism, interconnect, or memory capacity can remove the improvement.

For your own experiments, put the same items in the measurement plan. The procedure to set the throughput that meets a latency target is in Sizing and Verifying Acceptable Throughput. The TTFT and inter-token latency metrics are in Transformer Architecture and LLM Serving Optimization. The measurement order for KV cache techniques is in LLM Serving Seen Through the KV Cache.

References

• Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models, arXiv:2608.26612 (2026-08)
• Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits, arXiv:2609.38386 (2026-09)
• Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving, arXiv:2609.07883 (2026-09)
• SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC, arXiv:2609.18110 (2026-09)
• The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction, arXiv:2609.18063 (2026-09)

All numbers were checked against each original paper as of 2026-10. DLFP, SLOWeave, and SSD-LLaMA were checked at v1, and the Launch-Bound paper and Edge0 at v2.

Summary

A serving improvement holds only inside the boundary where it was measured. The fused RMSNorm kernel was 5.62 to 8.98 times faster in an isolated benchmark. But the operations of the two fused kernels covered only 7.87% of the forward pass, so the end-to-end ceiling was 1.072x.

The 27.7% decrease in P99 ITL for DLFP came from a single-GPU condition. In setups where the authors' analysis says the call interval no longer correlated with GPU completion time, SLO passes fell or stayed the same. This analysis is not based on a direct measurement of device completion time.

In SSD-LLaMA, CPU placement decreased GPU kernel time, but the added CPU work increased the time per token. The hit rate in SSD-LLaMA and the bytes read in Edge0 also moved in a different direction from throughput.

When you read a result, check the measurement boundaries in the checklist above before the size of the improvement. The evidence comes from small samples, single models, and batch-1 experiments, so measure again with the same boundaries before you apply this pattern elsewhere.