Sizing and Verifying Acceptable Throughput

Acceptable throughput is not the peak a server survives. It is the arrival rate the service sustains while holding its latency target.

Why a single maximum-throughput number is dangerous

A load test report that says nothing but "maximum 50,000 TPS" cannot be used in operations. That number is usually measured at a point where latency has already collapsed. The saturation point and the limit you can actually run at are different numbers.

The moment the arrival rate exceeds throughput, the excess accumulates in a queue every second. Feeding 15,000 requests per second into a service that completes 10,000 leaves 5,000 behind each second. After sixty seconds that is 300,000 requests. CPU and memory may have room and HTTP 200 may still be returned, but a system whose backlog keeps growing is already in an incident.

A buffer in front, such as Kafka or SQS, hides this risk from view. If a producer writes 20,000 events per second and the consumer processes only 15,000, the application metrics look healthy. Meanwhile lag accumulates at 18 million events per hour. The appearance of health is the dangerous part, because it delays the response.

Throughput therefore has to be measured in layers rather than collapsed into one number. Count what arrived, what started processing, what committed, what failed, what came back as a retry, and what is still waiting. The slope of that waiting backlog moves well before CPU utilization does, which makes it a good first trigger for autoscaling and load shedding.

Defining acceptable throughput

Acceptable throughput is the maximum arrival rate a service can process in steady state while satisfying every stated target. It is a set of constraints, not a single latency condition. A Service Level Objective (SLO) that names only throughput is useless, because a server can meet the number by queueing while the user experience falls apart.

acceptable throughput = max throughput
  subject to:
    p95 latency         <= 100 ms
    p99 latency         <= 300 ms
    error rate          <  0.1 %
    CPU                 <  75 %
    DB pool saturation  <  80 %
    queue lag           <  10 sec

As load rises, mean latency degrades gently while the tail explodes first. The limit therefore has to be drawn at the throughput just below the point where p99 crosses the target, not where the mean does. That is the line that matches what users feel. A report becomes operational data only when it states the conditions and the duration together. One example is "sustainable 18,000 TPS, p95 under 80 ms, p99 under 200 ms, errors under 0.01 percent, held for 30 minutes."

Before any number is written down, the unit has to be agreed on. The three common metrics get used interchangeably, but they measure at different layers.

The difference is how many requests make up one transaction. When a user action maps to one request, all three metrics agree. When placing an order calls three APIs for inventory, payment, and notification, 1 TPS becomes 3 RPS. Without agreement on which layer the target refers to, precise measurement still counts different things on each team.

Little's Law and peak conversion

Throughput, latency, and concurrency are tied together by one result from queueing theory, Little's Law.

L = λ × W
  L : requests concurrently inside the system (concurrency, in-flight)
  λ : throughput (RPS/TPS)
  W : time one request spends in the system (latency, seconds)
 
e.g.  λ = 500 RPS,  W = 0.2 s   →  L = 100 requests always in flight
inverted: λ_max = L_max / W
     200 workers, 50 ms service time →  200 / 0.05 = 4,000 TPS per instance

Latency multiplies concurrency. If a slow query doubles W, in-flight count doubles even though throughput is unchanged. What runs out first in that case is not CPU but the thread pool and the connection pool.

The throughput target is set from the peak, not the average, because traffic is not spread evenly across the day. The common shortcut is a Pareto assumption that 80 percent of traffic lands in 20 percent of the hours.

DAU                        = 1,000,000
requests per user per day  = 50
total daily requests       = 50,000,000
average RPS = 50,000,000 / 86,400s        ≈   580 RPS
 
peak window  = 0.2 × 86,400s              = 17,280s
peak traffic = 0.8 × 50,000,000           = 40,000,000
peak RPS     = 40,000,000 / 17,280s       ≈ 2,315 RPS
 
peak factor ≈ 2,315 / 580 ≈ 4x
momentary spike ×1.5~2 → design target ≈ 3,500~4,600 RPS

The peak factor varies widely by the nature of the service. Social and messaging products run 2 to 5 times the average, while event-driven traffic such as ticketing or first-come-first-served promotions reaches 20 to 50 times. Both ranges are industry rules of thumb, so they have to be re-measured against your own historical traffic. For event-driven services, a separate spike scenario is safer than multiplying an average by a factor.

Instance count falls out of Little's Law. A peak of 3,000 RPS at 200 ms mean latency means 600 requests in flight. If each instance runs 200 workers filled only to 70 percent, effective concurrency is 140, so roughly 5 instances are needed. Adding room for failures and deployments brings that to 7 or 8.

Adding servers does not raise throughput linearly. Neil Gunther's Universal Scalability Law (USL) models relative throughput as C(N) = N / (1 + α(N−1) + βN(N−1)). Here α is contention from serialized sections and β is the cost of keeping data coherent across nodes. The β term is why real systems lose throughput past a certain point, with the peak at N* = √((1−α)/β). Scaling out without reducing locks and distributed coordination lets the coherency cost eat the gain.

Why 70 percent of saturation is the limit

In an M/M/1 queue, response time R grows against utilization ρ as R = S / (1 − ρ), where S is service time. The formula says that latency rises nonlinearly, not linearly, as utilization climbs.

Latency stays nearly flat up to 70 percent, then the queue grows far faster than intuition suggests once utilization passes 80 to 85 percent. Because of that knee, operations targets 60 to 70 percent utilization in normal conditions. The margin left over is what absorbs spikes, failovers, and retry storms. The conclusion from queueing theory is to set acceptable throughput at 0.7 to 0.8 times the saturated throughput.

The knee shows up in measurements as well. The table below is a load test result from one service.

Calling 22K the maximum throughput here would be wrong. Between 18K and 20K, throughput rose 10 percent while p95 jumped 233 percent, from 60 ms to 200 ms. The practical limit is 18K to 20K, and above it queueing, lock contention, connection waits, and retries amplify one another.

What hits the limit first is usually not CPU but pool size. The HikariCP wiki page on pool sizing gives the PostgreSQL guidance formula connections = (core_count × 2) + effective_spindle_count. Here core_count means physical cores excluding hyperthreading, and effective_spindle_count approaches zero when the data is fully cached. Four cores with one HDD gives 9 to 10 connections. The same page records a case where that many connections served roughly 3,000 front-end users at 6,000 TPS.

Growing the pool can make things slower. An Oracle benchmark cited on the same wiki page shows response time dropping from about 100 ms to about 2 ms after nothing was changed except shrinking the connection pool. More connections mean more context switching and lock contention inside the database. The principle is to keep a small number of connections saturated and let the remaining application threads wait in a queue.

Two more checks apply when sizing a pool. The minimum that avoids deadlock is pool size = Tn × (Cm − 1) + 1, where Tn is the number of concurrent threads and Cm is the number of connections one thread needs at once. That value is a floor, not an optimum. And because the database connection limit is a global resource, it has to be divided by instance count. Ten instances each holding a pool of 20 push 200 connections onto the database.

Thread pools are sized by inverting Little's Law. A target of 2,000 TPS at 40 ms of service time per request needs 80 threads, and applying the 70 percent utilization ceiling brings that to about 115.

What load testing misses

Paper arithmetic misses bottlenecks, so the number produced by sizing has to be verified by measurement. The shape of the load combines three patterns. A ramp-up raises load gradually to find the saturation point where throughput stops rising and latency climbs alone. A spike test steps load up abruptly to observe how fast autoscaling and the rate limiter react. A soak test holds the target load for hours to catch connections, heap, and file descriptors leaking slowly.

The classic trap that makes a load test report better numbers than reality is Coordinated Omission, named by Gil Tene. A load generator that sends the next request only after the previous one finishes uses a closed model. During a five-second server stall it omits every request that should have been sent. Only one bad latency sample is recorded, and p99 comes out far better than it is. When production breaks while the test p99 looks fine, suspect this first.

The fix is an open model that keeps firing requests at a fixed rate. Gil Tene's wrk2 pioneered this approach, and Vegeta and autocannon adopted it. k6 and Gatling both offer arrival-rate executors, so defining load by arrival rate rather than by a fixed virtual user count is what avoids Coordinated Omission.

Three layers for shedding the excess

Once the limit is set, the excess needs a layer that rejects it, delays it, or buffers it. The first layer is rate limiting.

In a distributed setup the standard implementation uses atomic counters in Redis. The second layer is backpressure and load shedding. Backpressure propagates a signal upstream to slow down when the downstream cannot keep up. Load shedding rejects requests quickly when they cannot be served, protecting both the service and everything behind it.

Netflix's concurrency-limits exists because a fixed RPS limit goes stale quickly in an autoscaled environment. It detects queueing through latency and adapts the limit accordingly. Prioritized load shedding sits on top, dropping non-essential traffic such as logging and prefetch before essential traffic such as playback and payment. The goal is to hold latency for core functionality even while overloaded.

The third layer is a waiting room. When momentary traffic runs tens of times above the limit, serializing entry cuts the load reaching downstream into a bounded stream. The Pepero Day promotion published on the Woowa Brothers tech blog closed its 1,111-coupon giveaway in 1.89 seconds and absorbed 100 times normal traffic. Five Node.js instances plus Redis and AWS SQS buffered the participation records, and coupon issuance ran asynchronously in workers.

The same case also shows the consistency problem replication lag can cause. Participants arrived faster than the master counter replicated to the replica, so more than 1,111 winners were recorded. High-speed counting has to be performed atomically at a single master, and the decision has to read from the master as well.

A case on the Toss tech blog follows the same shape. Their point-issuing API takes hundreds of thousands of requests per second at peak, and Redis Increment manages the first-come cap. A distributed lock prevents duplicate issuance, and asynchronous Kafka inserts with consumer throttling hold down database load. They also merged three APIs into one and cut peak traffic volume itself by 50 percent. Reducing the request count is a valid response to excess load.

Autoscaling does not replace these layers. The scale trigger has to fire before the knee rather than at 100 percent utilization, so that margin remains during the tens of seconds to minutes that booting and warm-up take. Headroom is conventionally set at 1.5 to 3 times the forecast peak, and higher when traffic is more volatile. Above all, stateful downstreams such as databases and external payment providers do not grow on demand. Scaling only the front-end web tier reproduces the bottleneck one layer down.

The operating checklist comes down to the following.

• Agree first on what counts as one unit. Without pinning it down, each team counts something different.
• Write the target as throughput bound to latency and error-rate conditions, never throughput alone.
• Derive the peak by multiplying the average by a factor, but size event-driven services from a separate spike scenario.
• Run the load generator with an arrival-rate executor to avoid Coordinated Omission.
• Set the limit just below the knee rather than at saturation, and confirm it holds for 30 minutes or more.
• Trigger autoscaling on pre-knee indicators and backlog slope instead of CPU saturation.
• Bind the load reaching stateful downstreams within the ceiling using rate limiting and a waiting room.

Summary

Acceptable throughput is not the peak just before a server dies. It is the arrival rate that is sustained while latency and error-rate targets hold. Sizing starts from Little's Law and converts to a peak with a peak factor. Verification finds the saturation point with an arrival-rate load test, and 70 to 80 percent of it becomes the limit. Bottlenecks usually appear in pool size before CPU, so connection pools start near (core × 2) + spindles and get tuned by measurement.

Excess traffic is shed through rate limiting, backpressure, load shedding, and a waiting room, with the autoscale trigger set before the knee. Finally, backlog slope belongs in the monitoring set, because that is the signal that catches an incident accumulating quietly behind a queue.