Sizing and Verifying Acceptable Throughput

Acceptable throughput is not the peak a server survives. It is the arrival rate the service sustains while holding its latency target.

Why a single maximum-throughput number is dangerous

A load test report that says nothing but "maximum 50,000 TPS" cannot be used in operations. That number is usually measured at a point where latency has already collapsed. The saturation point and the limit you can actually run at are different numbers.

The moment the arrival rate exceeds throughput, the excess accumulates in a queue every second. Feeding 15,000 requests per second into a service that completes 10,000 leaves 5,000 behind each second. After sixty seconds that is 300,000 requests. CPU and memory may have room, and the service may still return HTTP 200. Even so, a system whose backlog keeps growing is already in an incident.

A buffer in front, such as Kafka or SQS, hides this risk from view. If a producer writes 20,000 events per second and the consumer processes only 15,000, the application metrics look healthy. Meanwhile lag accumulates at 18 million events per hour. The appearance of health is the dangerous part, because it delays the response.

Throughput therefore has to be measured in layers rather than collapsed into one number. Count what arrived, what started processing, what committed, what failed, what came back as a retry, and what is still waiting. The slope of that waiting backlog moves well before CPU utilization does, which makes it a good first trigger for autoscaling and load shedding.

Defining acceptable throughput

Acceptable throughput is the maximum arrival rate a service can process in steady state while satisfying every stated target. It is a set of constraints, not a single latency condition. A Service Level Objective (SLO) that names only throughput is useless, because a server can meet the number by queueing while the user experience falls apart.

The thresholds below are illustrative and must be replaced with the service targets.

acceptable throughput = max throughput
  subject to:
    p95 latency         <= 100 ms
    p99 latency         <= 300 ms
    error rate          <  0.1 %
    CPU                 <  75 %
    DB pool saturation  <  80 %
    queue lag           <  10 sec

As load rises, tail latency increases sharply before mean latency does. Set the limit just below the throughput where p99 crosses the target. This limit reflects user experience more closely than a limit based on mean latency.

A report becomes operational data only when it states the conditions and the duration together. One example is "sustainable 18,000 TPS, p95 under 80 ms, p99 under 200 ms, errors under 0.01 percent, held for 30 minutes."

Before any number is written down, the unit has to be agreed on. The three common metrics get used interchangeably, but they measure at different layers.

The difference is how many requests make up one transaction. When a user action maps to one request, all three metrics agree. When placing an order calls three APIs for inventory, payment, and notification, 1 TPS becomes 3 RPS. Without agreement on which layer the target refers to, precise measurement still counts different things on each team.

Little's Law and peak conversion

Little's Law relates throughput, mean time in the system, and mean concurrency in steady state. The following calculation uses illustrative assumptions and requires measuring time and concurrency at the same system boundary.

L = λ × W
  L : requests concurrently inside the system (concurrency, in-flight)
  λ : throughput (RPS/TPS)
  W : time one request spends in the system (latency, seconds)
 
e.g.  λ = 500 RPS,  W = 0.2 s   →  L = 100 requests in the system on average
inverted: λ_max = L_max / W
     200 workers, 50 ms service time →  200 / 0.05 = 4,000 TPS per instance

Latency multiplies concurrency. If a slow query doubles W, in-flight count doubles even though throughput is unchanged. What runs out first in that case is not CPU but the thread pool and the connection pool.

The throughput target is set from the peak, not the average, because traffic is not spread evenly across the day. This example assumes that 80 percent of traffic lands in 20 percent of the hours. Replace those proportions with observations from the service.

DAU                        = 1,000,000
requests per user per day  = 50
total daily requests       = 50,000,000
average RPS = 50,000,000 / 86,400s        ≈   580 RPS
 
peak window  = 0.2 × 86,400s              = 17,280s
peak traffic = 0.8 × 50,000,000           = 40,000,000
peak RPS     = 40,000,000 / 17,280s       ≈ 2,315 RPS
 
peak factor ≈ 2,315 / 580 ≈ 4x
momentary spike ×1.5~2 → design target ≈ 3,500~4,600 RPS

Peak factors depend on the service and its event schedule. Estimate them from historical traffic instead of applying fixed industry multipliers. Ticketing and similar synchronized events need separate spike scenarios.

Instance count can also be estimated under explicit assumptions. The 70% target and failure margin below are illustrative choices. A peak of 3,000 RPS at 200 ms mean latency means 600 requests in flight.

If each instance runs 200 workers at 70 percent utilization, effective concurrency is 140. This requires roughly 5 instances. A scenario reserving 2 or 3 additional instances for failures and deployment brings that to 7 or 8.

Adding servers does not raise throughput linearly. Neil Gunther's Universal Scalability Law (USL) models relative throughput as C(N) = N / (1 + α(N−1) + βN(N−1)). Here α is contention from serialized sections and β is the cost of keeping data coherent across nodes.

For β > 0 and 0 ≤ α < 1, the model has a continuous maximum at N* = √((1−α)/β). It predicts declining throughput beyond that point, so the coefficients need to be fitted to measurements. Without fewer locks and less distributed coordination, coherency costs can offset the gain from scaling out.

Setting utilization from a latency target

M/M/1 assumes Poisson arrivals, exponentially distributed service times, and one server. Mean response time is R = S / (1 − ρ), where S is mean service time and ρ is utilization. This steady-state formula applies for ρ < 1 (MIT lecture notes).

These values are calculated from the formula. At 70% utilization, mean response time is already about 3.3 times service time, so latency is not flat. The operating ceiling depends on the latency target.

Assume S = 10 ms and a mean response-time target R ≤ 40 ms. Solving the formula gives ρ ≤ 0.75, but only for this model and mean target. A p99 target or a real system with several workers requires load testing.

Seventy to eighty percent of saturation can be an initial test candidate, but queueing theory does not prescribe it universally. Verify the service-level targets and failure margin before setting the limit. Google SRE also explains why varying request costs make QPS alone a risky capacity metric.

The following table is a hypothetical example for choosing a limit. It is not a measured service result.

With a p95 target of 100 ms, 18K passes and 20K fails in this example. From 18K to 20K, throughput rises about 11.1% while p95 rises about 233%. Set the operating limit after sustaining a passing load and testing failures and retries.

Connection pool size can be a bottleneck alongside CPU. The HikariCP pool-sizing page introduces an estimate proposed for PostgreSQL: connections = (core_count × 2) + effective_spindle_count. Here core_count means physical cores excluding hyperthreading, and effective_spindle_count approaches zero when the data is fully cached.

Four cores and an effective spindle count of one give nine connections. The same page records a case where that many connections served roughly 3,000 front-end users at 6,000 TPS.

Growing the pool can make things slower. The same wiki page cites an Oracle benchmark that changed only the connection pool size. Shrinking the pool reduced response time from about 100 ms to about 2 ms. More connections mean more context switching and lock contention inside the database. The principle is to keep a small number of connections saturated and let the remaining application threads wait in a queue.

Two more checks apply when sizing a pool. The minimum that avoids deadlock is pool size = Tn × (Cm − 1) + 1. Tn is the number of concurrent threads. Cm is the number of connections one thread needs at once. That value is a floor, not an optimum.

And because the database connection limit is a global resource, it has to be divided by instance count. Ten instances each holding a pool of 20 push 200 connections onto the database.

Thread pools are sized by inverting Little's Law. A target of 2,000 TPS at 40 ms of service time per request needs 80 threads, and applying an illustrative 70 percent utilization target brings that to about 115.

What load testing misses

Paper arithmetic misses bottlenecks, so the number produced by sizing has to be verified by measurement. The shape of the load combines three patterns. A ramp-up raises load gradually to find the saturation point where throughput stops rising and latency climbs alone.

A spike test steps load up abruptly to observe how fast autoscaling and the rate limiter react. A soak test holds the target load for hours to catch connections, heap, and file descriptors leaking slowly.

The classic trap that makes a load test report better numbers than reality is Coordinated Omission, named by Gil Tene. A load generator that sends the next request only after the previous one finishes uses a closed model. During a five-second server stall it omits every request that should have been sent.

Only one bad latency sample is recorded, and p99 comes out far better than it is. When production breaks while the test p99 looks fine, suspect this first.

The fix is an open model that continues sending requests at a fixed rate. Gil Tene's wrk2 pioneered this approach, and Vegeta and autocannon adopted it. k6 and Gatling both offer arrival-rate executors. Define load by arrival rate, rather than a fixed virtual user count, to avoid Coordinated Omission.

Three layers for shedding the excess

Once the limit is set, the excess needs a layer that rejects it, delays it, or buffers it. The first layer is rate limiting.

In a distributed setup the standard implementation uses atomic counters in Redis. The second layer is backpressure and load shedding. Backpressure propagates a signal upstream to slow down when the downstream cannot keep up. Load shedding rejects requests quickly when they cannot be served, protecting both the service and everything behind it.

Netflix's concurrency-limits exists because a fixed RPS limit goes stale quickly in an autoscaled environment. It detects queueing through latency and adapts the limit accordingly. Prioritized load shedding sits on top, dropping non-essential traffic such as logging and prefetch before essential traffic such as playback and payment. The goal is to hold latency for core functionality even while overloaded.

The third layer is a waiting room. When momentary traffic runs tens of times above the limit, serializing entry cuts the load reaching downstream into a bounded stream. The Pepero Day promotion published on the Woowa Brothers tech blog closed its 1,111-coupon giveaway in 1.89 seconds and absorbed 100 times normal traffic. Five Node.js instances plus Redis and AWS SQS buffered the participation records, and coupon issuance ran asynchronously in workers.

The same case also shows the consistency problem replication lag can cause. Participants arrived faster than the master counter replicated to the replica, so more than 1,111 winners were recorded. High-speed counting has to be performed atomically at a single master, and the decision has to read from the master as well.

A case on the Toss tech blog follows the same shape. Their point-issuing API takes hundreds of thousands of requests per second at peak, and Redis Increment manages the first-come cap. A distributed lock prevents duplicate issuance, and asynchronous Kafka inserts with consumer throttling hold down database load.

They also merged three APIs into one and cut peak traffic volume itself by 50 percent. Reducing the request count is a valid response to excess load.

Autoscaling does not replace these layers. Trigger scaling before the knee, where latency starts rising sharply. This leaves capacity for the tens of seconds to minutes needed for booting and warm-up. Waiting for 100 percent utilization leaves no such margin. Required headroom depends on traffic variation, startup time, and the number of simultaneous failures to tolerate.

Above all, stateful downstreams such as databases and external payment providers do not grow on demand. Scaling only the front-end web tier reproduces the bottleneck one layer down.

flowchart LR
    A["Size<br/>Little's Law, DAU conversion"] --> B[Verify<br/>find saturation by load test]
    B --> C[Limit<br/>Load passing latency and error targets]
    C --> D[Target<br/>N TPS within p99 X ms]
    D --> E[Operate<br/>Scale before target violations, test failure margin]
    E --> F[Excess<br/>rate limit, backpressure, shedding, waiting room]

The operating checklist comes down to the following.

• Agree first on what counts as one unit. Without pinning it down, each team counts something different.
• Write the target as throughput bound to latency and error-rate conditions, never throughput alone.
• Derive the peak by multiplying the average by a factor, but size event-driven services from a separate spike scenario.
• Run the load generator with an arrival-rate executor to avoid Coordinated Omission.
• Set the limit just below the knee rather than at saturation, and confirm it holds for 30 minutes or more.
• Trigger autoscaling on pre-knee indicators and backlog slope instead of CPU saturation.
• Bind the load reaching stateful downstreams within the ceiling using rate limiting and a waiting room.

Summary

Acceptable throughput is the arrival rate sustained while latency and error-rate targets hold. Estimate concurrency with Little's Law, then verify the operating limit using arrival-rate load tests and failure scenarios. A fixed fraction of saturation is a test candidate and cannot replace the service's latency target. Observe CPU, pool usage, and backlog together, and control excess requests with rate limiting, backpressure, and load shedding.