Designing a Live Streaming System

At a scale of one million concurrent viewers, the starting point of the design is not mixing three kinds of traffic with different characteristics, video, chat, and payments, into one path.

Splitting traffic three ways

Live broadcasting at one million concurrent viewers is not a single load problem. Video delivery, chat, and payments differ in data size, latency tolerance, and consistency requirements. Bundling all three into one path means a surge in one takes the others down with it.

Design judgments need a scale to anchor them. This post assumes the following scale, with numbers as design-exercise estimates.

• 35 million daily active users, 5 million global peak concurrent connections
• Up to 1 million concurrent viewers on a single channel (extreme case)
• Peak delivery bandwidth: several Tbps for a large single channel, tens of Tbps or more at global peak
• Peak chat ingress: hundreds of thousands per second globally, tens of millions of fanout events per second
• Latency targets: seconds for media, tens to hundreds of milliseconds for chat and events

Laying out the character of the three traffic types in one table makes the reason for splitting them clear. The point is that each path has a different constraint it must never violate.

Splitting the paths means a chat burst does not block payments, and media traffic does not kill the API servers.

The video path

This path is the heaviest and tolerates several seconds of latency. It goes through five stages from ingest to playback, with each stage absorbing load progressively.

Ingest and transcoding

The streamer's broadcasting tool sends the source over RTMP (Real-Time Messaging Protocol). RTMP completes a handshake over TCP and then streams. SRT (Secure Reliable Transport), which handles lossy networks well, is also used for ingest. WHIP (WebRTC-HTTP Ingestion Protocol), which uses WebRTC, is another candidate for the same purpose.

Large services in practice place a media proxy at every point of presence worldwide. Whatever the protocol, it is terminated at the PoP and converted to a single canonical protocol inside the backbone. Authentication does not query the database every time; cached stream key validation reduces latency. Stream keys are handled as hashes or kept in a secret store rather than in plaintext.

The single ingested source is expanded into an adaptive bitrate ladder of multiple qualities. Transcoding is the most expensive stage on this path.

Transcoding one 1080p60 stream is generally said to take roughly 4-6 CPU cores (estimate). With tens of thousands of concurrent channels, this cost becomes a major axis of the infrastructure. That is the background to Twitch building its own transcoder instead of using FFmpeg (Twitch engineering blog).

Packaging and latency

Transcoding output is cut into short segments and a manifest (.m3u8) and pushed to a content delivery network (CDN). Standard HLS (HTTP Live Streaming) has 10-30 seconds of latency, which is not enough for live. Low-latency HLS (LL-HLS) brings that down to 2-5 seconds, and three mechanisms do the work (based on published LL-HLS material, ranges are estimates).

• Partial segments: a 6-second segment is cut further into 200-500 millisecond parts, so parts ship before the segment is complete
• Preload hints: the URL of the next part is advertised before it exists, and a request for it is held open until the data is ready
• Delta playlists and blocking reload: only the changed portion is sent, and requests are held until the next version is ready

Using the common container format (CMAF) lets HLS and DASH (Dynamic Adaptive Streaming over HTTP) share the same segments, saving storage and cache space. WebRTC is a candidate when ultra-low latency is required, but the choice diverges as scale grows.

When one million people watch one way, low-latency HLS is the realistic answer. WebRTC-grade ultra-low latency cannot make good use of CDN caches, so costs grow quickly at this scale.

CDN and quality selection

API servers do not return video binaries directly. A playback request returns only the CDN manifest URL, and the player then fetches segments from the edge itself. The edge tier carries the weight of the traffic.

• The edge caches segments for 30-60 seconds so multiple viewers reuse them, balanced against serving stale data after a broadcast ends.
• Concurrent requests converging on the same segment are coalesced into a single request to the origin, absorbing momentary surges.
• This tier absorbs several Tbps for a single large channel and tens of Tbps at global peak.

Quality switching is decided by the client, not the server. Adaptive bitrate algorithms fall into three families. Throughput-based algorithms decide from recent download speed and are strong when the buffer is empty, such as at startup or after a seek. Buffer-based algorithms (BOLA) look only at buffer level and weigh bitrate against rebuffering.

The hybrid of the two is the default in dash.js and is close to the industry standard. A comparison of these three families is laid out in the ACM paper 'Improving Bitrate Adaptation in the DASH Reference Player'.

The chat path

Messages are light but highly ephemeral, and they carry a fanout problem where one message spreads to tens of thousands of people. Handling tens of thousands per second outside a relational database is the axis of this path's design.

Tiered fanout

Twitch chat is a distributed system written in Go, built around two components (Twitch developer documentation). The edge is the termination point clients attach to, handling the Internet Relay Chat (IRC) protocol over both TCP and WebSocket. Pub/sub distributes messages internally among edge nodes.

One publish goes to pub/sub, which forwards only to the relevant edges, and each edge sprays it locally to the viewers attached to it. The structure splits into one publish and N fanouts at the termination points. Discord solves the same problem with the Elixir actor model, making channels and sessions into separate processes that pass messages (Discord engineering blog). Only the language differs; the skeleton of publish once, distribute only where there is interest, deliver locally at the edge is the same.

WebSocket gateways and sharding

For the WebSocket gateway that clients attach to, connection count is the load. A single node ideally tuned can hold hundreds of thousands of idle connections, but in practice 50,000 to 100,000 connections per worker is the usual target (estimate). There are three bottlenecks.

• File descriptors: one consumed per connection
• Memory: accumulates at several KB per connection
• CPU: the cost of processing messages

The load balancer in front pins connections to a specific node with consistent hashing or sticky sessions. Propagation between nodes is handled by a pub/sub backplane such as Redis, Kafka, or NATS. A single huge channel cannot fit on one server, so channel sharding splits viewers across multiple partitions, each delivering only to its own subscribers.

Handling bursts, and the client

When chat explodes to tens or hundreds of thousands per second during a large event, preserving every message at the same quality is not feasible. The priority at that point is protecting video and the basic chat experience. Load is shed through graceful degradation.

• Batching: bundling several messages to lower send frequency
• Throttling: capping the send rate at N per second (for example, GetStream throttles above 5 per second)
• Sampling and dropping: intentionally discarding some messages under overload

Data for moderation, reporting, replay, and analytics is instead stored in a separate asynchronous log pipeline. That pipeline is isolated from the live delivery path (GetStream live stream recommended practices). Another bottleneck remains where the server can send everything but the browser cannot keep up. Three measures keep the document object model (DOM) node count bounded. They are a virtual list that puts only visible messages into the DOM, batching that groups re-renders per frame, and trimming old messages.

The chat body carries only emote IDs, not image binaries. The client renders images from the CDN using metadata it fetched in advance.

{ "sender": "viewer_ko", "message": "lol", "emotes": ["pog_1", "kappa_2"] }

An asset version (asset_version) field invalidates the client cache when images are updated. Keeping the payload light is the most direct way to reduce fanout cost.

Viewer counting and the payment path

Concurrent viewer count and payments are handled on yet another path from chat. For one, availability matters more than accuracy; for the other, the reverse.

Concurrent viewer count must not be counted by updating the same row. The query below causes lock contention and replication lag on one row every time millions of people join and leave.

UPDATE live_streams SET viewer_count = viewer_count + 1 WHERE id = 'stream_123';

Joins and leaves go into a separate event stream instead. Viewers simply close the window, so the system does not rely on explicit leave events but infers them from heartbeats. Aggregation is approximated with Redis or a stream processing engine. A cardinality estimator such as HyperLogLog counts 10 million people in a fixed 12KB per counter with about 0.81% standard error (per the official Redis documentation). An approximation updated every few seconds suits a live screen better than an exact count to the single viewer.

Payments follow the opposite principle. Subscriptions and donations are handled with atomic, durable storage, and payment state is recorded explicitly. Once a payment is confirmed, a separate event is published and the on-screen overlay notification is delivered asynchronously over the WebSocket path. Money is stored exactly and notifications propagate on a best-effort basis, which separates consistency from speed.

Summary

The backbone of live streaming design is the separation principle: video, chat, and payments do not share one path. Video absorbs its weight through transcoding and CDN edges, while chat sheds surges through tiered fanout and graceful degradation. Concurrent viewer count is counted fast and approximately, while payments are stored atomically and exactly, so each satisfies its own constraint. Splitting this way keeps a burst on one path from dragging down the others and lets each tier scale independently.