Splitting an index into shards supports distribution. Replicating the shards supports availability.
Why an index is split into shards
In Elasticsearch, an index is a logical collection of documents, corresponding to a table in a relational database. Once data grows to terabyte (TB) or petabyte (PB) scale, it exceeds the storage and I/O bandwidth of a single server. An index is therefore split into independent pieces called shards and distributed across the cluster.
A shard is both the basic unit of data distribution and a self-contained Apache Lucene index instance. Physically, that Lucene index is itself made up of multiple segments. Every time documents are indexed a new segment is written to disk, and a background merge consolidates small segments into larger ones.
An internal integer identifier limit restricts a Lucene index to 2,147,483,519 documents. Capacity planning therefore starts by distributing documents across shards.
Primary shards and replica shards
Elasticsearch creates replica shards to preserve availability during node failures. Each replica copies a primary shard and resides on a different physical node. If the primary's node fails, a replica on another node becomes primary.
A replica is both a failover copy and a share of the read load. Read requests such as searches are served in parallel by primaries and replicas. Adding replicas therefore increases concurrent search throughput as well.
| Property | Primary shard | Replica shard |
|---|---|---|
| Responsibility | Stores the original data | Copy, spreads read load |
| Changing the count | Fixed once the index is created | Adjustable at runtime |
| Writes | Performed directly | Replicated from the primary |
| Placement rule | Distributed across nodes | On a different node from its primary |
The replica count can change during operation to match read load. The primary count is fixed when the index is created. The routing mechanism described next explains this difference.
How a document finds its shard
A routing formula determines the destination shard for a document. The formula connects the primary count with the hash function.
shard = hash(routing) % number_of_primary_shardsThe routing value defaults to the document's _id. Hashing it and taking the remainder modulo the primary count gives the destination shard. Because the modulo depends directly on the primary count, changing that count would send existing documents to the wrong destination. That is why the primary count cannot be changed after creation.
Search works in the opposite direction. A coordinating node scatters the query to the relevant shards and gathers the partial results from each into the final response.
flowchart LR
C[Client<br/>search request] --> CO[Coordinating node<br/>handles scatter and gather]
CO -->|Scatter| S0[Shard 0<br/>primary]
CO -->|Scatter| S1[Shard 1<br/>replica]
CO -->|Scatter| S2[Shard 2<br/>primary]
S0 -->|Gather| CO
S1 -->|Gather| CO
S2 -->|Gather| CO
CO --> R[Final response<br/>sorted and merged results]The diagram shows a search request sent to several shards, with results combined on one node. Indexing selects one shard through routing, while search collects results from multiple shards. Shard design therefore affects reads and writes differently.
The order in which writes are replicated
Writes first go through the primary. The coordinating node uses routing to choose that primary, which indexes the document in its own shard. It then copies the data to replicas. After replication is acknowledged, it reports success to the client.
flowchart LR
W[Client<br/>index request] --> CO[Coordinating node<br/>picks destination by routing]
CO --> P[Primary shard<br/>indexes locally first]
P -->|replicate| R1[Replica shard 1]
P -->|replicate| R2[Replica shard 2]
R1 -->|ack| P
R2 -->|ack| P
P -->|success response| WThe diagram traces the path a write takes from primary to replicas (standard Elasticsearch replication behavior). More replicas therefore means a single write propagates to more nodes and costs more. Read throughput and write cost are coupled through the replica count.
Separating node roles
A cluster is a group of nodes bound into one logical system. Each node does different work depending on the roles assigned to it.
| Node role | Responsibility |
|---|---|
| Master node | Cluster state and metadata, shard allocation and relocation |
| Data node | Stores documents and inverted indexes, handles search, aggregation, and indexing |
| Ingest node | Transforms, enriches, and filters documents just before indexing |
| Coordinating node | Distributes requests and merges responses, acting as a load balancer |
The master node controls shard allocation and cluster state. Isolating it from data I/O improves stability. Data nodes handle CPU- and memory-intensive searches and aggregations. Small environments can combine roles on one node. In production, separating master and data workloads improves resilience.
Choosing the number of shards
Too many shards is a problem, and so is too few. Too few and a single shard grows enormous; too many and management overhead balloons.
Elastic's general guidance is 10-50GB and fewer than 200 million documents per shard, with workload-dependent exceptions (Elastic shard sizing guide). Dividing 300GB evenly among 10 primaries yields about 30GB per shard. Actual placement also depends on node count, routing, and replica settings.
That decision is expressed as two settings at index creation time. number_of_shards is the primary count and cannot be changed afterward; number_of_replicas can be updated at runtime.
The following example uses the Elasticsearch Console HTTP request format.
PUT /logs-2026-06
{
"settings": {
"number_of_shards": 10,
"number_of_replicas": 1
}
}Setting replicas to 1 adds 10 replicas to the 10 primaries for 20 shards in total. The replica value can be raised or lowered later with PUT /logs-2026-06/_settings (standard Elasticsearch index settings).
Shard explosion means that too many shards have accumulated. Creating daily indexes for small datasets without cleanup can leave hundreds of unnecessary shards. Each shard is a Lucene instance that consumes CPU, JVM (Java Virtual Machine) heap, and disk I/O. This slows cluster state updates and node restarts.
ILM (Index Lifecycle Management) manages continuously growing data such as time-series logs. Indexes roll over at a specified size or age. Older indexes move through hot, warm, and cold tiers to reduce cost. This limits the growth of any one index's shards.
Rebalancing and failure behavior
Adding shards or losing a node causes shard relocation. This rebalancing moves data between nodes and uses substantial network and disk capacity. Excessive rebalancing during recovery can sharply reduce performance.
Failure behavior follows directly from the design above. When a node dies and a primary disappears, a replica on another node is promoted to primary. An unstable master, on the other hand, destabilizes shard allocation itself and escalates into a cluster-wide operational problem.
Shard and replica design therefore has to account for recovery cost, not just steady-state performance. With no replicas, losing a primary means losing data. With too many replicas, write and storage costs rise and there is more data to move during recovery.
Summary
Elasticsearch divides indexes into shards and replicates those shards. The primary count is fixed at index creation because routing depends on it. The replica count can change with read load. Separating node roles isolates the master's control work from data I/O.
The operational baseline is a shard size of 30-50GB. Avoid shard explosion and excessive rebalancing to maintain cluster stability.