Elasticsearch Query DSL and Aggregations

The moment you separate queries that compute a score from queries that only filter, the response time of the same search changes.

Two kinds of query

Elasticsearch expresses search and aggregation together through Query DSL, a JSON-based query language (domain specific language). A single request body holds the search conditions, sorting, pagination, and aggregations. Queries divide by how they are processed into structured queries and full-text queries.

A structured query decides conditions answerable with yes or no, such as an exact value match or a range. A full-text query splits text into tokens with an analyzer and computes how relevant a document is to the search terms. The two branches exist as individual leaf queries and are then combined inside a compound query such as bool.

The queries in the table are usable on their own, but real application logic mostly wraps them in bool to build compound conditions. Which clause each query lands in is what governs response time.

Query context and filter context

The same condition costs differently depending on the context it runs in. Query context answers "how relevant is this document to the search intent?" with a number. Elasticsearch runs the BM25 (Best Matching 25) algorithm here to assign a relevance score and sorts in descending order.

BM25 is a refinement of TF-IDF (term frequency-inverse document frequency). It accounts for term frequency, document length, and how rare the term is across the index. Filter context, by contrast, skips the score computation and decides only whether a document matches. The result is stored in a bitset cache, an in-memory structure on the node, and reused quickly when the same condition repeats.

The base rule is to put exact conditions in filter and relevance to search intent in query. Sending conditions that need no score, such as a category classification or a date range, to filter is where performance optimization starts.

Assembling conditions with a bool query

A bool query builds a search tree from four clauses. must requires a match and contributes to the score, should adds points when it matches. must_not forcibly excludes from the results, and filter requires a match while ignoring the score.

A request body over a products index that runs full-text search on the name while filtering exactly on stock and price looks like this.

{
  "query": {
    "bool": {
      "must":   [{ "match": { "name": "wireless earbuds" } }],
      "filter": [
        { "term":  { "in_stock": true } },
        { "range": { "price": { "lte": 200000 } } }
      ]
    }
  }
}

name matters for relevance, so it goes in must and receives a score. Stock status and the price ceiling have nothing to do with the score, so they go to filter and gain the cache benefit. That split sends the request down two paths: one that scores and one that only filters.

The diagram shows how one request splits into the two contexts and then feeds into aggregation. The clauses that need a score and the clauses that only filter separate, and the sorted document set enters the aggregation stage.

Three categories of aggregation

An aggregation recomputes and summarizes the documents that search found. It is therefore heavily affected by field type and cardinality (the number of distinct values). Aggregations fall into three categories by how they produce their output.

Metric aggregations summarize the whole set into a scalar such as an average or a sum. Bucket aggregations classify documents by category frequency or time interval. Pipeline aggregations take a bucket path produced by another aggregation rather than the original documents as input, and compute things like moving averages or rates of change.

An aggregation that groups the last seven days of revenue by day and sums it is written as follows.

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [{ "range": { "@timestamp": { "gte": "now-7d" } } }]
    }
  },
  "aggs": {
    "daily": {
      "date_histogram": { "field": "@timestamp", "calendar_interval": "day" },
      "aggs": {
        "revenue": { "sum": { "field": "amount" } }
      }
    }
  }
}

Setting size to 0 returns only the aggregation results without fetching document bodies. The outer date_histogram creates the date buckets, and the inner sum adds up revenue within each. Note that applying a terms aggregation broadly over a high-cardinality field explodes the bucket count and the cost with it. Estimate the number of distinct values in the target field first.

Handling deep pagination

The simplest way to page through search results is from and size. As pages get deeper, however, each shard has to collect and sort from + size documents, so cost accumulates. By default a request is rejected once from + size exceeds index.max_result_window (default 10,000).

search_after is the better fit for deep pagination. It passes the sort values of the last document on the previous page as a cursor for the next request, reading onward instead of skipping an offset. The sort must include a unique field to break ties, or results will be dropped.

{
  "size": 20,
  "query": { "match_all": {} },
  "sort": [
    { "created_at": "desc" },
    { "_id": "asc" }
  ],
  "search_after": [1719700000000, "doc-8842"]
}

The values in the search_after array must match the sort values of the previous page's last document in order and type. For exporting an entire dataset in one sweep, scroll, which holds a snapshot, was the older approach. Today the common alternative is to pin a moment with Point In Time (PIT) and iterate with search_after.

Summary

The first axis to get right in Query DSL is the split between queries that need a score and queries that only filter. Sending conditions that need no score to filter speeds up repeated searches thanks to the bitset cache.

Aggregations recompute over search results, which makes them sensitive to field type and cardinality, so estimate how far the bucket count will grow in advance. Pagination is fine with from and size while it stays shallow, but search_after costs less once it gets deep. Separating just these three concerns keeps search quality and response time in hand together.