Elasticsearch Query DSL and Aggregations

Separating queries that need relevance scores from queries that only filter affects search response time.

Two kinds of query

Elasticsearch expresses search and aggregation together through Query DSL, a JSON-based query language (domain specific language). A single request body holds the search conditions, sorting, pagination, and aggregations. Queries divide by how they are processed into structured queries and full-text queries.

A structured query decides conditions answerable with yes or no, such as an exact value match or a range. A full-text query splits text into tokens with an analyzer and computes how relevant a document is to the search terms. The two branches exist as individual leaf queries and are then combined inside a compound query such as bool.

The table's queries can run independently. Applications usually combine them with bool to express multiple conditions. Clause placement affects response time.

Query context and filter context

The same condition costs differently depending on the context it runs in. Query context answers "how relevant is this document to the search intent?" with a number. Elasticsearch runs the BM25 (Best Matching 25) algorithm here to assign a relevance score and sorts in descending order.

BM25 is a refinement of TF-IDF (term frequency-inverse document frequency). It accounts for term frequency, document length, and how rare the term is across the index. Filter context, by contrast, skips the score computation and decides only whether a document matches. The result is stored in a bitset cache, an in-memory structure on the node, and reused quickly when the same condition repeats.

Put exact conditions in filter and search relevance conditions in query. Categories and date ranges usually need no score. Moving these conditions to filter is a first step in reducing query cost.

Assembling conditions with a bool query

A bool query builds a search tree from four clauses. must requires a match and contributes to the score, should adds points when it matches. must_not forcibly excludes from the results, and filter requires a match while ignoring the score.

A request body over a products index that runs full-text search on the name while filtering exactly on stock and price looks like this.

{
  "query": {
    "bool": {
      "must":   [{ "match": { "name": "wireless earbuds" } }],
      "filter": [
        { "term":  { "in_stock": true } },
        { "range": { "price": { "lte": 200000 } } }
      ]
    }
  }
}

name affects relevance, so place it in must to receive a score. Stock status and the price limit need no score, so place them in filter to use the cache. The request then separates scoring work from filtering work.

flowchart TD
    R["search request<br/>JSON Query DSL"] --> B["bool query<br/>assembles compound conditions"]
    B --> F["filter clause<br/>match decision without scoring, bitset cache"]
    B --> Q["must / should clauses<br/>BM25 relevance scoring"]
    F --> H["document set<br/>sorted by score"]
    Q --> H
    H --> A["aggs<br/>recomputes bucket and metric aggregations"]

The diagram shows how one request splits into the two contexts and then feeds into aggregation. The clauses that need a score and the clauses that only filter separate, and the sorted document set enters the aggregation stage.

Three categories of aggregation

An aggregation recomputes and summarizes the documents that search found. It is therefore heavily affected by field type and cardinality (the number of distinct values). Aggregations fall into three categories by how they produce their output.

Metric aggregations summarize documents as scalar values such as averages or sums. Bucket aggregations group documents by category frequency or time interval. Pipeline aggregations use a bucket path from another aggregation to calculate moving averages or rates of change. They do not take the original documents as input.

An aggregation that groups the last seven days of revenue by day and sums it is written as follows.

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [{ "range": { "@timestamp": { "gte": "now-7d" } } }]
    }
  },
  "aggs": {
    "daily": {
      "date_histogram": { "field": "@timestamp", "calendar_interval": "day" },
      "aggs": {
        "revenue": { "sum": { "field": "amount" } }
      }
    }
  }
}

Setting size to 0 returns only the aggregation results without fetching document bodies. The outer date_histogram creates the date buckets, and the inner sum adds up revenue within each. Note that applying a terms aggregation broadly over a high-cardinality field explodes the bucket count and the cost with it. Estimate the number of distinct values in the target field first.

Handling deep pagination

The simplest way to page through search results is from and size. As pages get deeper, however, each shard has to collect and sort from + size documents, so cost accumulates. By default a request is rejected once from + size exceeds index.max_result_window (default 10,000).

search_after suits deep pagination. The next request uses the last document's sort values from the previous page as its cursor. It continues reading without skipping an offset. Include a unique field in the sort to break ties and avoid missing results.

{
  "size": 20,
  "query": { "match_all": {} },
  "sort": [
    { "created_at": "desc" },
    { "_id": "asc" }
  ],
  "search_after": [1719700000000, "doc-8842"]
}

The values in the search_after array must match the sort values of the previous page's last document in order and type. For exporting an entire dataset in one sweep, scroll, which holds a snapshot, was the older approach. Today the common alternative is to pin a moment with Point In Time (PIT) and iterate with search_after.

Summary

In Query DSL, first separate queries that need relevance scores from those that only filter. Conditions in filter use the bitset cache to accelerate repeated searches.

Aggregation cost depends on field type and cardinality. Estimate the number of buckets before running a query. For shallow pagination, from and size are sufficient, while search_after costs less for deeper pages. Designing query context, aggregation, and pagination separately helps manage search quality and response time.