Late Interaction Reranking

Reranking designs that close the gap between cross-encoder precision and bi-encoder speed by reordering when encoding and interaction happen.

Where reranking fixes both speed and precision

Retrieval-augmented generation (RAG) generally uses two-stage retrieval. A bi-encoder or BM25 retrieves a broad candidate set in the first stage. A reranker then reorders candidates by relevance to the query. Measure retrieval ranking and final answers together to establish the effect of reranking.

In that second stage, the reranker's encoding and interaction scheme determines accuracy and latency at the same time. When the query and the document meet is the central design axis. This article covers ColBERT late interaction and index cost.

flowchart LR
    Q["Query"] --> B["First-stage retrieval<br/>candidate recall"]
    B --> R["Reranker<br/>query-document interaction"]
    R --> O["Reranked documents"]
    O --> L["LLM response"]

Three interaction schemes

Start with the three baseline schemes compared by the point at which interaction occurs. Interaction timing changes what can be precomputed and the work required per query. Accuracy rankings must be measured for the model and data.

The bi-encoder compresses a document into one vector, which loses a lot of information. The cross-encoder has to run a full forward pass for every candidate, which makes it impractical on a large corpus. ColBERT bridges the two by precomputing document token vectors offline and performing only the interaction with query tokens at inference time.

ColBERT's MaxSim and index cost

The core of ColBERT (Khattab and Zaharia, SIGIR 2020) is the MaxSim score. Each query token picks the single most similar token in the whole document, and those similarities are summed.

S(Q, D) = Σ_i  max_j  (Q_i · D_j)
Q_i: i-th query token vector, D_j: j-th document token vector

Each query token contributes its matching score against document tokens. The document side is precomputed, so an approximate nearest neighbor (ANN) index can be used. The cost is a vector stored per token, which makes the index large. Section 5.3 of the ColBERTv2 paper reports 154 GiB for the original ColBERT index on roughly 9 million MS MARCO passages.

ColBERTv2 (Santhanam et al., 2022) reduces that cost two ways. Distillation from a cross-encoder teacher improves quality. Residual compression shrinks storage by encoding each token vector as a centroid index plus a quantized residual. On that same MS MARCO collection, the 1-bit and 2-bit residual indexes occupied 16 and 25 GiB, respectively. These sizes do not transfer directly to other corpus sizes.

PLAID is a follow-up search engine that reduces late-interaction computation through centroid-based candidate selection and pruning. It evaluates collections up to 140 million passages, a different condition from the MS MARCO storage figures above.

An independent SIGIR 2024 reproduction study found that parameter combinations and baselines affect the efficiency and effectiveness trade-off. Reranking BM25 candidates with ColBERTv2 was competitive at low latency. PLAID is therefore not always the more efficient choice.

Evaluation After Reranking

Measure retrieval ranking and answer quality separately. Lost in the Middle shows that evidence position affected answers for the evaluated models and tasks. It does not establish a particular reranker's superiority or a universal input order.

Compare retrieval ranks and final answers after changing only the order of the same candidates. Hold candidate count, document length, model version, and hardware fixed, and record latency and index size. Evaluate queries outside the training distribution separately.

Summary

ColBERT retains document-token vectors and matches them with query-time MaxSim. ColBERTv2 residual compression and PLAID candidate selection address separate storage and search stages. Do not infer an accuracy ranking from architecture alone; compare retrieval quality, answers, and cost under the same data and conditions.