Late Interaction Reranking

Reranking designs that close the gap between cross-encoder precision and bi-encoder speed by reordering when encoding and interaction happen.

Where reranking fixes both speed and precision

Retrieval-augmented generation (RAG) generally uses two-stage retrieval. A bi-encoder or BM25 casts a wide net for candidates in the first stage, and a reranker reorders them by relevance to the query in the second. The reranker is needed because of the 'lost in the middle' effect, where an LLM misses information in the middle of its context. Keeping the most relevant documents near the front is what preserves answer quality.

In that second stage, the reranker's encoding and interaction scheme determines accuracy and latency at the same time. When the query and the document meet is the central design axis. This article covers the three lines of work that followed the cross-encoder. Those are ColBERT's late interaction, the LBNL scheme in jina-reranker-v3, and position bias in listwise LLM reranking.

Three interaction schemes

Start with the three baseline schemes compared by the point at which interaction occurs. The earlier the query and document meet, the more precise and the slower the result.

The bi-encoder compresses a document into one vector, which loses a lot of information. The cross-encoder has to run a full forward pass for every candidate, which makes it impractical on a large corpus. ColBERT bridges the two by precomputing document token vectors offline and performing only the interaction with query tokens at inference time.

ColBERT's MaxSim and index cost

The core of ColBERT (Khattab and Zaharia, 2020) is the MaxSim score. Each query token picks the single most similar token in the whole document, and those similarities are summed.

S(Q, D) = Σ_i  max_j  (Q_i · D_j)
Q_i: i-th query token vector, D_j: j-th document token vector

Every query token gets a vote, which makes the method strong on polysemy and compound queries. The document side is precomputed, so an approximate nearest neighbor (ANN) index can be used. The cost is a vector stored per token, which makes the index large. The original ColBERT required 154 GiB for 140M passages (Khattab and Zaharia, 2020).

ColBERTv2 (Santhanam et al., 2022) reduces that cost two ways. Distillation from a cross-encoder teacher improves quality, and residual compression - encoding each token vector as a centroid index plus a quantized residual - shrinks storage. The index dropped to 16-25 GiB, 6-10 times smaller, with minimal loss in retrieval quality (Santhanam et al., 2022).

PLAID (Santhanam et al., 2022), the search engine built on top of it, first narrows candidates by centroid and runs full MaxSim only on the narrowed set. It exploits the tendency for token vectors of similar documents to cluster near the same centroid. Documents near distant centroids are unlikely to win the MaxSim comparison, so PLAID skips them. On 140M passages this gave a 2.5-7x speedup on GPU and 9-45x on CPU with no reported quality loss (Santhanam et al., 2022).

LBNL: reversing the order of encoding and interaction

ColBERT encodes documents independently and then matches, so it cannot share context between query and document or between documents. The LBNL (Last but Not Late) interaction in jina-reranker-v3 (Jina AI, 2025) reverses that order. The query and several candidate documents go into one shared context and interact through causal self-attention first. Embeddings are then pulled from designated token positions.

q̃ = final hidden state at the query position
d̃_i = hidden state at the designated token position of document i
score_i = cos(MLP(q̃), MLP(d̃_i))    # multilayer perceptron: 1024 → 512 → 256

Where ColBERT encodes independently and then matches, LBNL interacts within a shared context first and extracts embeddings afterward - the reverse order. Built on Qwen3-0.6B (28 layers, hidden size 1024, 131K context), it processes up to 64 documents in a single forward pass.

On the BEIR retrieval benchmark, measured by nDCG@10 (ranking quality over the top 10), jina-reranker-v3 (0.6B) scored higher than a model 2.5 times its size (Jina AI, 2025).

A sensitivity test that shuffled document order produced small deviations: 61.94 descending, 61.52 ascending, 62.54 random (Jina AI, 2025). That signals that the position bias common in listwise methods is mitigated by this structure. The trade-off is that the shared context imposes limits on document count and length under the 131K token budget. Pure late interaction, which can be indexed in advance, does not have that constraint.

Listwise LLM reranking and position bias

Another line of work hands the LLM the entire candidate list and lets it decide the order directly. When the candidates exceed the prompt length, a sliding window processes them in overlapping chunks.

window_size = 20
stride = 10
# rerank with overlapping windows moving from the back of the list toward the front
for start in range(len(candidates) - window_size, -stride, -stride):
    window = candidates[start:start + window_size]
    reranked = llm_listwise_rank(query, window)
    candidates[start:start + window_size] = reranked

This structure carries position bias that Qiao et al. (2026) define as the phenomenon where documents later in the list are less likely to move up even when they are highly relevant. One cause is an architectural bias in which attention responds more strongly to earlier tokens. The other is a training-data problem in which relevant documents cluster near the front, teaching the model to treat position as a proxy signal for relevance. The sliding window itself adds to the bias, since a document at the back must pass through several shifts to reach the front.

Three studies from 2026 approach the problem at different levels, intervening in training, architecture, and representation respectively.

A comparison of 22 methods by Abdallah et al. (2025, EMNLP Findings) supports the view that this bias is a pattern across LLM-based rerankers rather than an artifact of any one paper. The same study notes that LLM rerankers are strong on familiar benchmarks but generalize unevenly to unseen query distributions. Position bias may therefore be entangled with a failure to generalize relevance judgment, not purely a matter of position.

In practice you can quantify the bias in your own reranker. Feed the same candidate set several times with only the order changed, then measure the consistency of the top K results. Large deviation is a signal that the reranker cannot correct for a first-stage retriever that placed relevant documents near the back.

Summary

The central axis of reranking design is when the query and the document meet. ColBERT keeps documents as token vectors and matches with MaxSim to close the gap between cross-encoder precision and bi-encoder speed. Residual compression and PLAID manage its index cost and latency. LBNL in jina-reranker-v3 reverses the order of encoding and interaction and lets a 0.6B model beat one 2.5 times its size. The shared context in exchange leaves limits on document count and length.

Listwise LLM reranking reflects cross-document context in one pass at the price of position bias. That bias is structural enough to be attacked simultaneously at the training, architecture, and representation levels. In production the practical order is to diagnose bias first through the consistency of shuffled-order results. The next step is to choose between late interaction and LBNL according to corpus size and latency budget.