RAG Cross-Encoder Reranking

A two-stage retrieval strategy where a bi-encoder narrows the field quickly and a cross-encoder re-reads each candidate together with the query to fix the ranking.

Why a bi-encoder alone falls short

Retrieval-Augmented Generation (RAG) searches external documents and grounds its answer in them, so retrieval quality determines answer quality. A bi-encoder is one way to build a first-stage retriever. The DPR paper describes this retrieval structure.

A bi-encoder turns the query and each passage into vectors separately, then measures similarity as the dot product of the two vectors. Passage vectors can be computed ahead of time, before any query arrives, which makes retrieval fast. The query and passage do not interact during encoding. As a result, passages that merely share vocabulary are easy to overrate.

The real problem is precision. Even when the correct passage makes it into the candidate set (recall is satisfied), the LLM never sees it unless it climbs into the top few positions. That calls for a separate stage that picks the genuinely relevant items out of the candidates and lifts them to the top.

Paired input in a cross-encoder

A cross-encoder feeds the query and the passage in as a single input. Borrowing the BERT-style encoder format, that looks like [CLS] query tokens [SEP] passage tokens [SEP]. Here [CLS] is the special token that represents the input as a whole, and [SEP] marks the boundary between the two texts.

The query side gets segment 0 and the passage side gets segment 1. During self-attention, query tokens and passage tokens reference each other directly. The model therefore works out in detail which part of the query corresponds to which part of the passage. A linear projection of the final [CLS] vector produces a single relevance score.

The Sentence-BERT paper explains the computational difference between joint pair processing and independent embeddings. Additional token interaction does not remove the need to evaluate a particular model's ranking quality. The cost is high. Passage vectors cannot be precomputed; encoding starts only after the query arrives and the pair is formed. With a million passages, every single query would require a million encodings, which produces latency no real-time service can absorb.

Connecting the two separates candidate retrieval from per-candidate evaluation. Quality and latency determine whether the extra stage is useful.

The two-stage re-ranking pipeline

Re-ranking is the stage that narrows candidates with a fast retriever and then re-orders those candidates with a precise model. The overall flow looks like this.

flowchart LR
    Q["Query<br/>User input"] --> BI["Bi-encoder<br/>Approximate nearest neighbor search"]
    BI --> C["Hundreds to thousands of candidates<br/>Recall secured"]
    C --> CE["Cross-encoder<br/>Scores query-passage pairs"]
    CE --> R["Final top k<br/>Precision improved"]

• Stage 1: the bi-encoder pulls hundreds to thousands of candidates out of an approximate nearest neighbor index at low latency.
• Stage 2: the cross-encoder takes each candidate together with the query and assigns a relevance score.
• With 500 candidates that is only 500 encodings, which is nowhere near the cost of scoring a million-document corpus.

The first stage owns recall and the second owns precision. If the correct answer is absent from the candidate set, fix retrieval. If it is present but ranks poorly, fix re-ranking.

Ranking correction in code

The following example pulls five candidates with a bi-encoder and re-sorts them with a cross-encoder, using public models from sentence-transformers.

import numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder
 
# Bi-encoder: encode the query and the passages separately
bi_encoder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
 
query = "the effect of caffeine on sleep"
passages = [
    "Caffeine blocks adenosine receptors and suppresses drowsiness.",
    "The roasting temperature of coffee beans strongly affects flavor.",
    "Sleep deprivation causes reduced concentration and weakened immune function.",
    "Caffeine has a half-life of about 5 hours, so afternoon intake can affect bedtime.",
    "A single shot of espresso contains about 63mg of caffeine.",
    "Growth hormone is secreted during deep sleep stages.",
    "Caffeine is the most widely consumed psychoactive substance in the world.",
    "Blue light exposure before bed suppresses melatonin secretion.",
]
 
q_vec = bi_encoder.encode(query)
p_vecs = bi_encoder.encode(passages)
scores_bi = p_vecs @ q_vec                    # dot product similarity
top_indices = np.argsort(scores_bi)[::-1][:5] # top 5
 
# Cross-encoder: score the query-passage pairs fed in together
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
candidates = [passages[i] for i in top_indices]
pairs = [[query, c] for c in candidates]
scores_ce = cross_encoder.predict(pairs)
 
reranked = sorted(zip(top_indices, scores_ce), key=lambda x: x[1], reverse=True)

This code was not executed for this article. Check each model's supported languages, training data, and similarity function. No ranking or cause of a ranking change is claimed as measured here.

Candidate count and hybrid retrieval

The balance between precision and cost is tuned through the number of candidates the bi-encoder hands over. More candidates widen what the cross-encoder can see without guaranteeing better precision, and the extra encodings increase latency. Fewer candidates run faster, but any passage the bi-encoder missed stays invisible to the cross-encoder too.

To make the first stage sturdier, lexical and dense retrieval are often run together. BM25, a term-frequency-based algorithm, and dense vector search each produce candidates, and the two sets are merged with duplicates removed. Applying the cross-encoder to that union re-ranks the different candidates found by both retrievers in one pass.

The division of labor in this setup is clear. The union of BM25 and dense retrieval lifts recall, and the cross-encoder handles precision on top of it.

Slimming down with knowledge distillation

Putting a cross-encoder on the serving path adds that much latency. Knowledge distillation moves the cost into the training phase instead. The precise but slow cross-encoder becomes the teacher, and the fast but less precise bi-encoder becomes the student in a teacher-student setup.

RocketQAv2 (EMNLP 2021, arXiv:2110.07367) jointly trains the retriever and reranker through dynamic distillation of listwise score distributions. Teacher scores can be logits or normalized distributions depending on the implementation; they are not always calibrated probabilities between zero and one.

This study provides a specific way to connect the training of two components. It does not mean that every student matches its teacher or achieves the same cost reduction. Evaluate retrieval quality and latency for the model to be deployed.

Summary

A cross-encoder reads the query and the passage together and judges relevance at the token level. The trade-off is that every candidate has to be re-encoded, so it cannot be applied directly to a full corpus. This is why the two-stage structure is common: narrow candidates quickly with a bi-encoder, then re-rank with a cross-encoder.

The precision-latency balance is tuned through the candidate count, and when serving cost becomes a burden, knowledge distillation moves that judgment into a lighter model. When improving the pipeline, start by identifying whether recall or precision is the weak side, then pick the stage to work on.