RAG Re-ranking: Cross-Encoders

A two-stage retrieval strategy where a bi-encoder narrows the field quickly and a cross-encoder re-reads each candidate together with the query to fix the ranking.

Why a bi-encoder alone falls short

Retrieval-Augmented Generation (RAG) searches external documents and grounds its answer in them, so retrieval quality determines answer quality. The most widely used first-stage retriever is the bi-encoder.

A bi-encoder turns the query and each passage into vectors separately, then measures similarity as the dot product of the two vectors. Passage vectors can be computed ahead of time, before any query arrives, which makes retrieval fast. The cost is that the query and the passage never see each other, so passages that merely share vocabulary are easy to overrate.

The real problem is precision. Even when the correct passage makes it into the candidate set (recall is satisfied), the LLM never sees it unless it climbs into the top few positions. That calls for a separate stage that picks the genuinely relevant items out of the candidates and lifts them to the top.

Paired input in a cross-encoder

A cross-encoder feeds the query and the passage in as a single input. Borrowing the BERT-style encoder format, that looks like [CLS] query tokens [SEP] passage tokens [SEP]. Here [CLS] is the special token that represents the input as a whole, and [SEP] marks the boundary between the two texts.

The query side gets segment 0 and the passage side gets segment 1. During self-attention, query tokens and passage tokens reference each other directly. The model therefore works out in detail which part of the query corresponds to which part of the passage. A linear projection of the final [CLS] vector produces a single relevance score.

That token-level interaction is why a cross-encoder judges relevance more precisely than a bi-encoder. The cost is high. Passage vectors cannot be precomputed; encoding starts only after the query arrives and the pair is formed. With a million passages, every single query would require a million encodings, which produces latency no real-time service can absorb.

Each model holds one of the two properties, speed or precision. Rather than choosing one, it works better to chain them in sequence.

The two-stage re-ranking pipeline

Re-ranking is the stage that narrows candidates with a fast retriever and then re-orders those candidates with a precise model. The overall flow looks like this.

• Stage 1: the bi-encoder pulls hundreds to thousands of candidates out of an approximate nearest neighbor index at low latency.
• Stage 2: the cross-encoder takes each candidate together with the query and assigns a relevance score.
• With 500 candidates that is only 500 encodings, which is nowhere near the cost of scoring a million-document corpus.

The first stage owns recall and the second owns precision. If the correct answer is absent from the candidate set, fix the retrieval stage; if it is present but never rises to the top, fix the re-ranking stage.

Ranking correction in code

The following example pulls five candidates with a bi-encoder and re-sorts them with a cross-encoder, using public models from sentence-transformers.

import numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder
 
# Bi-encoder: encode the query and the passages separately
bi_encoder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
 
query = "the effect of caffeine on sleep"
passages = [
    "Caffeine blocks adenosine receptors and suppresses drowsiness.",
    "The roasting temperature of coffee beans strongly affects flavor.",
    "Sleep deprivation causes reduced concentration and weakened immune function.",
    "Caffeine has a half-life of about 5 hours, so afternoon intake can affect bedtime.",
    "A single shot of espresso contains about 63mg of caffeine.",
    "Growth hormone is secreted during deep sleep stages.",
    "Caffeine is the most widely consumed psychoactive substance in the world.",
    "Blue light exposure before bed suppresses melatonin secretion.",
]
 
q_vec = bi_encoder.encode(query)
p_vecs = bi_encoder.encode(passages)
scores_bi = p_vecs @ q_vec                    # dot product similarity
top_indices = np.argsort(scores_bi)[::-1][:5] # top 5
 
# Cross-encoder: score the query-passage pairs fed in together
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
candidates = [passages[i] for i in top_indices]
pairs = [[query, c] for c in candidates]
scores_ce = cross_encoder.predict(pairs)
 
reranked = sorted(zip(top_indices, scores_ce), key=lambda x: x[1], reverse=True)

Running it produces an ordering along these lines.

Bi-encoder top 5 (by dot product similarity)
  1. Caffeine has a half-life of about 5 hours, so afternoon intake can affect bedtime.
  2. Caffeine is the most widely consumed psychoactive substance in the world.
  3. Caffeine blocks adenosine receptors and suppresses drowsiness.
  4. A single shot of espresso contains about 63mg of caffeine.
  5. Sleep deprivation causes reduced concentration and weakened immune function.
 
After cross-encoder re-ranking
  1. Caffeine blocks adenosine receptors and suppresses drowsiness.
  2. Caffeine has a half-life of about 5 hours, so afternoon intake can affect bedtime.
  3. Sleep deprivation causes reduced concentration and weakened immune function.
  4. Caffeine is the most widely consumed psychoactive substance in the world.
  5. A single shot of espresso contains about 63mg of caffeine.

The bi-encoder pushed the psychoactive substance line and the espresso content line upward purely because the word 'caffeine' overlapped. The passage that actually explains the sleep mechanism, adenosine receptor blocking, was stuck in third place. The cross-encoder lifted that passage to first and pushed the sleep-irrelevant items down. The point is that two models sorted the very same candidate set differently.

Candidate count and hybrid retrieval

The balance between precision and cost is tuned through the number of candidates the bi-encoder hands over. More candidates widen what the cross-encoder can see and raise precision, but the extra encodings increase latency. Fewer candidates run faster, but any passage the bi-encoder missed stays invisible to the cross-encoder too.

To make the first stage sturdier, lexical and dense retrieval are often run together. BM25, a term-frequency-based algorithm, and dense vector search each produce candidates, and the two sets are merged with duplicates removed. Applying the cross-encoder to that union re-ranks the different candidates found by both retrievers in one pass.

The division of labor in this setup is clear. The union of BM25 and dense retrieval lifts recall, and the cross-encoder handles precision on top of it.

Slimming down with knowledge distillation

Putting a cross-encoder on the serving path adds that much latency. Knowledge distillation moves the cost into the training phase instead. The precise but slow cross-encoder becomes the teacher, and the fast but less precise bi-encoder becomes the student in a teacher-student setup.

The student learns to approximate the soft scores the teacher assigned to query-passage pairs, meaning continuous probabilities between 0 and 1. With binary relevant/irrelevant labels alone, the student struggles to learn the subtle distinctions near the decision boundary. Continuous scores such as 0.92 relevant and 0.71 relevant provide a richer gradient signal and let the student pick up fine gradations.

The student loss uses KL divergence (Kullback-Leibler divergence) or mean squared error (MSE) against the teacher scores. The original contrastive loss and the distillation loss are often weighted and optimized together. Once training is done, serving no longer needs the teacher. The student bi-encoder alone handles retrieval and ranking, keeping the precomputation advantage while approaching cross-encoder ranking quality.

Reports exist of distillation reaching quality close to the heavier model at lower cost, but the specific numbers vary by model and dataset, so they do not generalize as stated.

Summary

A cross-encoder reads the query and the passage together and judges relevance at the token level. That makes it more precise than a bi-encoder, which gets swayed by vocabulary overlap. The trade-off is that every candidate has to be re-encoded, so it cannot be applied directly to a full corpus. This is why the two-stage structure is common: narrow candidates quickly with a bi-encoder, then re-rank with a cross-encoder.

The precision-latency balance is tuned through the candidate count, and when serving cost becomes a burden, knowledge distillation moves that judgment into a lighter model. When improving the pipeline, start by identifying whether recall or precision is the weak side, then pick the stage to work on.