Dense Retrieval for RAG: DPR and the Dual Encoder

Dense retrieval turns queries and documents into vectors and matches them by semantic similarity. DPR is the dual encoder architecture that put the idea to work.

The problem dense retrieval solves

Retrieval methods split into sparse and dense. Sparse retrieval depends on whether terms appear and how often. BM25 is the canonical example, and a match requires the same term to occur in both the query and the document. The document vector has one dimension per vocabulary entry, but almost every value is zero, which is why it is called sparse.

Dense retrieval feeds queries and documents through a neural encoder and produces vectors of a few hundred dimensions. Relevance is then judged by similarity between those vectors. Most dimensions carry non-zero values, hence dense.

The two fail in different ways. Dense handles synonyms and paraphrase well but tends to miss rare proper nouns and exact strings. Sparse is strong on exact matches but misses synonyms. That is why hybrid configurations combining both are common in production.

The two encoders in DPR

DPR (Dense Passage Retrieval) applies dense retrieval to open-domain question answering. Its architecture is a dual encoder, sometimes called a bi-encoder. A dual encoder feeds the query and the document through encoders separately and compares the two vectors. What sets DPR apart from a typical dual encoder is that it splits the encoder in two.

General-purpose dual encoders such as Sentence-BERT often share one BERT between queries and documents. A single parameter set is lighter and produces general embeddings without distinguishing query from document. DPR instead keeps two encoders, both initialized from the same BERT-base, and updates their parameters independently during training.

One is the question encoder and the other is the passage encoder. This design is called an asymmetric dual encoder. Questions are short and their interrogative phrasing carries the key signal, while passages are long and mostly declarative. Splitting the encoders lets each specialize its representation for its input.

The split has a cost. Parameter count roughly doubles. With BERT-base that means two copies of roughly 110 million parameters, and generality is traded away in the process.

Alignment through contrastive learning

Because the two encoders hold different parameters, training has to place them in a shared semantic space. DPR aligns them with contrastive learning, which pushes related pairs together and unrelated pairs apart.

Positive pairs come from open-domain QA datasets. Natural Questions and TriviaQA provide questions along with answer strings. Finding a Wikipedia passage that contains the answer string yields a question-passage pair, and that pair becomes the positive for the contrastive objective.

ORQA, an earlier approach to learning retrieval representations, relied on large-scale unsupervised pretraining through the Inverse Cloze Task (ICT). DPR beat BM25 by a wide margin using only these supervised pairs, without that pretraining.

The loss function is InfoNCE, a contrastive loss based on noise-contrastive estimation. It raises the inner product with the correct passage and lowers it for incorrect ones. One positive competes against several negatives through a softmax, pushing the correct passage to the highest score.

Where the negatives come from largely determines training quality. DPR uses two kinds together.

With a batch size of 128, each question gets the gold passages of the other 127 questions as in-batch negatives. Using only in-batch negatives gives mostly easy examples and a weak training signal. Using only hard negatives skews toward difficult cases and destabilizes training. DPR mixes the two to balance stable gradients against discriminative power.

Hard negatives and gradients

Harder negatives give the model a stronger training signal, and the mechanism lives in the gradient of the loss. In a contrastive loss, the gradient flowing to a negative grows as that negative's score approaches the positive's.

When a negative is easy, meaning q·d⁻ is already far below q·d⁺, the loss is small and the gradient is weak. There is little to learn from a pair the model already separates. When a negative is hard, meaning q·d⁻ sits close to q·d⁺, the gradient is strong. That gradient is the signal that pushes the model to be more precise near the decision boundary.

The complication is that hard negative difficulty shifts during training. As the model improves, negatives that used to be hard become easy. ANCE, a follow-up method, handles this with asynchronous updates.

Re-encoding the entire corpus at every training step is not feasible. ANCE re-encodes the corpus with the current model at fixed intervals and rebuilds the ANN index, the approximate nearest neighbor index. Between updates it keeps mining negatives from the previous index. The index lags slightly behind the model, a staleness cost, but it is still far closer to the current model than BM25 negatives fixed before training began.

Indexing and query time

The core advantage of a dual encoder comes from being able to split encoding into two moments. Offline indexing builds passage vectors ahead of time, and online retrieval builds only one vector when a query arrives.

Offline indexing runs once, before the service takes any queries. Corpus passages go through the passage encoder to become vectors, which are stored in a searchable structure. With millions of passages, batching the encoding speeds this up considerably.

Online retrieval has three steps. The question encoder converts the query into a single vector. That vector goes into the ANN index, which returns the top k passage vectors by inner product. Finally, the IDs returned by the index are used to fetch the actual passage text from storage.

At small scale you can pick the top k by exhaustive comparison. Here is the minimal version that computes inner products against every passage and takes the top three.

import numpy as np
 
# passage_vectors: (N, d) passage vectors, query_vector: (d,) query vector
scores = passage_vectors @ query_vector      # inner product scores
top_k = np.argsort(scores)[-3:][::-1]         # indices of the top 3

With millions of passages that exhaustive comparison is too slow. ANN returns an approximate top k at sublinear cost, and that approximation is what makes real-time retrieval over a large corpus possible.

Pre-indexing is possible only because the encoders are separate. If queries and passages went through one encoder together, every query would require re-encoding the whole corpus. With a dual encoder the passage vectors already exist, so retrieval costs one query encoding plus vector comparison.

The value tuned most often at query time is top-k. The DPR paper measures performance at the top 20 and top 100. Raising k improves the recall of finding the correct passage but increases the number of candidates the next stage has to process, and therefore its cost.

Summary

Dense retrieval matches queries to documents by converting both into vectors and comparing semantic similarity. DPR is the dual encoder model that applied this to open-domain QA. Splitting the question encoder from the passage encoder, an asymmetric design, lets each representation specialize for its input. The two encoders are aligned by InfoNCE contrastive learning, mixing in-batch negatives with BM25 hard negatives to balance stability against discriminative power.

Hard negatives generate stronger gradients the closer they sit to the positive, and ANCE refreshes that difficulty asynchronously during training. In production, offline indexing builds passage vectors ahead of time and ANN retrieves the top k quickly online. Since dense and sparse retrieval miss different things, a hybrid of the two is a common way to cover both.