RAG

A technique where a large language model (LLM) searches an external database and generates an answer from the evidence documents it retrieves.

Concept

The NeurIPS 2020 RAG paper (arXiv:2005.11401) combines retrieval and generation. The following is a general explanation of that relationship, not the full implementation of the original paper.

Retrieval-Augmented Generation (RAG) retrieves relevant information from an external database and uses it during generation. It does not rely only on the LLM's internal knowledge. LLM here means large language model, and hallucination means the model producing plausible-sounding content that is not true.

flowchart LR
    D["Source documents"] --> E["Chunk embeddings"]
    E --> V["Vector index"]
    Q["User question"] --> R["Retrieve relevant documents"]
    V --> R
    R --> C["Combine question and evidence"]
    C --> L["LLM answer"]

How it works

1. Question input: the user submits a question.
2. Document retrieval: relevant documents are retrieved from a vector database.
3. Context assembly: the retrieved results and the question are passed to the LLM together.
4. Answer generation: the LLM generates an answer using the retrieved documents.

Why RAG is needed

An LLM cannot know anything after its training cutoff, makes things up when it does not know, and cannot cite evidence. These three limits break down as follows.

Advantages of RAG

Example

Question: "Who won the 2024 Nobel Prize in Physics?"

LLM alone "Information about the 2024 Nobel Prize in Physics is not included in my training data." (or a hallucination)

With RAG

1. Vector database search: "2024 Nobel Prize in Physics"
2. Relevant documents retrieved: news articles, Wikipedia
3. Documents passed to the LLM along with the question
4. Answer: "The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton."

How vector search works

Creating embeddings

Document: "The capital of Korea is Seoul" It is converted through an embedding model. Vector: [0.23, -0.45, 0.12, ...]

Similarity search

Question: "What is the capital of Korea?" Question embedding: [0.21, -0.43, 0.15, ...] The most similar document is retrieved from the vector database. Result: "The capital of Korea is Seoul" (similarity: 0.95)

The RAG pipeline

1. Indexing stage: split documents into chunks. Create embeddings and store them in a vector database.
2. Query stage: embed the question and retrieve similar documents. Pass both to the LLM to generate an answer.

Chunking strategies

Retrieval with a graph

Relations in a knowledge graph can accompany text as retrieval evidence. The following illustrates relation traversal, not the full Microsoft GraphRAG algorithm or its performance results.

Question: "Who discovered penicillin?" A path is traced through the knowledge graph: Alexander Fleming → discovered → Penicillin Alexander Fleming → year → 1928 Answer: "Alexander Fleming discovered it in 1928"

Knowledge graph example

Alexander Fleming

• discovered → Penicillin
• nationality → Scottish
• profession → Bacteriologist

Penicillin

• type → Antibiotic
• discovered_year → 1928

RAG vs fine-tuning

Implementation example

The following is pseudocode illustrating the flow, not real library signatures. k is the number of top documents to retrieve, and embedding_model is the model that turns documents into vectors. context is the step that concatenates the retrieved documents into the prompt.

# pseudo: not a real API, a simplified illustration of the RAG flow
class RAGSystem:
    def __init__(self, documents):
        # embed and store the documents
        self.vector_store = VectorStore.from_documents(
            documents,
            embedding_model="text-embedding-ada-002"
        )
        self.llm = LLM(model="gpt-4")
 
    def query(self, question):
        # 1. retrieve relevant documents
        relevant_docs = self.vector_store.similarity_search(
            question,
            k=3  # top 3 documents
        )
 
        # 2. build the prompt
        context = "\n".join([doc.content for doc in relevant_docs])
        prompt = f"""Answer the question using the documents below.
 
Documents:
{context}
 
Question: {question}
 
Answer:"""
 
        # 3. generate the LLM response
        answer = self.llm.generate(prompt)
 
        return {
            "answer": answer,
            "sources": [doc.source for doc in relevant_docs]
        }

Advanced techniques

Hybrid search

Vector search combined with keyword search:

Question: "What is the Python GIL?"

Vector search: retrieves semantically similar documents. Keyword search: retrieves documents containing "GIL".

Combining both results gives more accurate retrieval.

Re-ranking

Reordering the retrieval results with an LLM:

Fetch ten retrieval results, then have the LLM reorder them by relevance to the question. The reordered results are what gets used.

Query expansion

Expanding the question:

Original question: "Python speed improvement" Expanded questions: "Python performance optimization", "Python performance", "Python profiling"

The expanded questions retrieve a wider set of relevant documents.

Agentic RAG

Basic RAG ends after one pass of "retrieve, then generate". Agentic RAG puts the agent in active control of retrieval itself. It decides when to retrieve, what to retrieve again, and whether the results are sufficient.

ReAct is a foundational example connecting reasoning and retrieval actions. FLARE retrieves additional material using uncertain parts of an ongoing generation. These studies inform retrieval timing; they do not show that adaptive retrieval is more efficient for every multi-document question.

Limits

1. Depends on retrieval quality: fails if the relevant document is not found
2. Context limits: too many documents cannot be processed
3. Real-time information: database updates lag
4. Complex reasoning: combining several documents is hard. Agentic RAG partially mitigates this with sub-question decomposition and re-retrieval loops.

Related concepts

• ReAct: connecting reasoning with retrieval actions
• DPR: retrieval through question and passage encoding

Summary

RAG passes evidence documents obtained from external retrieval to the LLM, compensating for the knowledge cutoff and hallucination. Updated documents can supply new information, but indexes must be updated and sources checked. Comparing costs with fine-tuning requires request volume together with training, retrieval, and generation costs.

In exchange, answer quality depends heavily on retrieval quality, and the context limit and multi-document reasoning remain weak points. Chunking, hybrid search, and re-ranking are the corrective measures that raise retrieval quality. Complex questions that single-shot retrieval cannot handle extend to Agentic RAG, where the agent actively controls retrieval.