RAG

A technique where a large language model (LLM) searches an external database and generates an answer from the evidence documents it retrieves.

Concept

Retrieval-Augmented Generation (RAG) retrieves relevant information from an external database and uses it during generation. It does not rely only on the LLM's internal knowledge. LLM here means large language model, and hallucination means the model producing plausible-sounding content that is not true.

How it works

1. Question input: the user submits a question.
2. Document retrieval: relevant documents are retrieved from a vector database.
3. Context assembly: the retrieved results and the question are passed to the LLM together.
4. Answer generation: the LLM generates an answer using the retrieved documents.

Why RAG is needed

An LLM cannot know anything after its training cutoff, makes things up when it does not know, and cannot cite evidence. These three limits break down as follows.

Advantages of RAG

Example

Question: "Who won the 2024 Nobel Prize in Physics?"

LLM alone "Information about the 2024 Nobel Prize in Physics is not included in my training data." (or a hallucination)

With RAG

1. Vector database search: "2024 Nobel Prize in Physics"
2. Relevant documents retrieved: news articles, Wikipedia
3. Documents passed to the LLM along with the question
4. Answer: "The 2024 Nobel Prize in Physics went to John Hopfield and Geoffrey Hinton."

How vector search works

Creating embeddings

Document: "The capital of Korea is Seoul" It is converted through an embedding model. Vector: [0.23, -0.45, 0.12, ...]

Similarity search

Question: "What is the capital of Korea?" Question embedding: [0.21, -0.43, 0.15, ...] The most similar document is retrieved from the vector database. Result: "The capital of Korea is Seoul" (similarity: 0.95)

The RAG pipeline

1. Indexing stage: documents are split into chunks, embeddings are created, and the results are stored in a vector database.
2. Query stage: the question is embedded, similar documents are retrieved, and both are passed to the LLM to generate an answer.

Chunking strategies

GraphRAG: structured knowledge

Basic RAG retrieves chunks of text. GraphRAG uses a knowledge graph instead.

Question: "Who discovered penicillin?" A path is traced through the knowledge graph: Alexander Fleming → discovered → Penicillin Alexander Fleming → year → 1928 Answer: "Alexander Fleming discovered it in 1928"

Knowledge graph example

Alexander Fleming

• discovered → Penicillin
• nationality → Scottish
• profession → Bacteriologist

Penicillin

• type → Antibiotic
• discovered_year → 1928

RAG vs fine-tuning

Implementation example

The following is pseudocode illustrating the flow, not real library signatures. k is the number of top documents to retrieve, and embedding_model is the model that turns documents into vectors. context is the step that concatenates the retrieved documents into the prompt.

# pseudo: not a real API, a simplified illustration of the RAG flow
class RAGSystem:
    def __init__(self, documents):
        # embed and store the documents
        self.vector_store = VectorStore.from_documents(
            documents,
            embedding_model="text-embedding-ada-002"
        )
        self.llm = LLM(model="gpt-4")
 
    def query(self, question):
        # 1. retrieve relevant documents
        relevant_docs = self.vector_store.similarity_search(
            question,
            k=3  # top 3 documents
        )
 
        # 2. build the prompt
        context = "\n".join([doc.content for doc in relevant_docs])
        prompt = f"""Answer the question using the documents below.
 
Documents:
{context}
 
Question: {question}
 
Answer:"""
 
        # 3. generate the LLM response
        answer = self.llm.generate(prompt)
 
        return {
            "answer": answer,
            "sources": [doc.source for doc in relevant_docs]
        }

Advanced techniques

Hybrid search

Vector search combined with keyword search:

Question: "What is the Python GIL?"

Vector search: retrieves semantically similar documents. Keyword search: retrieves documents containing "GIL".

Combining both results gives more accurate retrieval.

Re-ranking

Reordering the retrieval results with an LLM:

Fetch ten retrieval results, then have the LLM reorder them by relevance to the question. The reordered results are what gets used.

Query expansion

Expanding the question:

Original question: "Python speed improvement" Expanded questions: "Python performance optimization", "Python performance", "Python profiling"

The expanded questions retrieve a wider set of relevant documents.

Agentic RAG

Basic RAG ends after one pass of "retrieve, then generate". Agentic RAG puts the agent in active control of retrieval itself. It decides when to retrieve, what to retrieve again, and whether the results are sufficient.

The Agentic RAG Survey (arXiv 2501.09136) organizes the designs into four patterns.

A-RAG (arXiv 2602.03442), for example, exposes three tools directly to the agent: keyword search, semantic search, and chunk reading. It beats prior methods with the same or fewer retrieval tokens. The direction is to solve complex reasoning and multi-document composition, where single-shot retrieval is weak, with a retrieval loop.

Limits

1. Depends on retrieval quality: fails if the relevant document is not found
2. Context limits: too many documents cannot be processed
3. Real-time information: database updates lag
4. Complex reasoning: combining several documents is hard. Agentic RAG partially mitigates this with sub-question decomposition and re-retrieval loops.

Summary

RAG passes evidence documents obtained from external retrieval to the LLM, compensating for the knowledge cutoff and hallucination. Updating documents alone reflects current information and makes sources verifiable, which costs less than fine-tuning with its retraining requirement. In exchange, answer quality depends heavily on retrieval quality, and the context limit and multi-document reasoning remain weak points. Chunking, hybrid search, and re-ranking are the corrective measures that raise retrieval quality. Complex questions that single-shot retrieval cannot handle extend to Agentic RAG, where the agent actively controls retrieval.

Related concepts

• LLM Agent Survey: memory mechanisms
• ReAct: tool use (including search)
• MADKE: knowledge-pool-based debate