AI knowledge systems from documents to answers

An AI knowledge system prepares documents as searchable evidence, selects evidence for a question, and uses it to answer or run tools.

Document preparation and question processing

Consider a system that accepts operational documents and answers questions about incident response. Reading document text, finding relevant content, and generating answers each have different inputs and outputs. Separating these roles helps identify where information was lost or became incorrect.

The overall flow has two parts: preparing and updating documents, and processing questions. Document preparation preserves the originals and creates retrieval units. Question processing selects evidence from those units and passes it to a generative model.

flowchart TD
    subgraph PREP["Document preparation and updates"]
        SRC["PDF · Office documents · Web · Markdown"]
        RAW["Preserve originals and sources"]
        PARSE["Parsing · OCR if needed"]
        CHUNK["Cleaning · Chunking · Metadata"]
        EMB["Document embeddings"]
        INDEX["Keyword and vector indexes"]
        EXTRACT["Extract and validate concepts and relations"]
        KG["Knowledge graph and evidence"]
        ONT["Ontology type and relation rules"]
        SRC --> RAW --> PARSE --> CHUNK
        CHUNK --> EMB --> INDEX
        CHUNK --> INDEX
        CHUNK -. "Optional extraction" .-> EXTRACT --> KG
        ONT -. "Apply rules" .-> EXTRACT
    end
    subgraph ANSWER["Question processing"]
        Q["User question"]
        RET["Hybrid search"]
        RANK["Reranking if needed"]
        CTX["Select evidence · Build context"]
        GEN["LLM answer generation"]
        OUT["Answer · Sources"]
        Q --> RET --> RANK --> CTX --> GEN --> OUT
    end
    INDEX --> RET
    KG -. "Optional relation traversal" .-> CTX

This diagram shows one possible configuration. OCR, reranking, and knowledge graphs depend on document formats and questions; not every system needs them. An agent can use these retrieval and generation functions as tools while choosing its next action.

What an LLM does

Among large language models (LLMs), autoregressive generative models calculate a distribution over the next token from the context. They add a selected token to the context and repeat this process to form sentences. A token is not exactly a word; it can represent part of a word or a character.

Transformer attention calculates relevance between information within the context. Different input and output formats support summarization, translation, classification, and relation extraction. However, a fluent sentence does not guarantee that its content is true. Google's explanation of Transformers

Embedding documents or adding them to a search index generally does not train the generative LLM's weights. RAG can include a trained retriever or reranker. However, incorporating new documents into answers does not always require fine-tuning the generative model.

Structure to preserve during parsing

Document parsing reads content and components from a file format. Extracted text does not prove that heading hierarchies, reading order, or table column relationships were also recovered. Optical character recognition (OCR) reads characters in images; it is distinct from recovering table or whole-document structure.

Collecting a web page and parsing its body from HTML are also separate tasks. A PDF consisting mainly of images may require OCR. A text layer allows a text parser, but the extracted content and order still need checking.

Collecting only product names and numbers from a price table can lose the relationship between each product and its price. The following illustrative JSON represents the document, page, section, and table rows and columns together. It is not a particular parser's output or an actual storage schema.

{
  "document_id": "guide-001",
  "source_page": 3,
  "section_path": ["Service guide", "Pricing"],
  "block_type": "table",
  "columns": ["Plan", "Monthly price", "Currency"],
  "rows": [["Basic", 10000, "KRW"]]
}

Creating retrieval units with ETL and chunking

ETL processes data in the order Extract, Transform, Load. ELT loads extracted data into the destination store first, then transforms it there. Explaining the difference requires specifying which store is the load destination. AWS comparison of ETL and ELT

Parsing, cleaning, chunking, and embedding documents before loading them into a search index can be considered ETL. Loading collected data into a data lake first and processing it there is ELT. A separate backup of the original files does not, by itself, make the whole process ELT.

Chunking divides long documents into units for retrieval and generation. Large chunks can mix topics, while small chunks can separate definitions from conditions. Evaluate chunk sizes and boundaries against document structure, model input limits, and actual questions.

Chunking can use headings and section boundaries, or fixed lengths with some overlap. Overlap restores context around boundaries but increases duplicate search results and storage. A sentence such as "Enable this setting" needs section information and surrounding context to identify the setting.

Document updates and deletions must also affect previous chunks and derived relations. Define identifiers and update rules to prevent duplicates when processing the same input again. Do not invent page numbers or locations that could not be extracted from the original.

Embeddings and vector stores

An embedding model represents an input as an array of real numbers, such as [0.12, -0.08, ...]. Retrieval models learn to compare questions and relevant documents in a compatible vector space. High similarity signals a relevant candidate; it does not prove that a statement is true or answers the question correctly.

Document vectors are usually computed during ingestion, while question vectors are computed when a search is requested. Some models require different input formats or prompts for questions and documents. Both inputs must follow that model's retrieval conventions.

Vectors from different models cannot simply be mixed, even when they have the same number of dimensions. Check compatibility when the model, revision, input format, or normalization changes, and re-embed if necessary. Changing only the model name while retaining the old index cannot guarantee retrieval quality.

A vector database (Vector DB) stores vectors, IDs, and metadata and provides similarity search. Existing search engines or databases can also provide this function, alongside dedicated products. Elasticsearch provides both keyword and vector search. Elastic's explanation of search and ranking

Exact nearest neighbor search and approximate nearest neighbor search (ANN) differ in search cost and recall. ANN reduces search cost but can miss relevant candidates, so it requires evaluation with actual data. Connect originals, chunks, and vectors with stable IDs if they occupy different stores. Do not design the system to reconstruct original text from vectors alone.

Hybrid search and reranking

Keyword and vector search use different signals. Exact term matching matters for an error code such as ECONNREFUSED. Embeddings may help find similar expressions for a question such as "The connection keeps getting refused." Hybrid search combines these signals. Elastic's explanation of hybrid search

Reciprocal Rank Fusion (RRF) combines rankings from multiple retrieval lists. A candidate's score can be calculated by adding 1 / (k + rank) for each list containing that candidate. The constant k controls the influence of rank, and RRF itself does not call a generative LLM. Elastic's explanation of RRF

Reranking evaluates retrieved candidates again. For example, a system could retrieve 50 candidates and select the top 5, but these numbers are illustrative. A CrossEncoder reranker takes the question and a candidate together to assess relevance.

Some methods use an LLM to compare candidates, but not all reranking uses a generative LLM. Measure the additional latency alongside actual ranking improvements. Reranking alone cannot recover documents missed by initial retrieval. Elastic's explanation of semantic reranking

How RAG generates answers

Retrieval-augmented generation (RAG) retrieves external material and provides selected evidence to a generative model. The system builds context from the question, evidence text, and sources, then the model generates an answer. The paper by Lewis and colleagues is a representative early study combining retrieval and generative models. Original RAG paper

flowchart LR
    Q["Question"] --> S["Retrieve relevant candidates"]
    S --> C["Rerank if needed · Remove duplicates"]
    C --> P["Build context from evidence text and sources"]
    P --> G["LLM answer generation"]
    G --> V["Compare answer claims with evidence"]

The question "Is reranking enabled in this project?" requires evidence of the current configuration or operational state. A general explanation of reranking does not establish that project's state. Even a document that accurately describes an earlier configuration may not answer a question about the current state.

Check answer quality separately for retrieval success, context construction, and the generated answer's faithfulness to evidence. A source link does not mean that the document supports every sentence. The answer must reflect missing or conflicting evidence.

Agent decisions and tool execution

A workflow follows paths defined by developers. An agent gives the model more responsibility for choosing the next action from the situation and tool results. One system can combine both structures. Anthropic's distinction between workflows and agents

flowchart TD
    USER["User request"] --> MODEL["Model selects the next action"]
    MODEL --> CHECK["Executor checks arguments, permissions, and constraints"]
    CHECK --> TOOL["Run search, file operations, or external APIs"]
    TOOL --> RESULT["Return result or error"]
    RESULT --> MODEL
    MODEL --> DONE["Check termination conditions and deliver results"]

Even when a model outputs data requesting search_knowledge, a program performs the actual search. The executor checks argument formats and permissions, then applies error handling and retry rules. The execution structure must also include termination conditions and duplicate execution prevention.

Finding documents, summarizing them, and saving a draft involves retrieval, evidence reading, writing, saving, and reading the result back. An agent can coordinate this process and verify the actual saved result. RAG can be a retrieval and generation component within it. A RAG system that only retrieves and answers is not necessarily an agent.

Connecting functions with APIs and MCP

An API defines a contract for calls between programs. For an HTTP API, this contract includes addresses, methods, request and response formats, and error handling. Embedding, generation, and search APIs expose different functions.

Ollama is a runtime that runs models and exposes them through APIs. Model Context Protocol (MCP) connects AI applications with servers that provide tools and resources. Host and server implementations handle access policies and actual execution. MCP architecture

The following illustrative JSON compares Ollama embedding and generation request formats. EMBEDDING_MODEL and CHAT_MODEL are placeholders for installed model tags. These API requests were not executed for this article. Retrieval inputs must follow the selected model's conventions.

POST /api/embed requests a vector for each input. The response's embeddings field contains a numeric array for each input. Ollama Embed API

{
  "model": "EMBEDDING_MODEL",
  "input": ["Document content to retrieve", "The user's question"]
}

POST /api/chat accepts messages and generates a response. Setting stream: false requests the response all at once. Ollama Chat API

{
  "model": "CHAT_MODEL",
  "messages": [
    {
      "role": "user",
      "content": "Evidence: The example system combines BM25 and vector search results. Question: How is search configured? Answer only from the supplied evidence."
    }
  ],
  "stream": false
}

In actual RAG, retrieved text and sources replace the example evidence string. A generation API call does not automatically perform retrieval or source verification. Even when MCP connects a search tool, the system must read its results and use them in the answer.

Data models and knowledge graphs

Data modeling defines what information to store, in which units, and with which relationships. Separating documents from chunks allows the system to manage both whole-document versions and individual evidence locations. Concept identifiers, types, and aliases complement names to handle ambiguous names and different expressions.

A knowledge graph stores entities or concepts and their relationships. Connecting "Service A uses store B" to source evidence allows traversal along that stored relation. Names appearing together in a document do not justify creating a usage relation.

Vector search finds candidates with related expressions, while graph traversal follows explicitly stored relations. GraphRAG designs can also use retrieved relations, subgraphs, or summaries as evidence for generation. Evaluate any quality improvement against relation extraction quality and question types. Storing relations does not necessarily require a dedicated graph database.

An ontology defines concept types and the meaning of relations in a domain. For example, it can distinguish software from algorithm and define what implements means. OWL is one standard language for formally expressing ontologies; not all ontologies use it. W3C OWL 2 Primer

An application can separately apply validation rules for types and relation directions. A relation can pass type checks and still disagree with the source. Checking a relation's structure and checking whether the source supports it are separate tasks.

An extraction model's confidence cannot be treated as the probability of truth without calibration and validation. Link relations to evidence chunks and input versions. These links allow the system to find and update relations derived from earlier content when a document changes.

Mapping the components to personal-docs

This section summarizes the 2026-10-22 verification record in the source material. Its reference commit is 54bd65f3f529d41316c4d2bdcff66518e21349c9. The project's code and operational environment were not rechecked while writing this article. Read the following details within the scope of that record.

personal-docs provides documents from a personal knowledge base (PKB) through search tools. The source material separates document preparation, retrieval, graph extraction, and answer generation by the host AI. These correspond to the components above as follows.

The source material cites document processing, retrieval, and the graph processing directory at that fixed commit. Repository access may be required to read these links.

According to the record, graph_query finds starting concepts using names, aliases, and embeddings, then expands stored relations. Evidence in its results contains locations such as doc_id and chunk_index, rather than source text. To inspect the actual evidence sentences, read the relevant chunks through get_document with chunk_range.

The source material also records successful test requests to Windows Ollama. However, despite a successful curl request from the Mac, the project's Python environment returned No route to host. This record cannot establish stability across all document processing or a completed project connection.

Check separately whether the model server responds, whether the application can reach it, and whether the actual processing path uses that server. Completed document ingestion also does not mean that extraction of every concept and relation is complete. Recording each stage separately makes the scope of completion clear.

Verifying each stage

Not every document pipeline stage uses a generative LLM. File parsing, hash calculation, BM25, RRF, and type checks can use ordinary code or search algorithms. Even when embeddings and reranking use models, their roles differ from those of generative models.

To find incident response procedures in an operational PDF, first check that tables and pages were extracted correctly. Then check that chunks retain the required sections, retrieval finds them, and the model uses that evidence. Changing only a generation prompt or embedding model does not restore relationships lost through incorrect table parsing.

Measure performance separately for parsing, embeddings, retrieval, and generation. Embedding throughput depends on chunk count, length, and batch configuration. Generation time depends on factors such as input and output length and whether the model is loaded. One short test request cannot determine the processing time for all documents.

Retrieval recall, ranking quality, answer faithfulness, and latency and failure rates by stage are different metrics. A single model score cannot represent the quality of the entire system. See DPR-based dense retrieval and CrossEncoder reranking for further retrieval details.

Summary

AI knowledge systems create retrieval units and evidence from original documents, then connect them to questions to produce answers or task results. Parsing, chunking, embeddings, retrieval, and generation have different roles, and each stage's inputs and outputs need verification. Add knowledge graphs and agents according to the questions and tasks that require them.

RAG provides retrieved evidence to a generative model but does not automatically guarantee answer accuracy. A valid relation type or successful API request alone cannot establish faithfulness to the source or completion of the entire process. Explain quality and completion through original evidence, actual execution paths, and verification results at each stage.