The Knowledge Cutoff Problem
Every AI language model has a secret weakness, and it is printed right on the label if you know where to look: the knowledge cutoff date.
When a model is trained, it absorbs information from its training data — books, articles, websites, and other text. But that training process ends on a specific date. After that date, the model knows nothing. It is like hiring a brilliant consultant who has been living on a remote island with no internet since 2024. They might be extraordinarily knowledgeable about everything up to that point, but ask them about something that happened last month, and they will either admit they do not know or — more dangerously — make something up.
This is not a minor limitation. In a world where information changes daily — stock prices, company policies, scientific discoveries, news events, product specifications — a model with a fixed knowledge cutoff is fundamentally unable to provide current information. And for many real-world applications, current information is exactly what people need.
This chapter introduces Retrieval-Augmented Generation, or RAG — the technique that solves this problem by giving AI models the ability to search for and use up-to-date information on the fly.
What Is RAG?
RAG stands for Retrieval-Augmented Generation. Let us break that down:
- Retrieval means searching for and finding relevant information from external sources.
- Augmented means enhanced or supplemented — in this case, the AI's response is enhanced with retrieved information.
- Generation refers to the AI generating its response, as language models always do.
Put simply, RAG is a technique where the AI searches for relevant information before answering your question, then uses that information to generate a better, more accurate response.
Think of it like the difference between a closed-book exam and an open-book exam. Without RAG, the AI is taking a closed-book exam — it can only rely on what it memorized during training. With RAG, it gets to flip through the textbook first, finding the specific pages relevant to the question before writing its answer.
Here is how a RAG system works in practice, step by step:
- You ask a question. For example: "What were Acme Corporation's quarterly earnings last quarter?"
- The system searches for relevant information. It looks through a database of documents — which might include Acme's latest financial reports, press releases, and analyst summaries — and retrieves the most relevant passages.
- The retrieved information is added to the AI's context. The relevant passages are essentially inserted into the prompt, so the AI can "see" them while generating its response.
- The AI generates an answer using both its training knowledge and the retrieved information. It combines its general understanding of finance and language with the specific, current data from the retrieved documents.
The result is an answer that is both well-written (thanks to the language model's capabilities) and factually grounded in real, current data (thanks to the retrieved information).
Vector Databases and Embeddings: The Search Engine Behind RAG
The retrieval step is where the technical magic happens. RAG systems do not use traditional keyword search the way Google did in the early 2000s. Instead, they use something called semantic search, powered by vector databases and embeddings.
What Are Embeddings?
An embedding is a way of representing a piece of text as a list of numbers — typically hundreds or thousands of numbers. These numbers capture the meaning of the text, not just the specific words used.
Here is an analogy. Imagine you wanted to describe every city in the world using just two numbers: temperature and population. New York might be (55, 8400000). Phoenix might be (75, 1600000). Each city gets a pair of numbers — its "coordinates" in this simplified space.
Now imagine doing something similar, but instead of two numbers, you use 1,500 numbers, and instead of describing cities, you are describing the meaning of sentences. Each number captures a different dimension of meaning — some might relate to topic, others to sentiment, others to specificity, and many to subtleties that do not have easy human labels.
The crucial property of embeddings is that texts with similar meanings end up with similar numbers. The embedding for "How do I make pasta?" will be numerically close to the embedding for "What is a good recipe for spaghetti?" even though the two sentences share almost no words. This is what makes semantic search possible — you are searching by meaning, not by keywords.
What Is a Vector Database?
A vector database is a specialized storage system designed to hold millions or billions of embeddings and find the ones most similar to any given query, very quickly.
When you set up a RAG system, you first process all your documents by:
- Splitting them into chunks. A long document might be divided into paragraphs or sections of a few hundred words each.
- Creating an embedding for each chunk. Each chunk gets converted into its list of numbers.
- Storing the embeddings in a vector database. Each embedding is stored alongside the original text it came from.
When a question comes in, the system creates an embedding for the question, then asks the vector database: "Which stored embeddings are most similar to this question embedding?" The database returns the top matches — the chunks of text most likely to contain relevant information.
This process is fast. Modern vector databases can search through millions of documents in milliseconds, making RAG practical for real-time applications.
Why Not Just Use Regular Search?
You might wonder why we need this elaborate embedding and vector database system. Why not just search for keywords the traditional way?
The answer is that keyword search misses meaning. If your company's policy document talks about "compensation adjustments" and a user asks about "salary changes," keyword search would not connect the two. Semantic search, powered by embeddings, understands that these phrases mean essentially the same thing.
Keyword search also struggles with context. The word "bank" means very different things in "I went to the bank to deposit a check" and "I sat on the river bank." Embeddings capture this contextual meaning, while keyword search treats both as identical.
That said, the best RAG systems often combine both approaches — using semantic search to capture meaning and keyword search to catch specific terms, names, and numbers that embeddings might not prioritize.
How Search and Generation Work Together
The interplay between the retrieval step and the generation step is what makes RAG so effective. Let us trace through a more detailed example.
Suppose a legal professional asks: "What are the key provisions of the new data privacy regulation that took effect this year?"
Step 1: Query processing. The system takes the question and creates an embedding that captures its meaning — it is about data privacy, regulations, recent changes, and key provisions.
Step 2: Retrieval. The vector database is searched. In this case, the database might contain thousands of legal documents, regulatory filings, and analysis papers. The search returns the five or ten most relevant passages — perhaps sections of the actual regulation, a law firm's summary, and a regulatory agency's FAQ.
Step 3: Context assembly. The retrieved passages are assembled into a prompt for the language model. This prompt typically looks something like: "Based on the following information, answer the user's question. [Retrieved passage 1] [Retrieved passage 2] [Retrieved passage 3]... User question: What are the key provisions of the new data privacy regulation?"
Step 4: Generation. The language model reads all of this and generates a coherent, well-organized response. It synthesizes information from multiple retrieved passages, organizes it logically, and presents it in clear language.
Step 5: Citation. Good RAG systems include citations or references back to the source documents, so the user can verify the information. The response might say "According to Section 4.2 of the regulation..." with a link to the original document.
This pipeline means the final answer has two important properties: it is written in natural, helpful language (the generation part), and it is grounded in actual, retrievable source documents (the retrieval part).
Why RAG Reduces Hallucinations
Hallucination — the tendency of AI models to generate plausible-sounding but false information — is one of the biggest problems in deploying AI for real-world tasks. RAG significantly reduces this problem, though it does not eliminate it entirely.
How Hallucinations Happen
Without RAG, when a language model does not know the answer to a question, it does not say "I don't know." Instead, it generates the most plausible-sounding response based on patterns in its training data. This is like asking someone who does not know the answer to fake it convincingly — they might sound confident, but the information they provide could be completely wrong.
How RAG Helps
RAG reduces hallucinations in several ways:
Grounding in source material. When the model has actual documents to reference, it is much more likely to base its answer on real information rather than making things up. The retrieved passages act as an anchor, keeping the model tethered to facts.
Reducing the need to guess. Many hallucinations occur when the model is asked about topics outside its training data. RAG brings relevant information into the model's context, so it does not need to guess.
Enabling verification. Because RAG systems can cite their sources, users can check whether the model's claims are actually supported by the retrieved documents. This creates an accountability mechanism that pure generation lacks.
Constraining the output. Well-designed RAG systems instruct the model to only answer based on the provided information and to say "I don't have enough information" when the retrieved documents do not address the question. This dramatically reduces the model's tendency to fill in gaps with fabricated details.
RAG Is Not a Complete Solution
It is important to be honest about limitations. RAG can still produce errors when:
- The retrieval step finds the wrong documents (garbage in, garbage out)
- The model misinterprets the retrieved information
- The source documents themselves contain errors
- The question requires synthesizing information in ways that go beyond what was retrieved
RAG brings hallucination rates down significantly — some studies show reductions of 50 percent or more — but any organization deploying RAG should still have verification processes in place for critical information.
Enterprise Use Cases
RAG has become one of the most widely adopted AI techniques in business settings, precisely because it solves the knowledge cutoff problem and reduces hallucinations. Here are some of the most common applications.
Internal Knowledge Bases
Large organizations have vast amounts of institutional knowledge scattered across documents, wikis, emails, Slack messages, and shared drives. Finding specific information can be like searching for a needle in a haystack.
RAG-powered internal search systems let employees ask natural language questions — "What is our refund policy for enterprise customers?" or "How did we handle the server outage last March?" — and get synthesized answers drawn from the company's own documents, with links to the sources.
Customer Support
Customer support teams deal with a constantly evolving set of products, policies, and procedures. RAG systems can power AI assistants that answer customer questions by searching through up-to-date product documentation, known issues databases, and policy documents. When a customer asks about a feature that was just released last week, the RAG system can find the relevant documentation and provide an accurate answer, even though the underlying language model has never heard of that feature.
Legal and Compliance
Legal teams use RAG to search through contracts, regulations, case law, and compliance documents. Instead of manually reading through hundreds of pages, an attorney can ask "Does this contract have a non-compete clause, and if so, what are its terms?" The system retrieves the relevant sections and generates a clear summary.
Research and Analysis
Research teams in fields ranging from pharmaceuticals to finance use RAG to stay on top of rapidly evolving bodies of knowledge. A researcher can ask about the latest findings on a specific drug interaction, and the system will search through recent papers and clinical reports to provide a current answer.
Technical Documentation
Software companies use RAG to help developers navigate complex technical documentation. Instead of searching through hundreds of pages of API references, a developer can ask "How do I authenticate with the payment processing API?" and get a concise answer with code examples drawn from the actual documentation.
Building a RAG System: The Key Decisions
If you are involved in deploying AI at your organization, understanding the key decisions in building a RAG system can help you evaluate vendors and proposals. Here are the choices that matter most.
What Data to Include
The quality of a RAG system depends entirely on the quality of its data. Including outdated, inaccurate, or irrelevant documents will lead to outdated, inaccurate, or irrelevant answers. Organizations need to curate their document collections carefully and keep them updated.
How to Chunk Documents
The way you split documents into chunks significantly affects retrieval quality. Chunks that are too small might lose important context. Chunks that are too large might dilute the relevant information with unrelated content. Finding the right balance is part art, part science, and often requires experimentation.
Which Embedding Model to Use
Different embedding models have different strengths. Some are better at technical content, others at conversational queries. The embedding model must match the type of content and the type of questions users will ask.
How Many Results to Retrieve
Retrieving too few documents might miss critical information. Retrieving too many might overwhelm the language model with irrelevant content or exceed its context window. Most systems retrieve between three and ten documents, though the optimal number depends on the application.
How to Handle Conflicting Information
When retrieved documents contradict each other — perhaps because one is outdated — the system needs a strategy. Some systems prioritize more recent documents. Others present both perspectives and let the user decide. This decision has significant implications for reliability.
The Bigger Picture
RAG represents a fundamental shift in how we think about AI knowledge. Instead of trying to cram everything into the model's training data — which is expensive, slow, and inevitably incomplete — RAG treats the model as a reasoning engine that can search for information as needed.
This approach has several profound implications:
AI stays current. By connecting to regularly updated document stores, RAG systems can provide information that is hours or days old, rather than months or years old.
Organizations keep control of their data. Instead of sending proprietary information to an AI company for training, organizations can keep their documents in their own databases and let the AI access them through RAG. This is much better for privacy and security.
Smaller models become more capable. A relatively small, inexpensive language model augmented with RAG can outperform a much larger model on domain-specific tasks, because the RAG system provides the specialized knowledge the smaller model lacks.
Transparency improves. Because RAG systems can cite their sources, users can verify claims and build trust in the system over time. This is a major advantage over black-box AI systems that provide answers without any indication of where they came from.
RAG is not the final answer to AI's knowledge problem, but it is the most practical solution available today, and it is becoming a foundational component of almost every serious enterprise AI deployment.
See This in the News
The infrastructure behind RAG systems — particularly data platforms that store and manage the information AI models retrieve — is a hot area of investment and acquisition. For a look at how the data layer of AI is evolving, read Databricks Acquires Tabular: AI Lakehouse on AIWire.