Why does RAG still hallucinate after connecting a knowledge base? Five root causes and control strategies

We connected a vector database and set up RAG. Why is it still making things up?

This question opened a conversation with a financial services customer last quarter. They had connected their compliance assistant to over three thousand regulatory documents. The demonstration answered questions correctly. Two weeks after launch, their risk manager reviewed a sample of responses and found that roughly one in seven cited a clause number that doesn't actually exist. The system was functioning as designed. The issue was that the organization's expectations for what RAG could accomplish had been wrong from the beginning.

RAG hallucination occurs when a model has access to an external knowledge base and performs retrieval, yet still generates answers that diverge from the retrieved sources or invents content that doesn't exist in those sources. In simpler terms, RAG reduces the probability of hallucination. It doesn't eliminate it. Anything that breaks in the retrieval-to-generation pipeline forces the model back into what it does best: delivering answers that sound fluent but may not be true.

Why RAG hallucination happens: the problem isn't the model, it's the entire retrieval chain

Most teams attribute hallucination to model limitations, so they try upgrading to a larger model. That solves less than 20 percent of these cases. The actual root causes sit at different points in the retrieval pipeline. They divide into five categories.

Retrieval failure is the first. The answer to the user's question doesn't exist in the chunks returned. The embedding model might not recognize domain-specific terminology. Synonyms might not align: the user searches for severance while documents say separation pay. The top-k parameter might be set too low and exclude the right passage. The model receives irrelevant material, is asked to respond, and invents content to fill the gap. This is the most common failure mode and the one most often mistaken for poor model capability.

Source conflict is the second. Your knowledge base contains both the 2022 version and the 2024 revision of the same policy, or two departments have different procedures. A single retrieval returns three conflicting documents. The model cannot determine which version is current, so it assembles an answer from fragments of each, something that looks complete but never actually existed.

Improper chunking is the third. A table gets split down the middle: headers remain in one chunk, values land in the next. An exception clause reading "except in the following cases" gets severed, leaving the main provision and exceptions in separate chunks. The model reads only half a sentence and draws a conclusion. Chunking isn't a minor technical detail. It directly determines whether the model understands the full context.

Unconstrained prompts is the fourth. Without an explicit instruction like "answer only based on provided sources; say so if you can't find it," the model defaults to attempting a response. It will use pretrained knowledge to fill gaps, and that's where hallucinations come from. Many organizations could reduce RAG hallucination substantially with a single line added to the system prompt.

Generation that disregards retrieval results is the fifth. The correct passage was successfully retrieved, but the model trusts its internal knowledge more, especially when the source material contradicts what it learned during pretraining. This is particularly costly in specialized domains, where company policies diverge from general industry practices.

Controlling hallucinations requires measurable feedback loops

Once you've mapped the five failure modes, the critical work begins. You can't solve hallucinations in one revision cycle. You can only reduce them to an acceptable error rate and maintain continuous monitoring. This means building a golden set before launch, a few hundred real questions with verified correct answers. Every time you make a change (embedding swap, chunking adjustment, prompt revision), run it through the golden set. Measure retrieval success rates and citation accuracy. Don't rely on intuition that things are improving.

In actual customer projects, time spent on evaluation and ongoing maintenance typically exceeds time spent on the initial build. A correct answer to ten demo questions proves nothing. What matters is whether the error rate remains stable across the first ten thousand production questions and whether people actually trust the system enough to use it. This is why we maintain long-term engineer engagement, treat hallucination as a permanent production metric requiring continuous supervision, and don't consider the project complete on delivery day.