What RAG system architecture really looks like: complete data flow breakdown from document to answer

A RAG (Retrieval-Augmented Generation) system architecture lets large language models retrieve relevant content from a company's own knowledge base before answering. At its core is a five-stage data pipeline: ingest → chunk → embed → retrieve → generate.

This sequence is straightforward to memorize. Each stage can fail independently, and the failure point usually isn't where you expect it.

Last month at an equipment maintenance company, we saw this problem firsthand. They'd invested heavily in GPT and built a RAG system, but when customer service asked about Model A's warranty terms, the system returned Model B's. The engineering team's first response was to blame the model, they wanted to upgrade to a larger version. When we traced back through the system, the problem wasn't in generation; it was at the very start. The warranty PDFs were scanned images, and the ingest stage was extracting only garbage characters. No model, no matter how capable, can work with information it never actually read.

Understanding the complete architecture shows where each stage typically fails and identifies the real cost drivers.

RAG system architecture: five data pipeline stages from document to answer

Picture a PDF contract that needs to become one customer service response: "Your coverage runs through next March." Here are the five stops it has to take.

Ingest. Pull raw files from everywhere: PDFs, Word documents, web pages, databases, Confluence, cloud storage. The task is to turn unstructured content into clean text. This stage is critical but widely underestimated. Scanned documents without OCR, tables flattened into single lines, page headers and footers leaking into body text, merged cells in Excel that parse wrong, every mistake at this stage cascades downstream. It's where garbage gets locked in.

Chunk. Split long text into chunks so the system can compare them one by one. Most teams just pick a rule like "cut every 500 characters" and move on. Then it breaks: a warranty clause gets sliced in half, first half goes to chunk 3, second half to chunk 4. When retrieval pulls that chunk, you only get half a sentence. The answer comes back incomplete. Chunking isn't about character count; it's about cutting at semantic boundaries.

Embed. Take each chunk and run it through an embedding model to turn it into a vector, a set of numbers representing its semantic position. The hidden cost: which embedding model you choose determines whether your system actually understands Chinese. Many open-source models struggle with Traditional Chinese and industry terminology, part numbers, GMP, policy codes. Two passages that mean nearly the same thing end up far apart in vector space, and your retrieval starts missing what it should find.

Retrieve. The user asks a question, that question gets vectorized, and you search your vector database for the most similar chunks. This is where your entire RAG system's quality lives or dies. Pull too few chunks and you miss critical information; pull too many and you flood the system with noise. That equipment company we mentioned? Pure vector similarity was their only tool, but "Model A" and "Model B" are so semantically similar the system couldn't tell them apart. You need hybrid search, keyword retrieval combined with vector retrieval, plus a reranking layer on top to surface the actually correct passage.

Generate. Combine the retrieved chunks with the question, feed them all to the LLM, and ask it to answer based only on that information. The failure modes here are hallucination and overreach: wrong information goes in and the model confidently makes something up anyway, or the data simply isn't there but the model invents a plausible sentence. Good generation stages require the model to cite sources, and when data is missing, to actually say "not found" instead of inventing.

Where each stage usually breaks

The following table collapses the pipeline into a reference so every failure point is obvious:

When RAG systems return the wrong answer, most people blame the model and want a bigger one. But the failure is usually in ingest, chunk, or retrieve, stages before the model even speaks. Upgrading to a larger model often throws money at the place you need it least.

Demos always work. Production doesn't always.

RAG systems have a persistent failure pattern: demos almost always succeed. Pick three clean documents, ask five questions you already know the answers to, and results look polished. The real test is connecting it to a knowledge base that's been accumulating for ten years, has formatting scattered everywhere, contains duplicate versions, and includes scanned images. Run hundreds of messy real-world questions through it every day. Your accuracy drops from ninety percent in the demo to fifty percent in production.

That gap doesn't close with a stronger model. It closes with real work at every single stage: choosing the right parser for ingest, chunking according to document structure, picking embeddings that understand Chinese, adding hybrid search and reranking to retrieval, binding source citations in generation. Each stage needs to be measured, then fixed, one by one.

When we build enterprise RAG systems, we don't start by optimizing the model. We pull a batch of real questions people actually ask, use them as a test set, and measure retrieval accuracy from the beginning, starting at ingest. Where answers fail often matters more than the failure itself. A demo doesn't prove anything. Real success is when you plug into that messy knowledge base and the answers still hold up.