RAG vs long-context: Do you still need retrieval after a million tokens?

A customer came to us last quarter with Gemini's marketing materials and asked directly: "Since models can now handle a million tokens, couldn't I just paste my entire knowledge base into the context window? Why do we need to spend money building RAG?"

It's a fair question. The answer isn't "you can't", it's "it depends, and the actual constraints are narrower than you probably think."

The conclusion: long-context isn't a replacement for RAG; it's a special case of it. When your total knowledge fits reliably in a single context window and every query genuinely needs the full picture, long-context is simpler. But once your knowledge base grows, queries become frequent, or you need auditability, RAG almost always wins. This distinction matters for architecture decisions.

Four dimensions determine the outcome: cost, latency, accuracy, and operational maintainability. These are the factors that actually matter when teams evaluate RAG versus long-context.

RAG vs long-context: four dimensions compared

Start with cost. Long-context re-tokenizes and re-bills your entire knowledge base on every query. If your knowledge base is 800,000 tokens and you're handling 5,000 queries a day, even with prompt caching, the token bill for long-context is orders of magnitude higher than RAG. RAG retrieves just 2 to 8 relevant chunks per query, usually 2,000 to 8,000 tokens total. That's not incremental savings; it's a different cost class.

Latency follows the same pattern. More tokens mean slower time-to-first-token. A million-token prompt typically takes 20 to 40 seconds to produce the first token. For customer service or conversational interfaces, that's unusable. RAG moves the heavy work offline, vectorization and retrieval happen before the query reaches the model. Your context stays small. Responses come back in 1 to 2 seconds.

Accuracy is where intuition breaks down. Most people think "put everything in" means "the model sees everything." It doesn't. Long-context suffers from the lost-in-the-middle problem: when critical information is buried in the document middle, recall drops sharply. RAG keeps the model focused by feeding only high-relevance chunks. But RAG has its own failure mode, if retrieval misses, the model can't answer. Accuracy depends on retriever quality.

Operational maintainability determines long-term survival and gets systematically underestimated. When knowledge updates, RAG re-embeds only the changed documents. Long-context requires rebuilding the entire prompt and revalidating. For teams that need source attribution, financial services, healthcare, RAG provides built-in provenance. Long-context generates text, but tracing that text back to a source document is difficult.

Where long-context actually works

Long-context has a clear use case. Clause-by-clause analysis of a single long contract. Cross-page financial reasoning in a single document. Any task that genuinely requires seeing the whole document to answer, these are scenarios where retrieval breaks reasoning. For these cases, the whole document goes through long-context.

Most production systems we see now use a hybrid approach. RAG narrows down candidate documents from a large knowledge base. Then complete candidate documents go into long-context for deeper reasoning. Retrieval handles breadth; long-context handles depth. It's division of labor, not either-or.

We calculated this for the customer who asked the question. Pure long-context would cost them the equivalent of half their annual RAG team budget just in token spend, and latency would fail in customer service scenarios. That changed the conversation.

When we work on deployments, we don't start by asking whether to use RAG or long-context. We measure three things: knowledge base size, query frequency, and whether you need auditability. Those three numbers determine the architecture. Both approaches look good in a demo. What matters is what survives tens of thousands of daily queries and passes an audit review.