RAG vs long-context: will million-token windows make RAG disappear?

A compliance-focused customer recently asked us: 'If the model can handle a million tokens, why don't I just dump my entire internal regulatory framework into it? Doesn't that mean RAG can retire?'

It's a question we hear often. The industry gets it wrong more often than not.

Long-context won't make RAG disappear. It changes how you chunk. They're not alternatives. They're different solutions across cost, latency, and recall. The actual choice isn't 'RAG or long-context,' but 'for this query, how much should I spend, how long should I wait, and how much recall loss can I tolerate.'

What each one actually solves

RAG (retrieval-augmented generation) works like this: split your knowledge base into segments, index them, then pull only the most relevant bits at query time. Long-context is different. It relies on the model's window to accept most or all of your raw content in a single prompt.

One works like a librarian pulling out a few relevant chapters. The other puts the entire library on your desk and asks you to find what you need. The second sounds safer. The real problems appear in three areas.

Cost, latency, and recall compared

The table shows the tradeoff clearly. Both have serious limitations. RAG's weakness is recall. Long-context struggles with cost and the 'lost in the middle' effect.

Larger windows change chunking, not RAG

Most people assume bigger windows mean no retrieval. Actually, larger windows change what you need from chunking. You don't have to cut documents into tiny pieces anymore.

With 4K or 8K windows, we had to cut documents into 300- or 500-token chunks to fit them. The cost was high. A single argument spanning three chunks might only retrieve the middle one, leaving an incomplete answer. This semantic breakage is RAG's biggest recall weakness.

With 100K to million-token windows, the strategy changes. Chunk larger. A retrieval pass can pull 3 to 5 complete chapters or even full contracts. It narrows scope from a hundred documents to five. The long-context model then reads those five in detail. Retrieval still exists, but in a different role.

This pattern is called coarse chunk plus retrieve-then-read. Retrieval keeps costs and latency under control. Large windows restore recall. We rebuilt a manufacturer's maintenance knowledge base using this approach. Answer completeness improved significantly. Token costs fell to less than one-tenth of the full-dump method.

How to choose

Two clear cases. Use RAG with large chunks if your knowledge base will grow, changes frequently, needs attribution, and faces high query volume. Use long-context directly if documents are one-time use, the total fits in your window, and you need complex reasoning across the entire text, like finding every inconsistency between two contracts. In that case, don't build an index.

Most enterprise scenarios fit the first case. Real-world knowledge bases aren't single documents. They're tens of thousands, and they change daily. You can't resend tens of thousands of documents with every query. Your bill and latency won't allow it.

Our experience shows this. Early on, we over-trusted large windows and skipped indexing work. The demo looked compelling. Production revealed the real cost: token bills and ten-second latencies. Users don't wait that long. We changed our approach. We don't measure success by demo quality. Instead, we ask whether someone at the customer site, under real query load, actually uses this regularly. Chunking sounds unglamorous. But it often determines whether something ships at all.