Tuning RAG retrieval quality: seven independent techniques for recall and precision with before and after results

Last month we took over a RAG system. The customer's compliance team was frustrated. They'd asked whether a product sold abroad required registration as an accredited-investor offering, and the system returned an unrelated internal marketing pitch. The engineer shrugged and said the model wasn't smart enough. We spent two days on it. Without changing a single line of the prompt, touching only the retrieval layer, the same question surfaced the correct regulatory text.

The problem is almost never in generation. It's retrieval putting the wrong material into context.

RAG retrieval quality tuning means improving recall, whether you find what you need, and precision, whether what you find is relevant, by adjusting which passages you retrieve, how you rank them, and how you filter them. This all happens without changing the LLM or the generation prompt. These two metrics constantly push against each other. The real work is knowing when to trade one for the other.

Below are seven techniques we actually turn on, turn off, and measure independently. Each one works on its own, so you can tune them one at a time.

Hybrid retrieval: Don't trust vectors alone

Pure vector retrieval excels at semantic similarity, but it struggles with exact terms. Part numbers, statute citations, proper names, vectors miss them. When a user searches for 'A-2024-091 batch inspection report,' the vectors get pulled toward 'inspection report' and skip the part number entirely.

The fix is to run sparse retrieval, keyword-based search via BM25, and dense retrieval using vectors in parallel. Collect results from both, then fuse them, typically using reciprocal rank fusion. This tends to be the most effective of the seven techniques and has almost no downside. In that compliance case, turning on just this one technique lifted recall from 0.61 to 0.78.

Reranking: cast wide, then filter tight

First-stage retrieval prioritizes recall, so it pulls top-30 or top-50 results. Better to fetch too much than miss the answer. But stuffing 50 passages into context is expensive and messy. Precision suffers.

Stage two inserts a cross-encoder reranker that reads the query and each passage together, rescores them, and keeps only the top 5. First stage handles not missing anything. Reranking handles not including junk. It's clean separation of concerns. This step has the biggest impact on precision. We've seen it jump from 0.55 to 0.81. The cost is latency. Add 100 to 300 milliseconds per call. Whether you absorb that depends on whether your use case needs real-time answers.

Query rewriting: users won't ask nicely

Real queries are short, messy, and often pack three things into one sentence. Feed them straight into retrieval and recall will suffer.

Three rewriting approaches are all worth testing. Split a multi-intent question into separate sub-queries, retrieve each independently, then merge results. This is multi-query retrieval. Use an LLM to generate a hypothetical ideal answer first, then search for it. This is HyDE, particularly effective for factual questions. For conversational use, you need anaphora resolution, turn 'what about that' into the full noun it refers to. Rewriting pushed recall from 0.78 to 0.86.

Metadata filtering: narrow the pool first

Many retrieval errors boil down to crossing boundaries that shouldn't be crossed. A system might surface a 2021 old-version policy as current, or mix Customer A's data into Customer B's answer.

Tag your chunks with metadata: date, department, document type, access level, version. Use hard filters before semantic matching. The precision gains are often underestimated, and it's your security baseline. Tenant isolation and permission control depend on it. On one manufacturing project, adding a single current-version-only filter lifted precision from 0.66 to 0.83. Customer complaints about incorrectly citing outdated rules dropped to zero.

Chunk strategy: you can't fix bad foundations

Chunks are your foundation. Cut them too large and three topics end up in one span, tanking precision. Cut them too small and context around key sentences breaks, tanking recall.

Stop hard-cutting by character count. Respect semantic boundaries first: headings, paragraphs, bullet points. Keep tables and legal clauses as complete units. Never bisect them. Preserve modest overlap to keep key sentences from landing on a seam. Parent-child chunking works well too: retrieve with small chunks for precision matching, then return the larger parent chunk to the LLM. You get both retrieval accuracy and context completeness.

Embedding model choice: generic models don't speak your language

Healthcare, law, semiconductors. Every domain has its jargon. A generic embedding model can't distinguish between concepts that only matter in your field. Stack on domain vocabulary and your vector space turns to mush. Swap in an embedding model stronger on your language and your industry, or fine-tune one with a small labeled dataset, and you often gain 5 to 10 percentage points of recall with minimal effort.

Top-k and recall window: don't let cost-cutting break you

This last one is easiest to overlook. Set top-k too small and the answer sits at position 6 but you only retrieve the top 5. You've welded your own recall ceiling shut. The right approach is to loosen top-k in stage one, start at 20, hand the filtering responsibility to reranking, and don't save tokens by starving k.

Measurements from one system

Below is a real measurement from one system, adding techniques incrementally. Numbers vary by corpus, but direction holds across projects.

These seven techniques are not a package. They're independent adjustments. What you should do first is build an evaluation set of a few dozen questions with ground-truth answers labeled. Every time you turn a technique on, measure recall and precision to see whether it helps or hurts. At Tenten, when we implement RAG, week one usually skips building features and just builds this offline evaluation pipeline instead. Without measurement, tuning is guesswork. Getting three questions right during a demo doesn't count. What counts is the system answering hundreds of queries reliably after you ship it to users.