What Is RAG Evaluation (eval)? Your Checklist of Answer Quality Metrics to Measure Before Go-Live
A polished demo doesn't mean you're ready to ship. We ran an evaluation before deploying a knowledge system for a manufacturing company and found only 68% faithfulness (roughly one in three answers was confidently making things up). This piece breaks down what RAG evaluation actually is, the three core metrics you need to measure before go-live (faithfulness, relevance, and retrieval recall), and a straightforward claim: without evaluation, don't go live.
By
Tenten AI 研究團隊
AI 基礎設施
Published
January 10, 2026
Read time
6 分鐘
RAG evaluation (RAG eval) means using a set of quantifiable metrics to systematically assess every step of your retrieval-augmented generation system, from document retrieval to answer generation, including whether it's finding the right documents, sticking to what those documents say, and whether answers actually address the question. It's not about sampling a few examples by hand and calling it good.
We learned this the hard way.
We deployed a knowledge Q&A system for a manufacturing company. Demo day went smoothly, and the exec greenlit production on the spot. Before we went live, we ran an evaluation and got 68% faithfulness. Translation: one out of every three answers pulled information that wasn't actually in the source documents the model cited. It wasn't that the model couldn't answer. It was confidently hallucinating. If we'd shipped it that day, production floor engineers would have been the first to suffer, adjusting machine settings based on a wrong answer about torque specs.
No evaluation, no launch.
What RAG evaluation actually measures
To understand RAG evaluation, you need to separate the two stages of RAG. First comes retrieval: a user asks a question, and the system pulls the most relevant documents from your knowledge base. Then comes generation: the model uses those documents to write an answer.
Both stages can fail, and they fail in different ways. Retrieval might pull the wrong documents, leaving a powerful generator with nothing to work with. Generation might pull the right documents but add detail that isn't there; hallucinating confidently from partial information. RAG eval tells you which one broke. I've watched teams spend three weeks refining prompts, only to realize the actual problem was retrieval all along.
A useful evaluation covers at least three core dimensions.
Three core metrics you must measure before go-live
Faithfulness. Every claim in an answer should be traceable to the retrieved documents. This catches hallucination. Low faithfulness means the model is adding detail that isn't in the source. For finance or healthcare, this is a regulatory risk.
Answer Relevancy. The answer should address what the user actually asked. Sometimes a model faithfully quotes documents but misses the question entirely. The cited material is accurate, but it doesn't answer what was asked. Answers can be technically correct but miss the point.
Retrieval Recall and Precision. Recall measures whether you're pulling the documents that contain the answer. Precision measures how many of the retrieved documents are actually relevant versus noise. If retrieval fails, generation has no good material to work with.
Together, these metrics show you what went wrong:
| Metric | Measured at | Low Score Means | Typical Fix |
|---|---|---|---|
| Faithfulness | Generation | Model is hallucinating | Tighten prompts, require citation of sources |
| Answer Relevancy | Generation | Answering the wrong question | Rewrite instructions, add query rewriting |
| Context Recall | Retrieval | Key documents aren't being retrieved | Adjust chunking, change embeddings, add reranking |
| Context Precision | Retrieval | Too much noise diluting the signal | Raise similarity threshold, add filtering |
Read these across and you can pinpoint most common failures. Bad answers with high faithfulness and low recall point to retrieval. High recall but low faithfulness points to generation. Both solid but low relevance usually means the prompt didn't capture what the user actually wanted.
How to actually run an evaluation
You need an evaluation set: real questions people will actually ask, paired with verified answers and the source documents that should be retrieved to answer them. Start with 50 to 100 questions, not thousands. Each one should be something someone will actually ask in production. The hardest part is covering the long tail: obscure but critical queries where recall usually breaks.
Most scoring now uses LLM-as-a-judge: you give another model a rubric and let it score each question. This approach has bias, so sample results and have domain experts verify the scores against human judgment. This calibration step is often skipped, but skipping it leaves you with numbers that don't mean anything.
Evaluation doesn't stop at launch. Your knowledge base evolves, models get updated, and user questions shift. Build evaluation into your CI so every change triggers a full run. Set a threshold and block deployment if scores drop below it. This turns hallucination control from a one-time check into continuous practice.
Evaluation needs to be defined early: ideally in week one, not as a pre-launch checklist. A polished demo doesn't prove anything works. What matters is maintaining faithfulness and recall on your test set while real people actually use the system every day.
One stuck workflow
is enough to begin
Tell us what the team does today, where it breaks down, and what a better working day should look like.