RAG was supposed to fix hallucinations. Ground the model in your own documents, the thinking goes, and it can only answer from facts. Then you ship it, and the bot cites the wrong policy version, mixes two customers’ contracts into one answer, or retrieves the right document and ignores it anyway.
Retrieval-augmented generation doesn’t remove the need for testing — it moves it. Instead of one system to evaluate, you now have three: the retrieval, the generation, and the seam between them. Each fails differently, and a test that only looks at the final answer can’t tell you which one broke. Here’s the checklist we use to evaluate RAG systems properly.
Why test retrieval and generation separately?
Because a wrong answer has two very different causes. If retrieval pulled the wrong documents, the model never had a chance — fix your chunking, embeddings, or search. If retrieval was right and the answer is still wrong, the model is ignoring or distorting its context — fix your prompt, model choice, or grounding checks. Teams that only score final answers end up guessing which knob to turn. Teams that score each stage know.
Part 1: Evaluate the retrieval
- Recall on known questions — for each test question, does the retriever surface the document that actually contains the answer? Build a set of question–document pairs and measure how often the right source appears in the top results.
- Precision and noise — how much irrelevant material rides along? Irrelevant chunks don’t just waste tokens; they actively invite the model to answer from the wrong source.
- Freshness — update a document, then ask about it. How long until answers reflect the change? Stale indexes are one of the most common silent RAG failures in production.
- Coverage gaps — ask questions your documents don’t answer. The retriever will return something anyway; what matters is what the system does with weak matches.
- Chunking sanity — check answers that live at chunk boundaries: tables split in half, a rule stated in one paragraph and its exception in the next. Bad chunking makes correct documents unusable.
Part 2: Evaluate the generation
- Groundedness, claim by claim — is every statement in the answer supported by the retrieved context? Score it sentence-by-sentence with an evaluator model; the dangerous answers are the mostly-grounded ones with a single invented detail. We covered this failure mode in depth in our guide to hallucination testing.
- Faithful refusal — when retrieval comes back weak, does the bot say “I don’t have that information” or does it improvise? Test with questions you know have no answer in the corpus.
- Citation accuracy — if the bot cites sources, verify the cited passage actually says what the answer claims. Wrong citations are worse than none: they look rigorous while being false.
- Conflicting sources — plant two documents that disagree and ask the question. A good system flags the conflict or prefers the authoritative source; a bad one silently picks whichever chunk ranked higher.
- Data boundaries — in multi-tenant systems, verify user A can never retrieve user B’s documents. This is a security test wearing a RAG costume, and it belongs in every regression run.
Part 3: Evaluate the system end to end
- Golden dataset on the full pipeline — 50+ real questions with expected answer properties, scored on every release, exactly as in our pre-launch chatbot checklist.
- Rephrasing robustness — ask the same question five ways. Retrieval is sensitive to wording in ways users never are; inconsistent answers across phrasings mean your embeddings are doing the hallucinating.
- Latency and cost under load — retrieval adds a network hop and context stuffing adds tokens. Measure p95 response time and cost per query at realistic traffic, not just in the demo.
- Regression on every change — re-run the whole suite when anything changes: the model, the prompt, the chunking, the embedding model, or the documents themselves. Every one of those can silently move your quality numbers.
The metrics that matter
You don’t need twenty dashboards. Four numbers, tracked per release, tell most of the story: retrieval recall (right document found), groundedness rate (answers supported by context), refusal correctness (says “I don’t know” exactly when it should), and p95 latency. When one of them moves, you know which part of the pipeline to open up.
Where to start this week
- Write 30 question–document pairs from your real corpus and measure retrieval recall
- Add 10 questions your documents can’t answer and check the bot refuses instead of improvising
- Spot-check 10 production answers claim-by-claim against their retrieved sources
- Automate all of it into your CI so the numbers update on every change
Or let us build your RAG evaluation for you
Evaluating RAG systems — retrieval metrics, groundedness scoring, and eval pipelines wired into CI — is part of our AI & LLM testing service. Book a free strategy session and we’ll look at your pipeline together. It’s free, and the whole conversation is about you.