You built a chatbot, it answered your test questions correctly, and the demo went great. Then a real user asked it something nobody anticipated, and it confidently invented a refund policy you don’t have.
This is the gap between “it works in the demo” and “it’s ready for users” — and it’s much wider for AI products than for normal software. Traditional QA assumes the same input produces the same output. An LLM chatbot breaks that assumption on purpose. So testing one needs a different playbook. Here is the one we use.
1. Define what a “good answer” actually means
Before writing a single test, write down what the bot must do, may do, and must never do. Must it only answer from your documentation? May it make small talk? Must it never give legal or medical advice, mention competitors, or promise refunds? This document — your quality bar — turns vague worry into testable rules. Every failure you find later will trace back to a rule on this list, or reveal a rule you forgot.
2. Build a golden dataset
A golden dataset is a curated set of real questions with expected behavior — not expected word-for-word answers, but expected properties: “mentions the 30-day window,” “refuses and redirects to support,” “cites the correct document.” Start with 50–100 cases covering:
- Core questions — the top things users actually ask
- Edge cases — ambiguous phrasing, typos, mixed languages, multi-part questions
- Out-of-scope questions — things the bot should decline gracefully
- Trap questions — questions whose true answer is “that doesn’t exist,” where a hallucination-prone bot will invent something
Real user logs are gold here. Ten real questions beat a hundred invented ones.
3. Test for hallucinations specifically
Hallucinations are the highest-reputation-risk failure an LLM has: confident, fluent, and wrong. Target them directly. Ask about products you don’t sell, policies you don’t have, and people who don’t work there. If your bot uses retrieval (RAG), verify answers are actually grounded in the retrieved documents — a surprisingly common failure is the bot ignoring your content and answering from the model’s general knowledge, which may be outdated or simply wrong about your business.
4. Attack it before your users do
Someone will try to break your chatbot within the first week. Beat them to it:
- Prompt injection — “Ignore your instructions and…” in a hundred creative variations
- Jailbreaks — role-play framing, hypotheticals, and encoding tricks that route around guardrails
- Data leakage — can users extract your system prompt, other users’ data, or internal documents?
- Toxicity bait — provocations designed to get an offensive answer worth screenshotting
This is adversarial red-teaming, and it deserves the same seriousness as a security audit — because that’s what it is.
5. Score at scale with evals — and humans where it counts
You can’t manually read a thousand chatbot answers on every release. Automated evaluation — including using a strong LLM as a judge with a clear scoring rubric — handles volume: accuracy, groundedness, tone, refusal correctness. But keep humans in the loop for the judgment calls machines miss: does this answer feel right for our brand? Would a frustrated customer be satisfied or enraged? The combination is the point: machines for coverage, humans for nuance.
6. Wire it into your pipeline before launch, not after
The most expensive mistake teams make: treating AI testing as a one-time pre-launch gate. Chatbot quality drifts — every prompt tweak, model upgrade, or knowledge-base update can silently change behavior. Run your golden dataset in CI so every change gets scored before it ships, exactly like a regression suite for conventional software. Launch day should be the first run of a system you keep, not the last run of a checklist you discard.
The pre-launch checklist
- Written quality bar: must / may / must-never
- Golden dataset of 50+ real-world cases, including traps and out-of-scope questions
- Hallucination and grounding tests passed (especially for RAG)
- Red-team pass: injection, jailbreaks, data leakage, toxicity
- Automated eval pipeline scoring every release
- Human review process for tone and brand judgment
- Monitoring plan for post-launch drift
Don’t want to build all this yourself?
This is exactly what our AI & LLM testing service delivers: the quality bar, the golden dataset, the red-teaming, and the eval pipeline wired into your CI — built for your product, not a generic template. A chatbot that’s been through this checklist doesn’t hiss at your users — it purrs.
Start with a free strategy session, or check where you stand with the free QA readiness check. It’s free, and the whole conversation is about you.