Why normal QA does not work
Traditional testing asserts that a given input produces a given output. LLM systems produce different output for identical input, and wrong answers look plausible. You need a different instrument — one that measures quality statistically rather than asserting equality.
Evaluation is the single most skipped component in AI projects and the one that most determines whether a system reaches production. Here is how to build it.
What you are actually measuring
| Dimension | Question | How to measure |
|---|---|---|
| Factual accuracy | Is the answer correct? | Compare against known-good answers |
| Groundedness | Is it supported by retrieved content? | Check claims against cited sources |
| Retrieval quality | Did we find the right documents? | Recall against known relevant docs |
| Refusal correctness | Does it decline when it should? | Cases it must not answer |
| Safety | Does it avoid harmful output? | Adversarial test set |
| Task completion | Did the agent finish the job? | End-state verification |
| Cost | What did each outcome cost? | Token accounting per case |
Most teams measure only the first row, and only informally. The rows that catch production failures are groundedness, refusal correctness and retrieval quality — because those are where systems degrade silently.
Building the evaluation set
- Take real queries. Support tickets, search logs, questions people actually asked. Synthetic questions test a system against your imagination rather than your users.
- Include the awkward cases — ambiguous phrasing, questions outside scope, multi-part questions, things the corpus does not cover.
- Write expected answers with a domain expert. This is the expensive part and the part that makes everything else possible.
- Mark which should be refused. Cases where the correct behaviour is declining to answer.
- Record which documents are relevant for each query, so retrieval can be scored separately from generation.
- Version it. Your evaluation set is an asset that grows as you find new failure modes.
Build this before building the agent. One day of collecting real queries and writing expected answers converts every later decision — model choice, chunking strategy, prompt change — from opinion into measurement. Teams who skip it argue about whether changes helped; teams who have it simply check.
Scoring approaches, and their limits
| Method | Good for | Limitation |
|---|---|---|
| Exact match | Extraction, classification | Useless for open text |
| Keyword or fact presence | Checking required facts appear | Misses added falsehoods |
| Semantic similarity | Rough closeness to expected | Similar wording, wrong fact scores well |
| LLM-as-judge | Scaled qualitative scoring | Verbosity bias; misses subtle errors |
| Human review | Ground truth | Does not scale; use on samples |
A practical combination: automated scoring across the full set for every change, LLM-as-judge for qualitative dimensions, and human review on a rotating sample to keep the automated scores honest.
Testing the parts separately
Score retrieval independently from generation. When an answer is wrong, you need to know whether the system failed to find the right information or failed to use it. Those have completely different fixes — chunking and search tuning versus prompting and model choice — and a single end-to-end score cannot distinguish them.
- Retrieval: was the relevant document in the top results?
- Generation: given the correct context, is the answer right?
- Tool calls: did the agent choose the right tool with valid arguments?
- End-to-end: did the whole task complete correctly?
Adversarial and refusal testing
Two categories that are routinely skipped and routinely cause the incident:
- Out-of-scope questions — the agent should decline, not improvise.
- Questions with no answer in the corpus — the most common source of confident fabrication.
- Prompt injection attempts embedded in documents the agent retrieves.
- Requests for actions beyond its permissions.
- Ambiguous queries where asking a clarifying question is the correct behaviour.
Monitoring after launch
Evaluation does not end at deployment — production is where drift appears:
- Sample live traffic and score it against the same rubric.
- Track refusal rate. A sudden change usually signals retrieval or content problems.
- Watch cost per outcome alongside quality, so trade-offs are visible together.
- Capture user feedback signals — escalations, thumbs-down, repeated rephrasing.
- Re-run the full evaluation whenever a model version changes. Provider updates can shift behaviour without warning.
Making it a gate
The discipline that makes this stick: no prompt, retrieval or model change ships without the evaluation running and the score not regressing. It is the same practice as a test suite in ordinary software, applied to a system where the failures are quieter.
Building AI without an evaluation framework, or unsure whether yours is measuring the right things? Tell us what you are deploying. See our AI agent service, why pilots stall, and AI observability.