Testing AI Agents: Building an Evaluation Set That Earns Its Keep
The question that separates agents that ship from agents that linger in pilot is simple: how do you know whether yesterday\u2019s change made it better? Without an answer, every prompt edit is a guess, and every regression is discovered by a customer.
An evaluation set is the answer, and it does not require a platform. Fifty cases in a file, a script that runs them, and a grading rule per case will tell you more than any leaderboard, because they are your cases.
What a case looks like#
A case is an input, the starting state of the world, and a checkable expectation. The expectation is almost never an exact string — an agent can be right in several phrasings. Grade the outcome instead: did it call the refund tool with order 4471; did the final answer contain the correct delivery date; did it decline to act and ask a question, as it should have. Where prose quality matters, a rubric graded by a model is acceptable, but keep it to a few explicit criteria, and spot-check the grader by hand from time to time.
Store the state each case needs — the fixture — with the case. A test that only passes on a Tuesday because of live data is not a test.
Fifty cases, and where they come from#
| Source | Roughly how many | Why |
|---|---|---|
| Common real requests from logs | 20 | Protects the everyday path |
| Known past failures | 10 | Stops regressions returning |
| Ambiguous or under-specified inputs | 8 | Should ask, not guess |
| Should-refuse cases | 6 | Out of scope, unauthorised, unsafe |
| Empty or broken tool results | 6 | The most common real incident |
Grade outcomes, not paths#
Two runs that reach the same correct result by different routes are both correct, and a test suite that insists on one transcript will fail constantly for no reason. Assert on what changed and what was said: the tool calls that had side effects, the final answer’s key facts, whether a human gate was requested. Keep the full trace stored for debugging, but do not assert on it — otherwise every prompt improvement looks like a hundred failures.
The four numbers to track over time#
- Task success rate on the whole set, and separately on the should-refuse subset.
- Unsupported claim rate — answers with facts not present in retrieved evidence.
- Median and 95th-percentile cost and latency per run.
- Human intervention rate: how often a person had to step in, and why.
Run it in the pipeline, and again after launch#
Run the set on every change to prompts, tools, model version or retrieval configuration — those four are the only things that alter behaviour, and all four change more often than people expect. After launch, keep sampling real traffic: pick a few runs a day, grade them by hand, and add anything surprising to the set. An evaluation set that stops growing stops representing your users, usually within a quarter.
Frequently asked questions
How many cases do I need to start?
Fifty is enough to catch real regressions and small enough to write in a couple of days. Twenty is enough to begin. The number matters far less than covering the awkward categories — ambiguous inputs, should-refuse cases and empty tool results.
Can I use a model to grade the outputs?
Yes, with care. Give it explicit criteria rather than asking whether the answer is good, keep the rubric short, and hand-check a sample regularly. Model graders drift, and a grader that has drifted will happily approve a regression.
Should evaluation block deployment?
Block on the safety-critical subset — should-refuse cases and anything involving irreversible actions. For general quality, track the trend and require a human decision on a drop rather than an automatic block, because a small movement can be noise.
evaluating ai agentsagent testingllm evaluation setregression testing llmagent quality metrics
