A demo shows that an AI agent can do a task once. Production asks a different question: does it do the task every time, and does it still do it after the prompt, the model or a tool has changed?
Evals answer that question. An eval suite is a set of your own tasks, each with a defined way to decide whether the agent's result is acceptable, run again on every change. This post covers why that is needed, how to build a suite you can trust, and where the evidence for evals is thinner than it is often presented.
Why a passing demo is not evidence
Three measured properties of language models make a single successful run a poor guide.
The same input does not give the same output. Atil et al. configured five models for deterministic output and ran the same tasks ten times each. Accuracy varied by up to 15% between runs, and none of the models delivered repeatable accuracy across all tasks.
The model can change under a stable name. Gao, Liang and Guestrin tested commercial APIs serving open models and found that 11 of 31 endpoints served a different output distribution than the reference weights. Hosted models are also retired on the provider's schedule: Anthropic, for one, commits to at least 60 days' notice. A system built on a hosted model will be moved to a different one, more than once.
Errors compound across steps. An agent plans, calls tools and acts over many turns, so a small error rate per step becomes a large one per task. In tau-bench, an agent that solved more than 60% of customer-service tasks in a single attempt solved fewer than 25% of them eight times in a row. A Princeton study published at ICML 2026 found that 24 months of model releases brought steady gains in accuracy and only modest gains in reliability.
People who run agents in production report the same. In a study of 86 deployed agent systems, reliability was the main development bottleneck, and three quarters of the teams interviewed evaluated without a fixed set of test cases. In a vendor survey of 1,340 practitioners, quality was the most cited barrier to production, and 29.5% of respondents ran no evaluation at all.
A team in that position is developing in the dark: a change to a prompt, a model or a tool may fix the case it was made for and break two others, and the first to find out is a user.
Illustration of an eval suite for a contract review agent. One case passes in a single run. Run eight times, the same case fails twice. In a suite of ten cases, each run eight times, it is the only case that does not pass every run. A fix for it passes that case but breaks two others, and the release gate blocks it. The revised fix passes all ten cases and is released. A new failure in production then becomes the eleventh case.
What the evidence does not show
We looked for a controlled study showing that teams with evals ship faster, or have fewer regressions, than teams without. We did not find one. The case for evals rests on the mechanisms above and on the experience of the people who build these systems, not on a measured comparison.
Evals are also not a guarantee. In September 2025 Anthropic described three infrastructure bugs that degraded its models' answers over several weeks, and wrote that its own evaluations did not capture the degradation users were reporting. A suite only protects the behaviour it covers. That is a reason to build it from real usage and to keep watching production, which the rest of this post is about.
Start from real failures
The first cases should come from your own tasks and the failures you have already seen, not from a public benchmark. Anthropic's guidance is that 20 to 50 simple tasks drawn from real failures is a good start, and that a good task is one where two domain experts would independently reach the same pass or fail verdict.
There is a research reason for starting there. Shankar et al. observed that people need criteria to grade outputs, but grading outputs is what helps them define the criteria. Nobody can fully specify what "good" means before seeing what the system produces. So a person who knows the domain has to read real outputs, write down what is wrong with them, and group the notes into types of failure. Those types become the eval cases.
Grade with code where a rule can decide
There are three kinds of grader, and the same guidance puts them in an order: deterministic graders where possible, a language model where necessary, and people to validate both.
- Code decides whatever a rule can decide: a calculated amount, a required field, a cited source that has to exist, a tool that must not be called. It is fast, cheap and repeatable.
- A language model as judge handles what needs judgment: whether a summary is faithful, whether an answer addresses the question.
- A person who knows the domain sets the standard the other two are checked against.
A model used as a judge is a classifier, and it has to be validated like one. Compare its verdicts with the expert's on cases it was not tuned on, and look at how many real failures it catches and how many good outputs it passes, not at overall agreement. The arithmetic shows why: if 1 output in 20 is wrong, a judge that passes everything agrees with the expert on the other 19 and catches nothing. A judge is also easier to validate when it answers one narrow question than when it scores several qualities at once.
Keep two suites, and read the numbers honestly
Anthropic's guidance separates regression evals, which should pass almost every time and act as the gate, from capability evals, which start at a low pass rate and show what to improve next. Mixed together, each hides the other's signal.
Small suites cannot see small differences. As a measure of how the agent does in general, a suite is a sample of the tasks it will meet, and a sample of 50 leaves a wide margin of error: at an 80% pass rate, about 11 percentage points either way. A suite of that size tells you that something broke. It cannot tell 80% from 85%. The practical remedy, from Anthropic's statistical guidance, is the paired comparison: run the old and the new version on the same cases and look at which cases changed.
Then read the failures. A score that nobody has opened is not a finding.
For agents: grade the outcome, repeat the run, report the cost
Grade the outcome. Anthropic's advice is to grade what the agent produced, not the path it took: the invoice lines, the record in the database, the file. Several paths can be valid, and a check on the exact sequence of tool calls fails runs that were correct. The trace is for what the outcome cannot show: where a run went wrong, and rules that must hold along the way, such as a tool that must never be called.
Repeat the run. Because runs differ, each task is run several times. For an agent that works unattended, the number that matters is how often all of the runs succeed, not whether one of them does.
Report cost with success. Tokens, tool calls and duration per task belong next to the pass rate, measured on the same cases. That is what lets you accept or reject a cheaper or faster model on evidence.
Public leaderboards do not replace this. An audit of ten widely used agent benchmarks found validity problems in most of them; on one, an agent that returned empty responses was scored as succeeding on 38% of tasks. Leaderboards help to shortlist models. Whether an agent can do your work is something only your cases can show.
Run them on every change
A suite earns its keep when it runs without anyone deciding to run it.
- Before a release. Every change to a prompt, a model version, a tool or the retrieval setup runs the regression suite, and a result below the threshold blocks the release. GitHub describes running more than 4,000 offline tests for Copilot, most of them in its CI pipeline.
- After a release. Sample production runs, grade them, and review what users flag.
- Back into the suite. Every failure found in production becomes a new case, so the same failure cannot ship twice.
This is also what turns a model upgrade into routine work. When a provider retires a model, the question "is the new one at least as good at our tasks?" is answered by running the suite, not by spot checks.
Evals are also your evidence
There is a second reason to keep eval results: they are documented evidence that you can show to others. The employees who are asked to hand work to an agent, and the customers it serves, can fairly ask how you know that it is reliable. A demo does not answer that, and neither does "we tested it". A suite's results do: they say which tasks the agent was tested on, how each result was judged and how often it passed, with a date on every run.
The same record counts when something goes wrong. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline liable for wrong information its chatbot gave a customer, finding that it "did not take reasonable care to ensure its chatbot was accurate". Eval results are the record of the care you took.
For some systems, testing this way is also a legal duty. Under the EU AI Act, high-risk systems are to be tested "against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose" (Article 9(8)), which is a description of an eval suite. That duty binds providers of high-risk systems, and what makes a system high-risk is what it is used for, not whose model it is built on. This is general information, not legal advice.
Questions to ask
Whether you build an agent yourselves or buy one, these questions separate a suite that protects you from one that only looks good in a presentation.
- Which real failures were the cases built from?
- Who decided what counts as correct, and do they know the domain?
- How was each automated judge checked against that person's verdicts?
- How many times is each task run, and how often do all of the runs pass?
- What happens to a release that scores below the threshold?
- When did someone last read the failures?
Without evals, an agent is still an experiment. Every agent we build ships with its own: your real tasks, turned into eval cases that measure accuracy, robustness and cost before launch, and check every change after it.