Evaluating Agents: From Offline Testing to Production Canaries

Six months ago, we took over an agent project. The previous version had been validated by something like "the engineers chatted with it a bit, it seemed fine, so we shipped it." By week three in production, the finance team reported back: on the same report query, the agent sometimes gave the right numbers, sometimes confused last quarter with this quarter, and once it called tools eleven times in a row, burned a minute and a half, then said "I'm not sure." Nobody could tell you what was actually broken. Because there was no regression test set.

Agent eval isn't mysterious. It just turns "does it work?" into measurable numbers you can reproduce and rerun each time you adjust a prompt or change models.

What agent eval is

Agent eval takes a fixed set of test cases and measures an AI agent (the kind that plans for itself, calls tools, and executes across multiple steps) on dimensions like task success rate, tool accuracy, cost, and latency. The crucial part is reproducibility. A score you measure today should come out exactly the same next week, even after you've shipped code changes. Numbers either go up or they go down. No ambiguity.

Regular chat model evals are simpler. You check if the single-turn response is correct. Agents work differently. An agent follows a trajectory: read the question, decide which tool to call, see what it returns, decide what to do next. One wrong step in the middle and the final answer is still wrong, often in ways that are difficult to trace. Agent eval has to examine both the result and what actually happened along the way.

Four metrics to measure

The first step is breaking down vague "accuracy" into separate, independent metrics. Mix them together and you'll never know what to fix.

The reason is practical. One project hit 88% task success rate, which seemed reasonable, but tool accuracy was only 62%. The model was essentially getting lucky. New questions exposed the weakness immediately. Another agent achieved high success rates but P95 latency ran to 40 seconds. Users had already closed the window. A single metric will mislead you.

Include latency and cost in your eval from the start. Don't wait for production. An agent that independently chooses whether to make extra tool calls carries variable costs. Record token consumption and tool call count for every test case. Pull out the worst 5% and examine it. That's usually where issues surface.

Offline evaluation: build a test set you can regress

Offline eval means preparing fixed inputs with correct answers or trajectories, then running them through to score. It's inexpensive, fast, repeatable, and serves as the first filter for any change.

Building test sets, three mistakes stand out. First, pull cases from real logs, not invented hypotheticals. Real users misspell things, ask ambiguous questions, request multiple things at once. Those are exactly where agents break. Second, include trap cases: queries the database can't answer, questions outside scope, empty tool responses. A good agent says "I can't do that," not guesses or fakes. Third, every time you fix a production bug, add that case to your test set. Keep it there. This is how the test suite gains value.

For scoring: anything computable programmatically should be computed. Numbers, SQL results, correct API calls. Write assertions for these. Turn to LLM-as-judge only for subjective dimensions like tone or completeness. But recognize that the LLM makes mistakes too. Have humans validate the first round to verify the judge itself is trustworthy.

Live canary: an offline pass doesn't mean production is safe

Offline scores are just lab data. Real traffic always finds inputs you didn't expect. Never jump from "offline pass" to 100% rollout. Instead, use canary deployment. Route 5% to the new version. Watch three signal groups: task success proxy metrics (whether users ask follow-ups or escalate to a human), latency and cost, error rates. Compare against the old version. Move to 20%, then 50%, only after performance holds steady.

In production, log the complete trajectory. Every step of every real conversation stays recorded. This serves two purposes: you can replay and debug when something breaks, and the live traffic becomes source material for the next test set. Offline and online form one loop: pull cases from live traffic, add them to your offline test suite, fix and regress, canary more traffic, pull more cases.

Take the agent that called tools eleven times. We didn't rewrite it. We spent two days organizing its past conversation logs into 120 regression cases. Initial measurement: 71% tool accuracy, P95 latency at 38 seconds. After fixes, success rate moved from 74% to 93%, latency dropped to 9 seconds. Nobody in that process said "I think this is better." Every decision came from a number.

When agentic workflows move into a client's operations, evaluation doesn't come last before shipping. It's a first-day asset built together. Demos get approval. What actually survives production is the test set you trust enough to run every single day.