Evaluating AI Agents: What to Measure Before Production

Most AI agent evaluations stop at the demo: the agent completed the task once, in front of an audience, on a good day. Production is not a demo. It is hundreds of runs against messy inputs, changing tools, and edge cases nobody scripted. Teams that skip serious evaluation discover their agent's failure modes from users, which is the most expensive possible way to learn them.

Task Success Is Necessary but Not Sufficient

The first metric is obvious: did the agent complete the task correctly? But 'correct' needs a definition per task — a written rubric, not a vibe. For document tasks, define what counts as a correct extraction; for workflow tasks, define the acceptable end states. Without rubrics, evaluation becomes opinion, and opinions do not catch regressions. Build a golden set of representative tasks with known-good answers, and run it on every change.

Task success should be measured across input variety, not just the happy path. Agents that score perfectly on clean inputs often collapse on ambiguous, incomplete, or adversarial ones. Your evaluation set should include the messy reality: typos, missing fields, contradictory instructions, and requests slightly outside the agent's scope. The interesting failures live there.

Measure Tool Discipline, Not Just Answers

Agents act through tools — API calls, database queries, document searches. A correct final answer reached through reckless tool use is a liability: excessive calls cost money and time, wrong tools produce side effects, and redundant calls signal the agent does not understand its environment. Track which tools the agent calls, in what order, and how often. Penalize tool-call bloat in your scoring.

Tool discipline also covers error recovery. When a tool call fails, does the agent retry sensibly, try an alternative, or spiral into repeated failures? Production agents will hit failing tools constantly — rate limits, timeouts, changed schemas. An agent that cannot recover gracefully from a failed tool call is not production-ready, regardless of its demo performance.

Cost and Latency Are Evaluation Metrics

Every agent run has a price: model tokens, tool calls, compute time. An agent that solves a task for twelve dollars when a deterministic script solves it for cents is an engineering failure wearing an AI costume. Measure cost per successful task, not just cost per run — failed runs that get retried are part of the true cost. Set budgets per task type and fail the evaluation when the agent exceeds them.

Latency matters for the same reason. An agent that takes nine minutes to do what a human does in five has not automated anything. Measure end-to-end time including tool calls, and compare against the manual baseline. Some tasks justify slower, more careful agents; most do not. Know which kind you are building.

Evaluate Failure Behavior Deliberately

The most important evaluation question is not 'does it succeed?' but 'what does it do when it cannot succeed?' Good agents recognize impossible or ambiguous tasks and escalate to a human with context. Bad agents hallucinate confidence, take irreversible actions, or loop forever. Design test cases where the correct behavior is to stop and ask — missing information, contradictory instructions, actions outside the agent's authority — and score refusal-to-proceed as a success.

Also test the agent's behavior under tool degradation: slow tools, failing tools, tools returning garbage. The agent's job in production includes surviving its own infrastructure. If it has never been tested against failure, its first failure will be in front of a user.

Human Review Calibration

Before full deployment, run the agent in shadow mode: it processes real tasks, but a human reviews every output before anything ships. Measure agreement rates between the agent and reviewers, and — more importantly — categorize disagreements. Are they judgment calls, data gaps, or genuine errors? Shadow mode is where you learn whether your rubrics match reality, and it is the cheapest production-like test available.

Use shadow-mode data to set the agent's autonomy boundaries: which task types it handles alone, which need spot checks, which stay human-led. These boundaries should come from measured performance, not optimism.

Regression Testing for Agents

Agents change constantly — new model versions, updated prompts, new tools. Every change needs the golden set re-run automatically. Agent behavior is non-deterministic, so run each test multiple times and track pass-rate distributions rather than single outcomes. A prompt change that drops success from 94% to 88% is a regression worth investigating, even if the demo still looks fine.

Version everything: prompts, tool definitions, model versions, evaluation sets. When production behavior changes, you need to know exactly what changed. 'We updated the prompt' is not a version.

What 'Ready for Production' Actually Means

An agent is production-ready when: task success is measured against written rubrics across varied inputs; tool use is disciplined and budgeted; cost and latency beat the manual baseline; failure behavior has been tested and escalates correctly; shadow mode shows acceptable agreement with human reviewers; and every change re-runs the evaluation suite automatically. Anything less is a pilot with production access — which is how incidents happen.