Skip to content
04 of 06AI Agents

Chapter 3 — How to evaluate an AI agent

Library

Series and technical notes.

You are in AI Agents · How to evaluate an agent.

Series

AI Agents

6 items

Watch video, summary and related content

Estimated reading3 min

A chatbot can often be evaluated by comparing an answer with a reference. An agent can produce the right final answer after using the wrong tool, spending ten times more steps than necessary, or leaving irreversible changes behind. Agent evaluation therefore has to inspect the whole task.

A demo measures an output. An agent needs a trace.
The final outcome matters, but so do the tools called, permissions used, unnecessary steps, and how the system recovers from failure.
Objectivereproducible task
→
Traceplan, tools, and state
→
Outcomecorrect or not
→
Costtime, tokens, risk
Trust gate

Reaching the answer is not enough. The agent should make its actions inspectable, show the evidence it used, and stop when authorization or confidence is insufficient.

The unit of evaluation is a task

A useful task should specify:

  • the initial state of the environment;
  • the user's objective;
  • allowed tools;
  • accessible data;
  • the success condition;
  • forbidden actions;
  • a budget for steps, time, or cost;
  • the expected result and abstention cases.

“Answer invoice questions well” is too vague. “Given a customer and a date range, locate overdue invoices, calculate the total, and prepare a draft without sending it” is concrete enough to build reproducible cases and separate reading, calculation, and writing.

Four dimensions worth measuring

1. Outcome

Did the task finish correctly? This is the most visible dimension, but not the only one. The metric should represent partial results and distinguish “could not complete” from “completed despite insufficient data.”

2. Trajectory

Which steps did the agent take? Did it use the correct tool? Did it repeat calls or query irrelevant information? The trajectory can reveal that a high task score was achieved by accident or through a cost profile that will not scale.

3. Security and compliance

Did the agent access only authorized resources? Did it attempt a forbidden action? Did it request approval when required? A task that reaches the desired output by violating policy is not a production success.

4. Operational economics

How long did it take? How many tokens, calls, and retries did it consume? What happens at the 95th percentile? The mean can hide agents that perform well in simple cases and become expensive when a tool degrades.

Benchmarks, domain cases, and judges

Public benchmarks are useful for comparing capabilities, but they do not replace your own cases. Domain rules, permissions, data, and the consequences of failure change the definition of success.

An LLM-as-judge can help evaluate open-ended text, but introduces another source of variability. When an objective condition exists, combine it with deterministic checks: a numeric result, valid JSON, an executable test, a documentary reference, or a concrete database mutation.

NIST is investigating evaluation probes: checks embedded into the workflow to inspect results and traceability. The work remains ongoing rather than a finished industry standard. The direction matters because it moves evaluation away from a final snapshot and toward the behavior of the system while it works.

The agent can also “cheat”

Once an agent has tools, the evaluation environment is no longer passive. It may find hints it was not supposed to use, modify task state in a way that fools the verifier, or exploit affordances the benchmark designer did not anticipate. NIST has documented this class of problem and recommends making allowed capabilities explicit and inspecting traces.

At minimum, an evaluation should record:

Text Only
task_id → objective → tools → arguments → results → state → verdict

If you store only the final answer, you do not have a reproducible agent evaluation; you have a collection of demos.

Designing a trust gate

Before deployment, a task should pass several gates:

  1. Correctness: the result satisfies the criterion.
  2. Traceability: the evidence and tools used are visible.
  3. Permissions: no out-of-scope actions occurred.
  4. Cost: execution remains within budget.
  5. Recovery: on failure, the system retries safely or stops.
  6. Abstention: without sufficient data or authorization, it does not improvise.

This changes the question from “what percentage does it get right?” to “under which conditions can I trust it to complete this task?”

What to remember

  • Agent evaluation starts with a reproducible task, not a polished answer.
  • Measure outcome, trajectory, security, and operational economics.
  • Public benchmarks help, but your environment determines the real risk.
  • Automated judges need deterministic checks and trace inspection where possible.
  • An agent that reaches the target by violating permissions has not succeeded.

References

Keep learning
Next chapterAgent securityAI Agents
← Anatomy of an agentAll series →