Tools · Agents · 11

Evaluate whether an agent solves the task and whether its trajectory is reliable.

A correct final result can hide a wrong tool call, excessive retries or a fragile path. The playground separates final outcome, observable trace decisions and operational failures, then compares them with release gates you define.

Outcomefinal success, first pass and recovery

Tracecorrect tools, omissions, steps and retries

Operationstimeouts, policy and projection

Task outcome

Evaluated tasks

First-pass successes

Tasks that entered retry

Tasks recovered after retry

Total additional retry attempts

Tools and trajectory

Expected tool decisions

Correct decisions

Wrong tool / invalid argumentsThe remainder is interpreted as a missed required decision.

Total observable steps

Unnecessary steps by rubric

Operations and traffic

Tasks with timeoutMay overlap other failures; do not add it to the failure rate.

Tasks with policy-adherence failureDo not count guardrails that correctly blocked an action.

Projected monthly traffic

Release gates

Minimum final success (%)

Minimum first-pass success (%)

Minimum tool-decision accuracy (%)

Maximum timeout rate (%)

Maximum policy-adherence failure (%)

Maximum unnecessary steps (%)

The gates compare point estimates. The 95% intervals are shown separately and are not used to auto-approve a release.

Copy linkExport JSONReset

Final success——

First pass—without retry

Correct tool decisions——

Retry recovery—among retried tasks

Timeouts—

Policy—

Missed tool—

Unnecessary steps—

Steps per task—

Attempt multiplier—

Not aggregatedfailures ≠ sum of incidences

Unittask + observable trace

Release gates

—

Final success——

First pass——

Tool decision——

Timeouts——

Policy adherence——

Unnecessary steps——

——

—

Expected timeouts—

Expected policy-adherence failures—

Limit: the projection assumes future traffic resembles the evaluated sample. The failure range comes from the Wilson interval for final success; it does not model dataset shift or dependence between tasks.

Outcome and trajectory answer different questions

End state

τ-bench evaluates the final environment state against an annotated goal state.

Observable trace

Tool choice, arguments, retries, handoffs and steps help locate the cause of a result. OpenAI calls this structured path evaluation trace grading.

Consistency

Measuring consistency requires repeated trials on the same task. We do not reconstruct pass^k from a single success rate.

Connect the trace to the evaluation system

These reliability metrics are one layer of a broader evaluation system. The AI model and system evaluation framework explains how to combine deterministic checks, reference sets, human evaluation and production signals without collapsing quality into a single score.

Sources

No hidden composite score.

The gates are configurable and are not presented as universal thresholds.

OpenAI · Trace grading
τ-bench
AgentBench
Reviewed: 2026-08-21