The result and the path
A hard violation blocks release even when other metrics improve. Save the trajectory and the verdict reason so the result is reproducible.
Links containing ?t= open the video at a specific second.
The ideas to retain
A trajectory is a sequence of transitions, not a list of tool names
For reasoning about one execution, represent each step as:
Start with the outcome: was the task actually solved?
When the result can be checked directly, the first grader should inspect the real state.
Then evaluate the decisions that matter along the way
Not every trajectory difference is important.
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — Do not count calls. Follow the state.
An action can change the environment before the agent observes it.
Connect the precondition, call, observation and resulting state.
The task succeeds when its success condition is actually true.
Illustrative example, not a benchmark result.
00:22 — Success cannot cancel a permission violation.
A refund can complete while violating a mandatory policy.
Check authorization before the write, not after the result.
A hard constraint cannot be averaged away by quality or speed.
Illustrative example, not a benchmark result.
00:44 — Timeout does not mean “nothing happened”.
The backend can commit an operation before its response is lost.
The agent sees an unknown result, not necessarily a failed operation.
A blind retry may execute the same side effect twice.
Illustrative example, not a benchmark result.
01:08 — Reconcile first. Then decide.
Query the operation state using a stable identifier.
If it is already committed, do not repeat the write.
Safe retries depend on the tool’s idempotency contract.
Illustrative example, not a benchmark result.
01:32 — There can be several correct paths.
Do not require the agent to copy one preferred call sequence.
Evaluate the constraints and final state the task requires.
Compare cost among valid trajectories; fewer steps are not always better.
Illustrative example, not a benchmark result.
01:54 — The gate must explain why it stops.
Separate task success, policy, state integrity and reliability.
A hard violation blocks release even when other metrics improve.
Save the trajectory and the verdict reason so the result is reproducible.
Illustrative example, not a benchmark result.
