What are we evaluating?
A broader test checks the task, policies and real effects. Preserve versions, cases and protocol to make comparisons reproducible.
Links containing ?t= open the video at a specific second.
The ideas to retain
Start by defining the object, not the metric
A useful abstraction for an AI system is:
Level 1 — Evaluate the model
A model eval tries to measure model behavior under a controlled harness.
What a model eval can support
It can support a statement such as:
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — The answer is not the outcome.
An agent can say “done” without completing the task.
Inspect the actions it took and the resulting state as well.
Define that boundary before choosing a metric.
Illustrative example, not a benchmark result.
00:22 — Isolate the change. Evaluate the model.
Give both models the same cases and the same protocol.
Hold context, tools and the scoring rubric constant.
The difference describes this test; it does not certify the whole product.
00:44 — Missing evidence. Do not change everything.
When the right policy never reaches the context, inspect retrieval.
Change only that component and rerun the same cases.
Then confirm the whole task improves, not just the ranking.
Illustrative example, not a benchmark result.
01:06 — The workflow defines what can happen.
Workflow policy allows, blocks or redirects actions.
Authorization must be checked before a write is executed.
Test rejection and recovery branches too.
Illustrative example, not a benchmark result.
01:28 — The trajectory shows what happened.
The workflow defines possible routes; a trajectory records one execution.
Follow arguments, responses, errors and state changes.
A correct final message does not erase a duplicated side effect.
Illustrative example, not a benchmark result.
01:50 — Diagnose narrowly. Confirm in the system.
A narrow boundary helps locate the failure.
A broader test checks the task, policies and real effects.
Preserve versions, cases and protocol to make comparisons reproducible.
