Cases that test the system
Changing a label or grader can change the score without improving the model. Create a new version; do not silently rewrite the earlier result.
Links containing ?t= open the video at a specific second.
The ideas to retain
The object is an eval release, not "the dataset"
Represent one evaluation release as:
The smallest unit must be auditable
A case should let us resolve at least:
Start with a coverage hypothesis
"Representative" does not mean "random" by default.
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — An eval is more than its questions.
A case needs an input, context and a verifiable expected result.
The rubric and harness determine what is observed and scored.
Version the complete evaluation, not just a list of prompts.
00:22 — Seek coverage. Not just volume.
Start with the tasks and failures that matter.
Cover relevant conditions: permissions, limits and partial failures.
Many easy cases cannot compensate for an untested boundary.
Illustrative example, not a benchmark result.
00:44 — Change one condition. The decision must change.
Suppose returns are allowed through day 30.
Two otherwise identical orders at 29 and 31 days require different decisions.
The pair exposes the shortcut “asks for a return, so approve”.
Illustrative example, not a benchmark result.
01:08 — Separate families. Not just rows.
A paraphrase can leak a development case into the test set.
Group variants by document, entity or seed before splitting.
A duplicate scanner cannot prove the absence of contamination.
Illustrative example, not a benchmark result.
01:30 — Development is not release measurement.
Use dev for tuning and a regression bank for known failures.
Reserve a controlled-access holdout for the release decision.
Repeatedly tuning against that holdout destroys its independence.
01:52 — The criterion changed. The evaluation changed.
Freeze cases, references, rubric, harness and protocol.
Changing a label or grader can change the score without improving the model.
Create a new version; do not silently rewrite the earlier result.
Illustrative example, not a benchmark result.
