Tests, verifiers and stop conditions
Diff review asks another question: what actually changed, and was it permitted?
Links containing ?t= open the video at a specific second.
The ideas to retain
Tests, verifiers, reviewers, and stop conditions are different objects
It helps to keep four concepts separate:
“The tests pass” only means something relative to a contract
Consider this task:
The outcome matters more than a ritualized trajectory
An agent can reach a valid solution through different paths. Requiring exactly this sequence:
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its visible explanatory text, not an invented voice.
Each check answers one question
The linter checks style, not whether the output is valid JSON.
A contract test checks the format. A regression test checks that the old mode still works.
Diff review asks another question: what actually changed, and was it permitted?
This green result belongs to A
Tests pass on revision A and produce a receipt.
A later edit creates B. The receipt remains valid history, but it does not prove B.
Delivery needs evidence bound to the current candidate and its environment.
Review the test change too
The test fails because it detects that the previous output disappeared.
Deleting that assertion produces green without fixing the behaviour.
Changes to the verifier also need review. Passing a weakened test does not satisfy the contract.
Two modes, two contracts
With no option, report keeps its text output.
With --json, it returns an object that can be parsed and validated.
Test both paths and invalid input. Checking only the new mode leaves a regression invisible.
A time limit is not a PASS
The run has a budget to prevent unbounded work.
The counter reaches zero while a mandatory check is still pending.
The correct status is incomplete, with the available evidence. A timeout does not become acceptance.
One run does not measure reliability
In this illustrative example, 16 of 20 runs satisfy the contract.
The other four fail for different reasons that must remain in the record.
That 80% describes this sample and environment. It is not a universal model guarantee.
