Chapter 5 — Tests, verifiers, diff review, stop conditions, and task evaluation¶
A coding agent can run pytest, get a green result, write “done,” and still not be finished.
It may have changed a test so that the wrong behavior now passes. It may have broken an uncovered API. The review may belong to an earlier commit. The evaluation environment may contain state left by another run. Or the task may require two conditions while the tests check only one.
The harness therefore needs to separate two questions:
what evidence did this candidate produce?
is that evidence sufficient and still valid for closing the task?
The second question is a system decision, not a sentence generated by the model.
“Done” is a decision derived from fresh evidence for the exact candidate
Tests, review, and additional judgment produce independent signals. The harness binds them to the same state and only then decides whether to accept, rework, or hand back the task.
Tests, verifiers, reviewers, and stop conditions are different objects¶
It helps to keep four concepts separate:
| Object | What it does | What it cannot establish by itself |
|---|---|---|
| Test | Runs assertions over behavior or state | That the assertions cover the whole contract |
| Verifier | Produces evidence about a candidate: tests, build, types, lint, external state, invariants | That one signal is sufficient for acceptance |
| Reviewer | Inspects semantics, risk, scope, and decisions that are hard to encode | That its judgment is deterministic or still current after another commit |
| Stop condition | Decides whether to continue, replan, accept, or hand back | That stop = success |
Anthropic uses a related separation in its agent-evaluation terminology: a task defines inputs and success criteria, a grader scores aspects of the trajectory or outcome, and the outcome is the real final state of the environment. It also distinguishes the evaluation harness that runs and aggregates an eval from the agent harness that enables the model to act.1
For a development harness, verifier is useful as a broader term than one individual test:
That evidence can be binary, numeric, or structured. The closure decision consumes multiple pieces of evidence.
“The tests pass” only means something relative to a contract¶
Consider this task:
Add --json to `acme users list`.
Preserve the existing text mode exactly.
Return exit code 2 when the filter is invalid.
Do not change the public Python API.
If the agent adds two tests for --json and both pass, three parts of the contract are still unaccounted for.
The harness should map acceptance criteria to evidence:
| Criterion | Minimum evidence |
|---|---|
--json emits valid JSON |
functional test / CLI invocation |
| text mode remains unchanged | regression tests / stable snapshot |
| invalid filter → exit 2 | negative test |
| public Python API remains stable | contract test / type surface / review |
| scope stays bounded | diff/path review |
Different criteria can require different mechanisms. What matters is being able to answer which evidence supports each criterion.
A green suite does not automatically fill a cell that was never evaluated.
The outcome matters more than a ritualized trajectory¶
An agent can reach a valid solution through different paths. Requiring exactly this sequence:
can turn the grader into a test of working style rather than a test of the result.
Anthropic recommends deterministic graders where possible, but also warns against rigid tool-call sequences when several valid trajectories can solve the task. In those cases, grading the outcome is often more robust.1
That does not make trajectories irrelevant. Some properties exist in the trajectory itself:
did the agent read or exfiltrate a secret?
did it use a forbidden tool?
did it bypass a required approval?
did it repeat a non-idempotent mutation?
The useful boundary is trajectory required for policy versus incidental implementation trajectory.
Verifiers should observe the real effect¶
A process that exits with code 0 proves only that the process reported success.
For a migration:
does not imply:
expected schema exists
expected constraints exist
data is preserved
rollback or forward recovery is defined
For publication:
does not imply:
And for a coding task:
does not prove that those tests ran against the commit we intend to accept.
When an external system exposes observable state, a strong verifier checks the postcondition, not just the agent’s intent or stdout.
Evidence must be bound to the exact candidate¶
A common failure looks like this:
The tests and review belonged to A, not B.
GitHub documents this concretely for Copilot code review: new pushes are not automatically re-reviewed unless Review new pushes is enabled or another review is requested. A Copilot approval can also be dismissed after later commits.2
The broader lesson does not depend on Copilot:
A useful record should contain at least:
candidate_sha: 73ab91f...
contract_version: 3
environment_fingerprint: py311-linux-lock-8f42
verifier_version: cli-contract-v5
command: pytest tests/cli tests/regression
result: pass
started_at: 2026-09-10T17:21:04Z
finished_at: 2026-09-10T17:22:11Z
This is an illustrative harness contract, not a standard.
If the candidate SHA changes, the harness must invalidate every piece of evidence whose result can depend on the change.
Not every change requires rerunning absolutely everything¶
Invalidating evidence does not mean always running the most expensive suite from scratch.
A harness can represent dependencies:
src/cli/output.py
├─→ cli_contract_tests
├─→ text_mode_regressions
└─→ type_check
docs/cli.md
└─→ docs_build
If only docs/cli.md changes, a database test does not become more informative by being rerun.
But this optimization needs an explicit, conservative dependency model. If the system cannot establish whether a change can affect a piece of evidence, the fail-closed choice is to rerun it.
The goal is fresh evidence, not the largest possible command count.
Tests are also part of the diff, and they can be wrong¶
A coding agent can often modify all of these at once:
That creates an important risk: the oracle can move to match the bug.
For example:
The suite stays green while the contract is violated.
Diff review should therefore treat changes to tests and graders as changes to the acceptance criterion, not as incidental support files.
Depending on the task, useful defenses include:
hidden or out-of-band tests
contract tests outside the editable workspace
base → head comparison of tests and snapshots
protected CI or graders
reference solution used to validate the evaluator
Anthropic recommends a reference solution that establishes task solvability and validates grader configuration, and it recommends designing graders to resist bypasses.1
Diff review covers questions a suite may never encode¶
A useful diff review is not a second full read of the repository.
It targets risk boundaries:
scope: did the change touch paths outside the task?
public API: did signatures, formats, or contracts change?
config/CI: was a protection or verifier weakened?
dependencies: were unnecessary packages or code added?
data: are there migrations or destructive changes?
tests: was coverage added, or was the target moved?
generated: were accidental artifacts committed?
security: did permissions, network, or secret access expand?
Some of this can be automated with path rules, diff-size limits, schema checks, or scanners. Hard semantic questions may need a human or model-based review.
GitHub, for example, explicitly says that pull requests created by Copilot cloud agent deserve the same thorough review as any other contribution before merge.3
A model reviewer is probabilistic evidence, not an oracle¶
A second model can notice inconsistencies that tests do not express:
does the change actually solve the issue?
does the new abstraction violate an implicit invariant?
is there an uncovered edge case?
does the diff introduce unnecessary complexity?
But an LLM grader or reviewer is still non-deterministic. Anthropic recommends calibrating LLM-as-judge graders against human experts and giving the grader an explicit option such as Unknown when it lacks enough information.1
Using “another agent” does not guarantee independence either. It may share:
the same model
the same incomplete context
the same repository instructions
the same conceptual mistake
A model-based reviewer therefore complements tests and human review. It does not turn an inference into ground truth.
Reviews become stale too¶
Even a strong review can become obsolete when any of these change:
candidate SHA
relevant base branch
contract version
verifier configuration
instructions used by the reviewer
GitHub documents another concrete boundary: Copilot code review reads custom instructions and skills from the PR’s head branch. Those instructions are therefore part of the review context and can change alongside the code being reviewed.2
A serious harness stores enough provenance to know what was reviewed and under which configuration.
A stop condition is not the same as a success condition¶
An agent loop may stop for several reasons:
ACCEPTED
REWORK_REQUIRED
HAND_BACK_TO_HUMAN
BLOCKED_BY_AUTHORITY
ENVIRONMENT_FAILURE
BUDGET_EXHAUSTED
NO_PROGRESS
Every one of these is a stop condition. Only one means acceptance.
This avoids a dangerous shortcut:
The model may stop because it incorrectly believes it is finished, because context was lost, or because it does not know how to continue.
The harness should derive closure from the contract and evidence.
A success gate can be made explicit¶
As a conceptual model, not a product standard:
For candidate \(h\):
- \(C\): every required acceptance criterion has evidence
- \(V\): required verifiers pass
- \(D\): diff/review leaves no known blocker
- \(P\): required policy and approvals remain valid
- \(F\): all required evidence is fresh for the current candidate and environment
If any condition fails, the state is not ACCEPTED.
That is intentionally different from:
Failure stop conditions protect quality and cost as well¶
A loop without an exit policy can repeat the same action forever.
A reasonable handback can trigger on an observable condition such as:
a required credential or approval cannot be obtained by the agent
the spec has two materially different interpretations
the evaluation environment is unstable and cannot produce valid evidence
the same failure repeats without new information
a required action exceeds policy
an explicit attempt/time/cost budget is exhausted
“Same failure without new information” is more useful than pretending there is one universal retry count. The concrete budget depends on the task, cost, and criticality.
HAND_BACK_TO_HUMAN is not necessarily a model failure. It can be the correct outcome when the next decision requires authority the agent does not possess.
The evaluation environment is part of the result¶
The same patch can produce different outcomes when any of these differ:
operating system
runtime version
lockfile/dependencies
external services
fixtures
clock/timezone
prior state
available CPU/memory
SWE-bench uses Docker containers to create reproducible evaluation environments. Its harness prepares environments, applies patches, runs tests, and determines task outcomes inside them.4
Anthropic recommends starting each trial from a clean, isolated environment. Shared state, caches, and resource exhaustion can introduce correlated failures or artificial advantages.1
A result without environment identity is therefore weaker evidence than it first appears.
Evaluating a coding agent means evaluating model + harness + task + grader¶
A coding-agent benchmark does not measure a model in isolation.
The observed result depends on:
model
agent harness
prompt/instructions
tools
context/repository snapshot
sandbox/environment
budget
task specification
graders/tests
scoring rule
Anthropic states the boundary directly: when evaluating “an agent,” it is evaluating the harness and model together.1
Two percentages should not be treated as the same measurement if the harness, budget, environment, or suite version differs.
A green benchmark can still measure the wrong thing¶
In February 2026, OpenAI stopped reporting SWE-bench Verified for frontier launches after finding contamination plus specification and test problems. Among other issues, its audit found tests that rejected functionally correct solutions and evidence of exposure to benchmark problems or solutions during training.5
The lesson is not “SWE-bench is bad.” The lesson is that the benchmark also needs evaluation.
That point became more important in July 2026. An OpenAI audit of SWE-Bench Pro estimated that approximately 30% of tasks were problematic under its audit methodology, including task-specification, test, and grading issues.6
That ~30% is the result of that specific audit. It is not a universal benchmark error rate.
Before interpreting a score, inspect:
which task ran
which snapshot and environment were used
what the agent could see
what budget it received
which grader decided pass/fail
whether the grader accepts alternative valid solutions
whether contamination or leakage is plausible
One run does not estimate reliability¶
Agents are stochastic. Passing a task once demonstrates possibility, not consistency.
Anthropic distinguishes two questions:
- pass@k: probability of obtaining at least one successful result in
kattempts - pass^k: probability that all
kattempts succeed1
If a product can try several candidate solutions and select a valid one, pass@k may be relevant. If every user expects repeatable success, the consistency captured by pass^k answers a different question.
They are not interchangeable.
For a task with per-trial success probability \(p\), assuming independent and identically distributed trials:
The independence assumption matters. If every trial shares a broken service or contaminated state, those formulas do not describe the experiment correctly.
Capability evals and regression gates answer different questions¶
A capability eval asks:
A regression suite asks:
Anthropic recommends regression evals that protect established behavior at pass rates close to 100%, while capability evals should retain enough difficulty to distinguish progress.1
For CI on one concrete task, we usually want a deterministic regression gate. When evaluating an agent harness, we also want a capability and reliability distribution across many tasks and trials.
Conflating the two creates opposite mistakes:
using a probabilistic benchmark as the merge gate for one PR
using a trivial 100% suite to claim capability progress
Worked example: closing the --json task correctly¶
Return to the example task.
The agent ends with this candidate SHA:
The harness runs:
1. contract verifier
- `acme users list --json` → valid JSON
- invalid filter → exit 2
2. regression verifier
- existing text-mode tests → PASS
- Python API contract tests → PASS
3. static verifier
- type check → PASS
- lint → PASS
4. diff verifier/review
- paths outside scope → none
- existing tests weakened → no
- CI / permissions → unchanged
5. freshness gate
- every result belongs to 73ab91f
- working tree clean
- contract_version = 3
The evidence bundle might look like this:
candidate_sha: 73ab91f
contract_version: 3
environment_fingerprint: py311-linux-lock-8f42
required_checks:
cli_contract: pass
text_regression: pass
python_api_contract: pass
typecheck: pass
lint: pass
diff_review:
blocker_count: 0
test_oracle_changed: false
freshness:
all_evidence_matches_candidate: true
stop_reason: ACCEPTED
Again, this is an illustrative structure.
If the agent then changes one line and creates 84cd120, the correct state returns to:
A recent PASS is not inherited by proximity in time.
What the harness should store to explain closure¶
An auditable closure needs less narrative and more provenance:
task_id + contract_version
base_sha + candidate_sha
workspace/environment identity
changed paths + diff digest
verifier names + versions + commands
start/end time + exit/result
relevant artifacts
reviewer identity/config + reviewed SHA
approval/policy evidence when applicable
known blockers
stop_reason
With that information, the system can reconstruct why a task was accepted and what changed if a regression appears later.
Without it, “it passed yesterday” is difficult to distinguish from “we think something similar passed yesterday.”
Trade-off: verification coverage, latency, and cost¶
More checks do not automatically produce a better system.
A suite can take hours, depend on expensive services, or be flaky. A model-based reviewer adds cost and uncertainty. Human review on every iteration can destroy throughput.
The useful pattern is to layer evidence:
fast + deterministic + local → on each relevant iteration
dependency-selected → after a bounded scope change
expensive / integration / human → at milestones or risk boundaries
full regression → before release when the contract requires it
But the harness must not silently downgrade a required check because it is expensive. If a mandatory verifier cannot run, the state is BLOCKED/UNVERIFIED, not PASS.
Production implication: “done” should be a derived state¶
The model proposes code and may suggest that it has finished. The harness computes final state from verifiable evidence.
The operational rule is:
That supports autonomy without turning model confidence into merge authority.
A strong harness can close quickly when evidence is sufficient, reopen verification when the candidate SHA changes, and hand back when authority or verifiability is missing.
The next chapter extends this logic to work that lasts far longer than one session: memory, subagents, recovery, integration, merge, and observability must preserve not only the work but also the provenance of which evidence remains valid.
Primary references¶
-
Anthropic, Demystifying evals for AI agents, January 9, 2026. Used for task/trial/grader/outcome/evaluation-harness definitions, grader types, stable isolated environments, outcome-based grading, reference solutions, bypass resistance, human calibration, and pass@k/pass^k. ↩↩↩↩↩↩↩↩
-
GitHub Docs, Using GitHub Copilot code review on GitHub. Used only for current re-review/new-push semantics, approvals, and custom-instruction context. Other Copilot capabilities are not generalized to the harness. ↩↩
-
GitHub Docs, Review output from Copilot. Used for GitHub's first-party guidance to review cloud-agent changes before merge and its approval/workflow boundaries as one external-control example. ↩
-
SWE-bench, Evaluation Harness Reference. Used for the Docker evaluation harness that prepares environments, applies patches, runs tests, and determines reproducible outcomes. ↩
-
OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities, February 23, 2026. Used for findings on contamination, task/test mismatch, and limits of treating that benchmark as frontier-capability signal. ↩
-
OpenAI, Separating signal from noise in coding evaluations, July 8, 2026. Used for its SWE-Bench Pro audit and contextualized estimate of problematic tasks. The number is not extrapolated beyond the reported methodology. ↩