--- title: "Tests and verifiers for coding agents: when the harness can say a task is done" description: "How to combine tests, verifiers, diff review, commit-bound evidence, stop conditions, and repeated evaluation to decide whether a coding agent may close a task." date: 2026-09-10 date_modified: 2026-09-10 keywords: "coding agents, agent harness, tests, verifiers, diff review, stop conditions, evals, software engineering agents" tags: - AI - Agents - Software - Coding agents - Evaluation --- # Chapter 5 — Tests, verifiers, diff review, stop conditions, and task evaluation A coding agent can run `pytest`, get a green result, write “done,” and still not be finished. It may have changed a test so that the wrong behavior now passes. It may have broken an uncovered API. The review may belong to an earlier commit. The evaluation environment may contain state left by another run. Or the task may require two conditions while the tests check only one. The harness therefore needs to separate two questions: ```text what evidence did this candidate produce? is that evidence sufficient and still valid for closing the task? ``` The second question is a system decision, not a sentence generated by the model. {{ include_html("snippets/articulos-tecnicos/coding-agent-verification-stack.html") }} ## Tests, verifiers, reviewers, and stop conditions are different objects It helps to keep four concepts separate: | Object | What it does | What it cannot establish by itself | |---|---|---| | Test | Runs assertions over behavior or state | That the assertions cover the whole contract | | Verifier | Produces evidence about a candidate: tests, build, types, lint, external state, invariants | That one signal is sufficient for acceptance | | Reviewer | Inspects semantics, risk, scope, and decisions that are hard to encode | That its judgment is deterministic or still current after another commit | | Stop condition | Decides whether to continue, replan, accept, or hand back | That `stop = success` | Anthropic uses a related separation in its agent-evaluation terminology: a **task** defines inputs and success criteria, a **grader** scores aspects of the trajectory or outcome, and the **outcome** is the real final state of the environment. It also distinguishes the evaluation harness that runs and aggregates an eval from the agent harness that enables the model to act.[^anthropic-evals] For a development harness, `verifier` is useful as a broader term than one individual test: ```text verifier(candidate, environment, contract) → evidence ``` That evidence can be binary, numeric, or structured. The closure decision consumes multiple pieces of evidence. ## “The tests pass” only means something relative to a contract Consider this task: ```text Add --json to `acme users list`. Preserve the existing text mode exactly. Return exit code 2 when the filter is invalid. Do not change the public Python API. ``` If the agent adds two tests for `--json` and both pass, three parts of the contract are still unaccounted for. The harness should map acceptance criteria to evidence: | Criterion | Minimum evidence | |---|---| | `--json` emits valid JSON | functional test / CLI invocation | | text mode remains unchanged | regression tests / stable snapshot | | invalid filter → exit 2 | negative test | | public Python API remains stable | contract test / type surface / review | | scope stays bounded | diff/path review | Different criteria can require different mechanisms. What matters is being able to answer **which evidence supports each criterion**. A green suite does not automatically fill a cell that was never evaluated. ## The outcome matters more than a ritualized trajectory An agent can reach a valid solution through different paths. Requiring exactly this sequence: ```text search → open → edit → test → edit → test ``` can turn the grader into a test of working style rather than a test of the result. Anthropic recommends deterministic graders where possible, but also warns against rigid tool-call sequences when several valid trajectories can solve the task. In those cases, grading the outcome is often more robust.[^anthropic-evals] That does not make trajectories irrelevant. Some properties exist in the trajectory itself: ```text did the agent read or exfiltrate a secret? did it use a forbidden tool? did it bypass a required approval? did it repeat a non-idempotent mutation? ``` The useful boundary is **trajectory required for policy** versus **incidental implementation trajectory**. ## Verifiers should observe the real effect A process that exits with code `0` proves only that the process reported success. For a migration: ```text command exit code = 0 ``` does not imply: ```text expected schema exists expected constraints exist data is preserved rollback or forward recovery is defined ``` For publication: ```text API returned 200 ``` does not imply: ```text the intended artifact is deployed ``` And for a coding task: ```text agent_message = "all tests pass" ``` does not prove that those tests ran against the commit we intend to accept. When an external system exposes observable state, a strong verifier checks the **postcondition**, not just the agent’s intent or stdout. ## Evidence must be bound to the exact candidate A common failure looks like this: ```text A → tests PASS A → review PASS A → agent fixes one review comment → B B → merge ``` The tests and review belonged to `A`, not `B`. GitHub documents this concretely for Copilot code review: new pushes are not automatically re-reviewed unless `Review new pushes` is enabled or another review is requested. A Copilot approval can also be dismissed after later commits.[^github-code-review] The broader lesson does not depend on Copilot: ```text verification_result without candidate identity = incomplete evidence ``` A useful record should contain at least: ```yaml candidate_sha: 73ab91f... contract_version: 3 environment_fingerprint: py311-linux-lock-8f42 verifier_version: cli-contract-v5 command: pytest tests/cli tests/regression result: pass started_at: 2026-09-10T17:21:04Z finished_at: 2026-09-10T17:22:11Z ``` This is an illustrative harness contract, not a standard. If the candidate SHA changes, the harness must invalidate every piece of evidence whose result can depend on the change. ## Not every change requires rerunning absolutely everything Invalidating evidence does not mean always running the most expensive suite from scratch. A harness can represent dependencies: ```text src/cli/output.py ├─→ cli_contract_tests ├─→ text_mode_regressions └─→ type_check docs/cli.md └─→ docs_build ``` If only `docs/cli.md` changes, a database test does not become more informative by being rerun. But this optimization needs an explicit, conservative dependency model. If the system cannot establish whether a change can affect a piece of evidence, the fail-closed choice is to rerun it. The goal is **fresh evidence**, not the largest possible command count. ## Tests are also part of the diff, and they can be wrong A coding agent can often modify all of these at once: ```text implementation existing tests new tests fixtures snapshots CI config ``` That creates an important risk: the oracle can move to match the bug. For example: ```diff - assert run("bad-filter").returncode == 2 + assert run("bad-filter").returncode == 0 ``` The suite stays green while the contract is violated. Diff review should therefore treat changes to tests and graders as changes to the **acceptance criterion**, not as incidental support files. Depending on the task, useful defenses include: ```text hidden or out-of-band tests contract tests outside the editable workspace base → head comparison of tests and snapshots protected CI or graders reference solution used to validate the evaluator ``` Anthropic recommends a reference solution that establishes task solvability and validates grader configuration, and it recommends designing graders to resist bypasses.[^anthropic-evals] ## Diff review covers questions a suite may never encode A useful diff review is not a second full read of the repository. It targets risk boundaries: ```text scope: did the change touch paths outside the task? public API: did signatures, formats, or contracts change? config/CI: was a protection or verifier weakened? dependencies: were unnecessary packages or code added? data: are there migrations or destructive changes? tests: was coverage added, or was the target moved? generated: were accidental artifacts committed? security: did permissions, network, or secret access expand? ``` Some of this can be automated with path rules, diff-size limits, schema checks, or scanners. Hard semantic questions may need a human or model-based review. GitHub, for example, explicitly says that pull requests created by Copilot cloud agent deserve the same thorough review as any other contribution before merge.[^github-agent-review] ## A model reviewer is probabilistic evidence, not an oracle A second model can notice inconsistencies that tests do not express: ```text does the change actually solve the issue? does the new abstraction violate an implicit invariant? is there an uncovered edge case? does the diff introduce unnecessary complexity? ``` But an LLM grader or reviewer is still non-deterministic. Anthropic recommends calibrating LLM-as-judge graders against human experts and giving the grader an explicit option such as `Unknown` when it lacks enough information.[^anthropic-evals] Using “another agent” does not guarantee independence either. It may share: ```text the same model the same incomplete context the same repository instructions the same conceptual mistake ``` A model-based reviewer therefore complements tests and human review. It does not turn an inference into ground truth. ## Reviews become stale too Even a strong review can become obsolete when any of these change: ```text candidate SHA relevant base branch contract version verifier configuration instructions used by the reviewer ``` GitHub documents another concrete boundary: Copilot code review reads custom instructions and skills from the PR’s **head branch**. Those instructions are therefore part of the review context and can change alongside the code being reviewed.[^github-code-review] A serious harness stores enough provenance to know what was reviewed and under which configuration. ## A stop condition is not the same as a success condition An agent loop may stop for several reasons: ```text ACCEPTED REWORK_REQUIRED HAND_BACK_TO_HUMAN BLOCKED_BY_AUTHORITY ENVIRONMENT_FAILURE BUDGET_EXHAUSTED NO_PROGRESS ``` Every one of these is a stop condition. Only one means acceptance. This avoids a dangerous shortcut: ```text model stops calling tools → task complete ``` The model may stop because it incorrectly believes it is finished, because context was lost, or because it does not know how to continue. The harness should derive closure from the contract and evidence. ## A success gate can be made explicit As a conceptual model, not a product standard: \[ \operatorname{Accept}(h)= C(h)\land V(h)\land D(h)\land P(h)\land F(h) \] For candidate \(h\): - \(C\): every required acceptance criterion has evidence - \(V\): required verifiers pass - \(D\): diff/review leaves no known blocker - \(P\): required policy and approvals remain valid - \(F\): all required evidence is **fresh** for the current candidate and environment If any condition fails, the state is not `ACCEPTED`. That is intentionally different from: ```text Accept(h) = agent_says_done(h) ``` ## Failure stop conditions protect quality and cost as well A loop without an exit policy can repeat the same action forever. A reasonable handback can trigger on an observable condition such as: ```text a required credential or approval cannot be obtained by the agent the spec has two materially different interpretations the evaluation environment is unstable and cannot produce valid evidence the same failure repeats without new information a required action exceeds policy an explicit attempt/time/cost budget is exhausted ``` “Same failure without new information” is more useful than pretending there is one universal retry count. The concrete budget depends on the task, cost, and criticality. `HAND_BACK_TO_HUMAN` is not necessarily a model failure. It can be the correct outcome when the next decision requires authority the agent does not possess. ## The evaluation environment is part of the result The same patch can produce different outcomes when any of these differ: ```text operating system runtime version lockfile/dependencies external services fixtures clock/timezone prior state available CPU/memory ``` SWE-bench uses Docker containers to create reproducible evaluation environments. Its harness prepares environments, applies patches, runs tests, and determines task outcomes inside them.[^swebench-harness] Anthropic recommends starting each trial from a clean, isolated environment. Shared state, caches, and resource exhaustion can introduce correlated failures or artificial advantages.[^anthropic-evals] A result without environment identity is therefore weaker evidence than it first appears. ## Evaluating a coding agent means evaluating model + harness + task + grader A coding-agent benchmark does not measure a model in isolation. The observed result depends on: ```text model agent harness prompt/instructions tools context/repository snapshot sandbox/environment budget task specification graders/tests scoring rule ``` Anthropic states the boundary directly: when evaluating “an agent,” it is evaluating the harness and model together.[^anthropic-evals] Two percentages should not be treated as the same measurement if the harness, budget, environment, or suite version differs. ## A green benchmark can still measure the wrong thing In February 2026, OpenAI stopped reporting SWE-bench Verified for frontier launches after finding contamination plus specification and test problems. Among other issues, its audit found tests that rejected functionally correct solutions and evidence of exposure to benchmark problems or solutions during training.[^openai-swebench-verified] The lesson is not “SWE-bench is bad.” The lesson is that **the benchmark also needs evaluation**. That point became more important in July 2026. An OpenAI audit of SWE-Bench Pro estimated that approximately **30% of tasks were problematic under its audit methodology**, including task-specification, test, and grading issues.[^openai-swebench-pro] That ~30% is the result of that specific audit. It is not a universal benchmark error rate. Before interpreting a score, inspect: ```text which task ran which snapshot and environment were used what the agent could see what budget it received which grader decided pass/fail whether the grader accepts alternative valid solutions whether contamination or leakage is plausible ``` ## One run does not estimate reliability Agents are stochastic. Passing a task once demonstrates possibility, not consistency. Anthropic distinguishes two questions: - **pass@k**: probability of obtaining at least one successful result in `k` attempts - **pass^k**: probability that all `k` attempts succeed[^anthropic-evals] If a product can try several candidate solutions and select a valid one, `pass@k` may be relevant. If every user expects repeatable success, the consistency captured by `pass^k` answers a different question. They are not interchangeable. For a task with per-trial success probability \(p\), assuming independent and identically distributed trials: \[ P(\text{at least one success in }k)=1-(1-p)^k \] \[ P(\text{all }k\text{ succeed})=p^k \] The independence assumption matters. If every trial shares a broken service or contaminated state, those formulas do not describe the experiment correctly. ## Capability evals and regression gates answer different questions A capability eval asks: ```text which difficult tasks can this system solve, and with what success distribution? ``` A regression suite asks: ```text does the system still solve the behavior we already consider mandatory? ``` Anthropic recommends regression evals that protect established behavior at pass rates close to 100%, while capability evals should retain enough difficulty to distinguish progress.[^anthropic-evals] For CI on one concrete task, we usually want a **deterministic regression gate**. When evaluating an agent harness, we also want a **capability and reliability distribution** across many tasks and trials. Conflating the two creates opposite mistakes: ```text using a probabilistic benchmark as the merge gate for one PR using a trivial 100% suite to claim capability progress ``` ## Worked example: closing the `--json` task correctly Return to the example task. The agent ends with this candidate SHA: ```text 73ab91f ``` The harness runs: ```text 1. contract verifier - `acme users list --json` → valid JSON - invalid filter → exit 2 2. regression verifier - existing text-mode tests → PASS - Python API contract tests → PASS 3. static verifier - type check → PASS - lint → PASS 4. diff verifier/review - paths outside scope → none - existing tests weakened → no - CI / permissions → unchanged 5. freshness gate - every result belongs to 73ab91f - working tree clean - contract_version = 3 ``` The evidence bundle might look like this: ```yaml candidate_sha: 73ab91f contract_version: 3 environment_fingerprint: py311-linux-lock-8f42 required_checks: cli_contract: pass text_regression: pass python_api_contract: pass typecheck: pass lint: pass diff_review: blocker_count: 0 test_oracle_changed: false freshness: all_evidence_matches_candidate: true stop_reason: ACCEPTED ``` Again, this is an illustrative structure. If the agent then changes one line and creates `84cd120`, the correct state returns to: ```text 84cd120 → NOT YET VERIFIED ``` A recent `PASS` is not inherited by proximity in time. ## What the harness should store to explain closure An auditable closure needs less narrative and more provenance: ```text task_id + contract_version base_sha + candidate_sha workspace/environment identity changed paths + diff digest verifier names + versions + commands start/end time + exit/result relevant artifacts reviewer identity/config + reviewed SHA approval/policy evidence when applicable known blockers stop_reason ``` With that information, the system can reconstruct **why** a task was accepted and what changed if a regression appears later. Without it, “it passed yesterday” is difficult to distinguish from “we think something similar passed yesterday.” ## Trade-off: verification coverage, latency, and cost More checks do not automatically produce a better system. A suite can take hours, depend on expensive services, or be flaky. A model-based reviewer adds cost and uncertainty. Human review on every iteration can destroy throughput. The useful pattern is to layer evidence: ```text fast + deterministic + local → on each relevant iteration dependency-selected → after a bounded scope change expensive / integration / human → at milestones or risk boundaries full regression → before release when the contract requires it ``` But the harness must not silently downgrade a required check because it is expensive. If a mandatory verifier cannot run, the state is `BLOCKED/UNVERIFIED`, not `PASS`. ## Production implication: “done” should be a derived state The model proposes code and may suggest that it has finished. The harness computes final state from verifiable evidence. The operational rule is: ```text agent intent → candidate candidate → evidence evidence + contract + policy → stop decision ``` That supports autonomy without turning model confidence into merge authority. A strong harness can close quickly when evidence is sufficient, reopen verification when the candidate SHA changes, and hand back when authority or verifiability is missing. The next chapter extends this logic to work that lasts far longer than one session: memory, subagents, recovery, integration, merge, and observability must preserve not only the work but also the provenance of which evidence remains valid. ## Primary references [^anthropic-evals]: Anthropic, [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents), January 9, 2026. Used for task/trial/grader/outcome/evaluation-harness definitions, grader types, stable isolated environments, outcome-based grading, reference solutions, bypass resistance, human calibration, and pass@k/pass^k. [^github-code-review]: GitHub Docs, [Using GitHub Copilot code review on GitHub](https://docs.github.com/en/copilot/how-tos/copilot-on-github/use-copilot-agents/copilot-code-review). Used only for current re-review/new-push semantics, approvals, and custom-instruction context. Other Copilot capabilities are not generalized to the harness. [^github-agent-review]: GitHub Docs, [Review output from Copilot](https://docs.github.com/en/copilot/how-tos/copilot-on-github/use-copilot-agents/review-copilot-output). Used for GitHub's first-party guidance to review cloud-agent changes before merge and its approval/workflow boundaries as one external-control example. [^swebench-harness]: SWE-bench, [Evaluation Harness Reference](https://github.com/SWE-bench/SWE-bench/blob/main/docs/reference/harness.md). Used for the Docker evaluation harness that prepares environments, applies patches, runs tests, and determines reproducible outcomes. [^openai-swebench-verified]: OpenAI, [Why SWE-bench Verified no longer measures frontier coding capabilities](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/), February 23, 2026. Used for findings on contamination, task/test mismatch, and limits of treating that benchmark as frontier-capability signal. [^openai-swebench-pro]: OpenAI, [Separating signal from noise in coding evaluations](https://openai.com/index/separating-signal-from-noise-coding-evaluations/), July 8, 2026. Used for its SWE-Bench Pro audit and contextualized estimate of problematic tasks. The number is not extrapolated beyond the reported methodology.