Chapter 2 — Offline eval sets: curation, hard negatives, contamination, and versioning¶
An offline set is not a folder of examples that "look representative." It is a versioned measurement instrument.
If its cases are ambiguous, too easy, duplicated across splits, exposed during development, or silently mutated, the final number can be perfectly reproducible while answering the wrong question.
This chapter asks one concrete question:
How do we build an eval set that measures a real hypothesis, finds useful failures, and keeps its meaning as the system and data change?
That requires separating five problems:
- where cases come from
- what coverage we intend
- how to build difficult cases without turning the set into a collection of curiosities
- which kinds of exposure or leakage can invalidate a conclusion
- which exact version of data, labels, and protocol produced a result
NIST AITE provides one useful extreme of this design: blind data inside a sequestered environment to reduce train/test contamination risk.2 Product teams often operate at the other end, growing internal banks from bugs and user feedback. Both approaches are only interpretable when the provenance and history of each case are known.
Version the evidence; rotate what you want to discover
A trustworthy eval set is not created by saving examples to JSONL. Each item needs provenance, a split unit that preserves dependencies, and an explicit role. Dev, regression, holdout, and challenge are not four names for the same bank. The release used for comparison stays frozen; new failures feed the next version.
How to read it: first decide which unit must not cross splits, then assign each case to a bank with an explicit role and freeze the release with a reproducible manifest. A useful hard negative tests a concrete boundary. An overlap scanner helps investigate leakage, but it does not prove the model never saw the content. New failures enter the next version instead of rewriting history.
The object is an eval release, not "the dataset"¶
Represent one evaluation release as:
where:
videntifies an immutable versionid_iis a stable case identifierx_icontains the task/input and, when needed, initial environment statey_icontains the reference, success criteria, or expected checksm_icontains metadata required for stratification and reproductionp_icontains provenance: source, date, permissions, transforms, and lineage
This definition prevents a common mistake: calling a file eval_v1 while editing it in place. If an item, label, split, or result-affecting transform changes, we are no longer measuring with exactly the same D_v.
The smallest unit must be auditable¶
A case should let us resolve at least:
item_id
source_id / provenance
source_snapshot_or_timestamp
split
capability_or_risk_tags
input / initial_state
reference_or_success_criteria
labeling_guideline_version
transform_pipeline_version
created_at / reviewed_at
These fields do not have to live in one file. They do have to be resolvable from the release manifest.
Start with a coverage hypothesis¶
"Representative" does not mean "random" by default.
Before sampling, define which distribution or risk regions you want to observe. For a support assistant, relevant axes might include:
- intent: refund, shipping, billing, account
- complexity: one policy, multiple policies, policy conflict
- dependency: no tool, one tool, several tools
- risk: reversible, external side effect, sensitive action
- language, input length, and noise
- state: eligible, nearly eligible, ineligible
We can describe coverage through strata z:
but the size of each stratum must follow the evaluation question. To estimate production performance, weights should approximate the relevant operating distribution. To find rare but expensive regressions, deliberately oversampling those regions can be correct, but the resulting average must not be presented as a production rate.
The rule is simple:
the sampling policy is part of the metric.
A 92% score on a deliberately adversarial bank and a 92% score on stratified traffic do not mean the same thing.
Where cases should come from¶
A healthy eval set usually mixes sources because each source exposes different failures.
1. Curated production cases¶
Bugs, escalations, support tickets, and failed traces provide realism.
Strength: they sit close to risks that have already occurred.
Risk: if we only turn known failures into tests, the suite becomes a historical archive and does not explore nearby failure modes.
2. Expert-designed cases¶
Experts can target invariants, policy boundaries, security properties, and corners that have not yet appeared in traffic.
Strength: strong control over the mechanism being tested.
Risk: authors can introduce linguistic artifacts or assumptions that do not exist in real use.
3. Generated or transformed cases¶
We can vary entities, quantities, ordering, noise, tool state, or constraints.
Strength: inexpensive coverage expansion.
Risk: many synthetic variants are not genuinely new situations. One thousand paraphrases of one mechanism are not one thousand independent units of evidence.
4. Adversarial cases and hard negatives¶
These target a boundary where the system should change its decision.
Dynabench formalized one version of this idea through human-and-model-in-the-loop data creation: annotators try to produce valid examples that fool the current target model.3 Adversarial NLI used a related iterative process to collect hard examples against successive models.4
The lesson is not "always use adversarial data." It is that difficulty can be designed around a specific failure and then revalidated, rather than assuming random examples will cover that boundary.
What makes a useful hard negative¶
A hard negative is not simply "a difficult example."
It is especially useful when almost everything stays constant while one condition changes the correct decision.
Support example:
A: order delivered 29 days ago + item intact → return allowed
B: order delivered 31 days ago + item intact → return denied
If the governing policy uses a 30-day boundary, the pair forces the system to use the causal variable rather than a shortcut such as "mentions a return → approve."
Represent one pair as:
where Δ* is the smallest difference intended to justify the label change. The ≈ is deliberate. In language and real systems, perfect identity of every nuisance variable is rarely possible.
A pairwise consistency signal can be:
PC does not replace overall accuracy. It answers a different question: does the system make the correct decision on both sides of a deliberately constructed boundary?
Hard does not mean exotic¶
A case can be hard because it:
- discriminates between nearly identical conditions
- combines two rules that conflict
- mixes relevant evidence with a plausible distractor
- requires the system not to act
- requires correct recovery after a partial failure
- breaks a spurious correlation that worked on easy examples
The goal is not to fill the set with riddles. The goal is to apply pressure where the system could learn the wrong shortcut.
The anti-pattern: generating every hard case against the current model¶
If every difficult case is selected because model_A fails it, the dataset is conditioned on model_A.
That can be useful for diagnosis, but it creates two risks:
- evaluating a future version almost entirely on the weaknesses of a previous one
- treating "beats this model" as equivalent to "represents an important product risk"
Dynabench explicitly treats benchmarking as dynamic: models and data collection feed back into one another.3 That is a property of the protocol, not permission to mix rounds without identifying them.
For a product eval, preserve where a hard case came from:
hardness_source: human | production_failure | model_adversarial | transform
hardness_target: model_revision | workflow_revision | policy_boundary
round: 2026-09-r2
That lets us ask whether an improvement generalizes or only solves the generator that produced the set.
Four contamination problems that should not share one label¶
"Contamination" is used for several different failures. Naming them separately makes the response clearer.
1. Training exposure¶
Test content, or sufficiently close material, may have appeared in pretraining, fine-tuning, or distillation.
This matters especially for public benchmarks. A high score can combine generalization and memorization, and outside the training pipeline we usually cannot prove total absence of exposure.
NIST AITE reduces this risk by keeping evaluation data blind inside a sequestered environment.2
LiveBench uses another strategy: it introduces new questions periodically and draws from recent sources with objectively verifiable answers to limit potential contamination.5
Neither strategy establishes a universal property of "contamination-free." Each reduces a specific exposure channel under its own protocol.
2. Cross-split leakage¶
The same case, a near duplicate, or the same entity/template can cross from authoring or dev into the holdout.
Exact duplicates are the easy case. More subtle examples include:
- paraphrases of one item
- multiple windows from the same source document
- related issues in one repository that share a solution
- conversations from the same customer split across banks
- synthetic variants produced from the same seed
- initial and final states from one episode separated across train/test
The split unit therefore should not always be a row. It might need to be a document_id, customer_id, repo_id, incident_family, or generation_seed.
3. Development leakage¶
A test set can be absent from model training and still stop functioning as a holdout.
This happens when a team repeatedly:
- inspects failures from the final test set
- tunes a prompt or policy against those same items
- selects models after every iteration by looking at test results
- changes a grader to accommodate outputs observed on the test
After enough iterations, the team has optimized the system against the set without performing gradient descent on it.
The practical response is to separate dev/calibration from holdout/release, and control who can inspect the holdout and when.
4. Temporal leakage¶
If a task is meant to reproduce a decision at time t, it cannot use information that only existed after t.
Examples include:
- RAG evaluated against a corpus snapshot from after the ticket date
- incident prediction using a postmortem written after the incident
- a research agent evaluated on a web snapshot that already contains a later published answer
The snapshot date is part of the case.
A duplicate scanner cannot certify the absence of contamination¶
Define an overlap audit:
This is useful for finding leakage candidates. It is not proof of a "clean dataset."
The result depends on:
- representation
- similarity function
sim - threshold
τ - normalization
- comparison unit
- available provenance
An exact-match scan has few false positives but misses paraphrases. An embedding scan can surface semantic equivalence and also flag legitimately similar cases. Provenance review remains necessary.
The GPT-4 Technical Report is a useful historical example of why contamination is reported as a methodology rather than a magic boolean. OpenAI measured overlap between benchmark items and training data, then separated contaminated and non-contaminated subsets under its detector while documenting limitations.8
Split on the unit that can carry leakage¶
Suppose a RAG eval has 20 questions per document. A random split by question can place 15 questions from one document in dev and 5 from that same document in test. The test is no longer asking, "does the development process generalize to unseen documents?" It is asking something narrower.
Depending on the objective, the split may need:
For a coding agent:
For support:
There is no universal split unit. It should match the channel through which information could transfer artificially between banks.
Dev, regression, and holdout banks serve different jobs¶
A useful organization separates at least three banks.
Dev / calibration set¶
Visible during construction.
Use it to:
- write and repair tasks
- debug graders
- inspect transcripts
- iterate on prompts and policies
- discover broken cases
It should not later be presented as independent evidence for an improvement that it helped shape.
Regression bank¶
Known cases the system is expected to pass.
Anthropic distinguishes capability evals from regression evals and notes that mature capability cases can graduate into regression coverage once behavior stabilizes.1
Its purpose is to catch breakage of an already acquired property.
Holdout / release set¶
This bank should not participate in every product iteration.
Its purpose is to confirm that a decision made with dev and regression evidence generalizes to cases not used to optimize that decision.
A holdout that the whole team reads every day no longer acts as a holdout, even if the file is still named holdout.jsonl.
A fourth bank can help: a rotating challenge set¶
Systems and models saturate static evals.
LiveBench illustrates a public rotation strategy: periodically adding new questions from recent sources to limit exposure and preserve difficulty.5
A product team can apply the same principle without copying that benchmark:
- preserve a stable release for longitudinal comparability
- add a fresh challenge bank to find new capabilities and failures
- do not mix the two averages without identifying their version and composition
This avoids a false choice between two competing goals: historical comparability and freshness.
Versioning means being able to reconstruct the result¶
A score without dataset identity is incomplete evidence.
At minimum, record:
eval_suite: support-policy
dataset_version: 3.2.0
manifest_sha256: ...
split: release
item_count: ...
source_snapshot: ...
labeling_guidelines: 7
transform_pipeline: 4
model_revision: ...
workflow_revision: ...
grader_revision: ...
harness_revision: ...
Hugging Face Datasets can pin a dataset revision by tag, branch, or commit SHA, and maintains fingerprints derived from data state and applied transforms.67 Those are useful reproducibility mechanisms, but a technical fingerprint does not replace the semantic version of an evaluation.
Two datasets can have different hashes because of an irrelevant ordering change. Conversely, keeping the name v3 means little if the content is silently changed.
Canonical manifest¶
A robust pattern is to build an ordered manifest and hash it:
The manifest can include item IDs, payload hashes, split, source revision, label revision, and transform revision.
The exact function matters less than two properties:
- the same input produces the same identity
- any material change becomes visible and requires a new identity
Which changes require a new version¶
Not every change carries the same semantics.
Patch¶
A correction that is not intended to change the task:
- typo in metadata
- caption that is not used by the grader
- repaired provenance link
The artifact hash still changes and the patch should remain recorded.
Minor¶
Coverage changes while the main construct remains the same:
- new items
- new hard negatives
- a new region or language
- removal of invalid cases
Do not compare the averages from v3.1 and v3.2 as if their denominators were identical without running both systems on the same release.
Major¶
The meaning of success or the evaluated boundary changes:
- new rubric
- new outcome
- policy change that redefines labels
- move from single-turn behavior to a complete workflow
- new environment or tool contract
That is a new evaluation even if the product name stays similar.
The major.minor.patch convention is only an example. What matters is documenting what each class of change means.
Do not silently relabel history¶
Suppose we discover that 12 items had incorrect labels.
There are two different questions:
- what result did the system produce under
D_3.1as it existed then? - what result does that same system produce under corrected
D_3.1.1?
Rewriting history destroys the first answer.
A reproducible practice preserves:
- the original release
- the corrected patch
- a changelog of added, removed, and relabeled items
- recomputed results when systems need to be compared on the same basis
Curation is also about finding artifacts¶
A dataset can look diverse while enabling a trivial shortcut.
Before trusting it, try deliberately weak baselines based on features such as:
- input length
- words or templates correlated with one label
- presence of tool names
- answer position
- accidental metadata
- filename format
- stylistic differences between positive and negative cases
If a cheap heuristic predicts the label, the problem may be the dataset rather than the evaluated system.
Adversarial NLI and Dynabench were motivated in part by the observation that static datasets can contain exploitable patterns and quickly become saturated. Their collection procedures use interaction with models to find examples that break current heuristics.43
Reference solutions and solvable tasks¶
Anthropic recommends unambiguous tasks with a known reference solution that passes the graders. Extremely low pass rates can indicate a broken task or grader rather than only a weak agent.1
That suggests an item admission gate:
source/provenance valid
→ task understandable
→ reference solution exists
→ grader accepts reference
→ no forbidden leakage
→ split assignment valid
→ item versioned
A hard case without a verifiable solution is just difficult noise.
Three design examples¶
Case A — Retriever for internal policies¶
We want to measure whether a retriever finds the correct policy.
Design:
- split by
policy_document_id, not query - positives with explicit evidence
- hard negatives with a neighboring but incorrect policy
- fixed corpus snapshot
- visible dev set and a holdout built from documents not used for tuning
- local retrieval metric followed by end-to-end confirmation
When a new policy enters production, record whether it belongs in the next release or in a challenge bank. Do not silently add it to v1.
Case B — Support agent near an eligibility boundary¶
We want to prevent incorrect approvals near a policy cutoff.
Construct pairs such as:
Then vary product type, language, and customer history so the system cannot solve the set by memorizing one template.
The hard pair measures sensitivity to the policy boundary. A stratified traffic bank measures general performance. Do not mix those distributions into one percentage without declaring the weights.
Case C — Coding agent¶
We want to evaluate repository-level bug fixes.
Leakage risks include:
- related issues from the same repository
- later commits containing the solution
- tests introduced after the task date
- clones or forks placed in different splits
Provenance should therefore include repository, base commit, issue family, date, and test revision. A mutable snapshot of main is not a reproducible task.
Item admission checklist¶
Before adding a case to an offline release, answer:
- Which hypothesis or risk does it cover?
- What is its source/provenance, and are we allowed to use it?
- Which unit must remain grouped to prevent leakage across splits?
- Is there a verifiable reference or success criterion?
- Does the grader accept the reference and reject an obvious failure?
- Is this a normal case, regression, hard negative, or challenge case?
- Has it been used for tuning or inspected during development?
- Which temporal snapshot does it require?
- Which guideline and transform versions produced its label?
- Which item ID and hash let us reconstruct it?
If we cannot answer question seven, we also cannot claim the case is a clean holdout.
What to report with an offline result¶
Do not publish only:
Attach:
dataset_version / manifest hash
split + access policy
sampling / strata / weights
item count
source snapshot / cutoff
known contamination status
model + workflow + harness + grader revisions
number of trials where relevant
metric definition
excluded / invalid items with reason
When comparing two systems, run them on the same release unless the purpose is explicitly to study distribution change.
Production implication¶
A healthy offline set needs two properties that seem to conflict:
- enough stability to catch regressions and compare versions
- enough change to absorb new failures, product changes, and benchmark saturation
The answer is not to mutate one file forever. Maintain a family of artifacts with different jobs:
Each has a purpose, access policy, and identity.
The central principle is:
difficulty without provenance is noise; freshness without versioning breaks comparability; a holdout used for optimization stops being a holdout.
The next chapter adds another source of uncertainty. Even with a well-built dataset, the result still depends on how outputs are judged and how well human and automated graders agree.
References¶
-
Anthropic Engineering, Demystifying evals for AI agents, 9 Jan 2026. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ↩↩
-
NIST, Announcing NIST's Artificial Intelligence Technology Evaluation (AITE), 27 Jul 2026. https://www.nist.gov/news-events/news/2026/07/announcing-nists-artificial-intelligence-technology-evaluation-aite ↩↩
-
Douwe Kiela et al., Dynabench: Rethinking Benchmarking in NLP, NAACL 2021. https://aclanthology.org/2021.naacl-main.324/ ↩↩↩
-
Yixin Nie et al., Adversarial NLI: A New Benchmark for Natural Language Understanding, ACL 2020. https://aclanthology.org/2020.acl-main.441/ ↩↩
-
LiveBench project, official repository and release changelog. The benchmark periodically refreshes questions to limit potential contamination. https://github.com/LiveBench/LiveBench ↩↩
-
Hugging Face Datasets, Load. The
revisionparameter can pin a tag, branch, or commit hash. https://huggingface.co/docs/datasets/main/loading ↩ -
Hugging Face Datasets, The cache. Dataset fingerprints track data state and applied transforms. https://huggingface.co/docs/datasets/main/about_cache ↩
-
OpenAI, GPT-4 Technical Report, 2023. Appendix contamination analysis describes benchmark overlap methodology and contaminated/non-contaminated subsets under the detector. https://cdn.openai.com/papers/gpt-4.pdf ↩