Skip to content

Evaluating AI models

Evaluating AI means measuring whether a model, system or product meets an objective under specific conditions. A public benchmark answers a limited question. By itself it does not prove that an application is reliable, fast, safe or useful to its users.

The first decision in an evaluation is not which metric to use. It is what object is being evaluated.

EVALUATION TARGET
The same bad answer can fail at different layers
Before choosing a metric, define the system boundary you want to measure. Each layer adds behaviour and new failure modes.
Practical rule Do not use a level-1 metric to claim that level 4 works.

Each layer introduces different failures and criteria.

The 60-second answer

EVALUATION STACK
Six layers, six different questions
A robust evaluation combines complementary signals. Each layer reduces a different kind of uncertainty; no single score summarizes the whole system.
DECISION
Model / prompt / system / productThe useful metric is the one that changes a concrete decision about that layer.

No layer replaces the others. The key is connecting every metric to a product decision.

1. Define the task and the cost of error

“Quality” is too broad. An evaluation needs an observable contract.

For a data extractor:

  • required fields;
  • valid formats;
  • precision and recall;
  • handling of missing values;
  • cost of a false positive.

For an agent with tools:

  • action selection;
  • correct arguments;
  • operation order;
  • idempotency;
  • final state;
  • user-facing message.

For a voice assistant:

  • intent understanding;
  • entities;
  • time to first audio;
  • interruptions;
  • task success;
  • duplicate closure.

The metric must reflect the failure that matters. Optimizing textual similarity when the real problem is executing a transfer twice measures the wrong surface.

2. Build a case taxonomy

An average hides where the system fails. The evaluation set should label relevant dimensions:

  • intent or task type;
  • difficulty;
  • language and market;
  • length and noise;
  • ambiguity;
  • need for external knowledge;
  • tool use;
  • impact of error;
  • affected population or segment.

Then calculate performance by segment, not only one global number.

A taxonomy enables actionable questions: “Does the new model improve long queries but regress on Spanish proper names?” That helps a decision. “It gained two points” does not.

3. Use your own golden set

The golden set contains real or designed examples representative of the domain. Every case needs:

  • input;
  • relevant context;
  • expected result or rubric;
  • segment labels;
  • failure severity;
  • provenance and date.
COVERAGE → CASES → REGRESSIONS
A useful reference set represents failure modes, not just examples
The taxonomy defines which dimensions matter; the reference set turns those dimensions into reproducible cases; real incidents return as permanent regressions.
REAL INCIDENTsomething fails in production
REGRESSIONit becomes a reproducible case
VERSION N+1the failure is no longer forgotten
SEGMENTreport by relevant groups; a global average can hide a regression
VERSIONkeep stable cases for comparison and add new failures as regressions
TRACEinput, criterion, labels, severity, and provenance should travel with every case
Primary sources: Liang et al., HELM (2022) · OpenAI Evals API

It should be versioned like code. When an incident appears, add a regression case. When the product changes, update the distribution while keeping a stable subset for comparing versions.

Size alone does not guarantee coverage. Prioritize representative cases and critical failures, then expand the set where uncertainty or risk requires it.

4. Understand what a benchmark measures

A benchmark provides standardization and comparison. MMLU, for example, measures multiple-choice answers across many academic domains.1 HELM proposed a broader and more transparent evaluation framework built around scenarios, metrics and documented limitations.2

Before using a score, ask:

  • does the task resemble the application?
  • does the format privilege one capability?
  • are the answers unambiguous?
  • could the model have seen the data during training?
  • does the metric capture severity or only average correctness?
  • are there confidence intervals and enough samples?

A benchmark may measure academic knowledge while saying almost nothing about tool calling, conversation, latency or operational reliability.

Contamination and saturation

Public benchmarks can appear in training corpora or inspire very similar data. When labs repeatedly optimize against a static test, the score becomes a weaker estimate of generalization.

Contamination can be exact or semantic. It is difficult to detect when training data is not public.

PUBLIC TEST → EXPOSURE → SCORE → NEW EVIDENCE
A benchmark loses strength as a test when it enters the optimization loop
The score remains observable, but it can mix capability with test exposure or repeated adaptation to the same target. The defense is not to ignore benchmarks: it is to refresh and separate the evidence.
CONTAMINATIONthe test enters the data or signals used to build the system
test→direct / semantic exposure→evaluation advantage

Exact overlap is only one case. Detecting semantic contamination is harder, especially without access to the training corpus.

SATURATION / OVER-OPTIMIZATIONthe test becomes a repeated target of the development loop
static benchmark→iterations against the same score→less information about new cases

This does not require copying the test into training. Reusing a static test as a selection target can reduce its value as independent evidence.

PUBLIC ≠ CLEANa benchmark being standard does not prove it remains independent of training
SCORE ≠ GENERALIZATIONthe score needs context about exposure, date, protocol, and distribution
REFRESH + VERIFYcombine new or private cases with executable signals and contamination analysis when possible
Primary sources: Sainz et al. (2023) · contamination levels and measurement · Jain et al., LiveCodeBench (2024) · recent problems + execution · Magnusson et al., PALOMA (2023) · benchmark decontamination

LiveCodeBench designed a code evaluation that updates with recent problems and executes solutions to verify them.4 The broader principle is useful: where possible, a live and verifiable test resists superficial optimization better.

5. Use deterministic metrics when they exist

Not everything needs a generative judge.

Use rules or execution to:

  • validate JSON and schemas;
  • compare numeric values;
  • run tests;
  • check citations and URLs;
  • verify API arguments;
  • inspect final state;
  • measure latency and cost.

A deterministic metric is usually cheaper, reproducible and auditable. Generative evaluation should be reserved for dimensions that genuinely require judgement.

SIGNAL → VERIFIER → DECISION
Check the failure where the ground truth lives
Before asking another model to judge an output, ask whether success is already directly observable in structure, execution, or state. Only dimensions without a deterministic criterion need a rubric.
ASTRUCTURE / VALUE

Deterministic checker

JSON / schemanumeric valueURL / citation
output→rule→pass / fail

Use it when the condition can be decided without interpretation.

BEXECUTION / STATE

Run and inspect

testsAPI argsfinal statelatency / cost
action→environment→state

The source of truth is the system, not the text describing it.

CJUDGEMENT

Rubric + human reference

usefulnesstoneclaritypreference
case→human→criterion
AT SCALEcalibrated LLM judge→human if disagreement / high impact
SAME CASE, MULTIPLE SIGNALSAn agent books an appointment
FORMATvalid JSON✓ checker
ACTIONcorrect arguments✓ checker
STATEevent created once✓ system
MESSAGEclarity for the user→ rubric
VERIFY AT THE SOURCEif the result exists in an API, database, test, or clock, measure it there
COMBINEthe same case can have deterministic checks and a subjective rubric
SCALE AFTERWARDan automated judge enters only where judgement is required and within the calibrated perimeter
Primary sources: Jain et al., LiveCodeBench (2024) · OpenAI Graders API · Zheng et al., LLM-as-a-Judge (2023)

6. Human evaluation

Humans can assess correctness, usefulness, tone, clarity or preference. For that signal to be reliable, the protocol has to specify which dimension is being judged, which examples anchor the rubric, what information is hidden from the rater, where annotation is duplicated, and how disagreements are handled.

RUBRIC → BLINDING → INDEPENDENT JUDGMENTS → DISAGREEMENT → EVIDENCE
Reliable human evaluation needs a protocol, not just votes
The goal is to turn human judgments into an auditable signal: define what is judged, reduce irrelevant cues, observe disagreement, and preserve it by dimension and segment.
PAIRWISE COMPARISON Reduces the burden of assigning an absolute scale, but only measures the preference defined by the task
ANSWER Acontent
A bettertieB better
ANSWER Bcontent

Chatbot Arena uses anonymous, randomized battles to collect pairwise preferences. That design is useful for preference; it does not replace an external source of truth when correctness can be verified.

BLINDING ≠ OBJECTIVITYhiding identity and randomizing order reduces presentation bias; it does not remove subjectivity from the criterion
AGREEMENT ≠ TRUTHagreement helps diagnose the protocol; a poorly defined criterion or shared bias can produce consensus
PREFERENCE ≠ CORRECTNESSwhen an external check exists, run it; fluency or style should not replace factual evidence
Primary sources: Chiang et al., Chatbot Arena (2024) · anonymous randomized pairwise comparison · Oortwijn et al. (2021) · systematic disagreement resolution · Shimorina & Belz (2022) · reproducible human-evaluation documentation

Preference is still not the same as truth. A more fluent answer can beat a more correct one. When an external check exists, use it and reserve human judgment for dimensions that genuinely require it.

7. LLM as a judge

A judge model can scale open-ended evaluations. It receives the input, responses and a rubric, then produces a score or comparison.

It is useful for:

  • filtering regressions;
  • comparing many variants;
  • evaluating format and coverage;
  • prioritizing samples for human review.

Risks include:

  • position bias;
  • preference for longer answers;
  • affinity with its own model family;
  • prompt sensitivity;
  • shared errors with the evaluated model.

A judge needs calibration. Compare it against a human-annotated set, measure agreement by segment and review important disagreements. For high-impact decisions, it should not be the sole authority.

AUTOMATED JUDGE → CALIBRATION → GATE
An LLM judge scales a rubric; it does not replace its validation
Before using it at scale, compare it with human decisions on the same cases, locate the segments where it disagrees, and separate automation from authority.
COMPAREthe judge against a human reference on the same cases
SEGMENTa global average can hide failures concentrated in one class of cases
RECALIBRATEif the judge model, rubric, domain, or input distribution changes
Primary source: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)

8. Evaluate the system, not only the answer

A system with retrieval or tools can fail before text generation.

SYSTEM DIAGNOSTICS
Do not score only the answer: locate where the failure began
RAG and agents add observable steps. Separating them turns a wrong output into a concrete cause and a verifiable fix.
RAGDid the right evidence arrive and get used correctly?
Queryintent
Retrieverrecallmissing document
Rankingorderingevidence ranked too low
Contextcoverage
Modelevidence useunsupported inference
Answer + citationsfaithfulnesscitation does not support claim
AGENTDid the right action produce the right state?
Intentgoal
Toolselectionwrong action
Argumentscontractinvalid input
Resultobservationerror / retry
Statereal effectduplicate / inconsistent
Final answeruser-facing message
PRINCIPLE The same “answer failure” requires different fixes when the cause was retrieval, ranking, tool selection, arguments, state or generation.

RAG

A wrong answer can come from a missing document, poor ranking, insufficient evidence, an incorrect inference or a citation that does not support the claim. Without that decomposition, the proposed fix is guesswork.

Agents and tools

Evaluate task success, unnecessary steps, forbidden actions, retries, duplicates, final state and error recovery. The trajectory shows whether the failure is in the decision, execution or final user-facing message.

Voice and realtime systems

Add temporal and acoustic measures. A correct answer arriving after an awkward pause can still fail as a product.

9. From offline to online

Offline evals provide reproducibility and fast comparison. Online metrics reveal what happens with real users.

OFFLINE → MEASURABLE HYPOTHESIS → ONLINE → PRODUCT DECISION
An offline eval is not the user outcome
Offline gives you repeatability and diagnosis. Online checks whether those signals translate into real behavior. The useful link is explicit: every offline proxy should have an associated online metric, guardrails, and a rollback criterion.
1 · PROMOTEOffline clears the threshold

The variant clears regression, critical-segment, latency, cost, and safety gates before exposure.

→
2 · EXPERIMENTA/B or controlled rollout

Measure the target metric alongside guardrails for errors, latency, cost, and risk.

→
3 · DECIDEShip, iterate, or rollback

An offline improvement that does not move the product—or breaks a guardrail—is not a win.

PROXY ≠ OUTCOMEthe offline metric forms a hypothesis; the online experiment validates the product effect
SEGMENTa global average can hide regressions by language, market, risk, or task type
GUARDRAILSdo not promote a variant on one metric if it worsens safety, errors, latency, or cost
Sources: OpenAI Evals · identify shortcomings and prevent regressions · Microsoft Research · controlled online experimentation · Microsoft Research · A/B metric design

An online experiment still needs guardrails. A variant should not reach production merely because an automated judge improved.

10. Statistical uncertainty

A small difference can be sampling noise. If you compare two versions on the same cases, preserve the per-example pairing and analyze the per-case difference before aggregation. The interval should accompany the estimated effect.

SAME TEST → PAIRED DIFFERENCES → UNCERTAINTY → DECISION
A point estimate is not a conclusion
Compare versions on the same cases, preserve the per-example difference, and report the effect together with its uncertainty. Then separate a detectable difference from one that is actually worthwhile in the product.
3 · READ MAGNITUDE + UNCERTAINTY Three conceptually different outcomes
favors A0 · no differencefavors B
AUnresolved by the sample

The interval compatible with the data still crosses 0.

BEstimated difference, small impact

The interval stays on one side of 0, but the effect remains inside the practical threshold defined for the product.

CDifference compatible with practical value

The effect and its interval exceed the improvement threshold that justifies additional cost, latency, or risk.

0 ±δpractical interval Illustrative geometry · not measured results.
CASE SAMPLING What would happen with another sample from the domain?

Case count and segment coverage determine how much you know about the target population.

STOCHASTIC GENERATION What changes when you repeat the same case?
xᵢ→run 1run 2run k

If randomness is part of the product, also measure run-to-run variability; do not confuse it with changing the sampled cases.

DESIGNsame cases → paired comparison; preserve Δᵢ before aggregation
REPORTeffect + interval + n + segments + run-to-run variability when applicable
DECISIONstatistical evidence does not replace the product's practical-value threshold
Sources: NIST · paired observations · NIST · intervals for differences · Koehn · significance tests for NLP evaluation (2004) · ASA · significance ≠ effect size/importance (2016)

For stochastic systems, separate uncertainty caused by which cases were sampled from run-to-run variability on the same case when that randomness is part of the product. The final decision should combine magnitude, uncertainty, and cost, latency, or risk; a statistically detectable difference does not by itself imply practical value.

An operational evaluation loop

OPERATIONAL LOOP
Every change must come back as evidence
Offline evals reduce uncertainty before users are exposed. Production reveals product failures that return to the evaluation set as reproducible regressions.
OFFLINEcompare under reproducible conditions
ONLINEobserve real behavior with limited exposure
1SIGNAL
Incident or need

Turn an observation into a reproducible case.

real inputcriterionseverity
2BASELINE
Measure the current system

Run the reference before changing anything.

same casessegmentscost
3CHANGE
Model / prompt / system

Form a hypothesis and measure the delta by segment.

variabilitycomparisonhuman review
4ROLLOUT
Limited rollout

Expose the variant with explicit guardrails and rollback.

guardrailsrollbackcohort
5PRODUCTION
Observe the product

Online metrics reveal what offline evals cannot see on their own.

successlatencyincidents
6REGRESSION
New failure → permanent case

What was learned returns to the evaluation set and is versioned.

reproduceversiondo not forget
BACK TO SIGNAL

Evaluation is not a final phase. It is the loop that lets a system change without forgetting what it has already learned about its failures.5

Where to go deeper in 5sigmas

Frequently asked questions

What is the best benchmark for choosing an LLM?

There is no universal one. Use benchmarks for general capabilities and your own set for the product's task, language, data, latency and risk.

How many examples does a golden set need?

It depends on diversity and the size of the improvement you need to detect. Start with representative cases and critical failures, measure segment coverage and expand where uncertainty is high.

Can open-ended answers be evaluated automatically?

Yes, through partial rules, references, execution or a judge model. Automation should be calibrated and combined with human review for subjective or high-impact dimensions.

Does a benchmark score predict user experience?

Only when the task, distribution and metric resemble the product. In many systems, latency, retrieval, tools and interface quality explain more value than a small difference between base models.

Primary sources

  1. Dan Hendrycks et al., Measuring Massive Multitask Language Understanding, 2020. ↩

  2. Percy Liang et al., Holistic Evaluation of Language Models, 2022. ↩

  3. Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. ↩

  4. Naman Jain et al., LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, 2024. ↩

  5. OpenAI, GPT-4 Research — OpenAI Evals, 2023. OpenAI describes Evals as a development tool for identifying shortcomings, preventing regressions and tracking performance across model versions. ↩