Chapter 3 — LLM-as-a-judge and human evaluation: calibration, bias, variance, and agreement¶
An automated judge can turn thousands of open-ended outputs into labels or scores. That does not make its decisions ground truth.
An expert human panel can add domain judgment. That does not remove disagreement, ambiguity, or presentation bias.
This chapter asks one concrete question:
When is a human or LLM-based grader trustworthy enough to use as evidence in a release decision?
The answer is not "when it looks reasonable," nor is it "when it agrees with humans a lot" without specifying which humans, on which cases, under which protocol. We need to separate five properties:
- rubric validity: whether the criterion encodes the behavior that actually matters
- reliability: whether the judge is stable when nothing material changes
- agreement: how often it matches an independent reference or other independent raters
- systematic bias: whether position, style, length, identity, or model family changes the judgment without changing relevant quality
- scope: the distribution and decision type for which the judge was validated
Anthropic recommends deterministic graders where possible and model-based graders where flexibility is needed, with frequent calibration against expert human judgment for subjective tasks.1 OpenAI's Graders API exposes programmatic, label-model, and score-model graders among its available mechanisms.2 That availability is a tooling capability. It does not certify a rubric or judge as valid for our product.
The judge is part of the measurement system too
The useful signal is not 'the LLM gave it 4/5.' First define the construct and rubric, calibrate on a development bank, freeze the judge identity, preserve independent judgments, and only then measure agreement, variance, and bias. The final result is an allowed scope, not universal authority.
How to read it: calibrate the rubric during development, freeze one concrete judge version, and validate it on an independent bank. Preserve human labels and repeated LLM judgments before adjudication or aggregation. Reverse positions and control style/family to probe bias. Agreement, κ, variance, and swap consistency are different diagnostics; together they define where the judge may automate and where human fallback is required.
Define what "calibration" means first¶
In this chapter, judge calibration means an operational process: validate one concrete grader version against independent decisions and controlled cases before scaling it.
That is not the same thing as probabilistic calibration.
If a grader emits pass/fail or an ordinal 1–5 quality score, that number is not automatically a probability. We should not say "70% calibrated confidence" unless the protocol actually produces a probability with an explicit statistical interpretation and that interpretation has been validated.
A useful judge identity is:
judge_version =
model revision
+ rubric/prompt revision
+ decoding/config revision
+ evidence/context supplied
+ output parser
Changing any of these can change the decision policy. Calibration therefore belongs to a judge version, not to a model brand or family in the abstract.
Choose the narrowest grader that answers the question¶
Do not use an LLM to judge a property that can be checked exactly.
Deterministic¶
Use deterministic checks when there is a verifiable invariant:
- a test passes
- a JSON field satisfies a schema
- an action produced the expected state
- a citation points to an allowed identifier
- a number falls within a defined tolerance
The main challenge here is specifying the check correctly, not approximating human preference.
LLM-as-a-judge¶
Model-based graders are useful when the criterion is semantic or admits multiple valid answers:
- clarity of an explanation
- compliance with a natural-language policy
- quality of a synthesis
- whether an answer is supported by supplied evidence
- conversational behavior that an exact rule cannot capture well
The grader should receive only the context needed to decide and should have a valid escape hatch when evidence is insufficient, such as unknown or insufficient_evidence.1
Human evaluation¶
Human evaluation is especially valuable when:
- we are defining a new rubric
- the cost of a false positive is high
- cases are ambiguous or require domain expertise
- we need to understand why criteria disagree
- the automatic judge has not yet been calibrated for that distribution
The goal is not to declare a universal winner between humans and LLMs. It is to route each decision to the mechanism capable of producing defensible evidence.
A vague rubric creates variance that a stronger model cannot fix¶
Suppose the prompt is:
What does quality mean: correctness, coverage, tone, concision, grounding, or safety?
Two judges can apply different functions while both following the instruction literally.
A useful rubric separates dimensions and anchors each one to observable criteria:
groundedness:
pass: every material claim is supported by supplied evidence
fail: at least one material claim lacks support or contradicts evidence
unknown: supplied evidence is insufficient to decide
policy_compliance:
pass: action satisfies all listed policy constraints
fail: action violates at least one listed constraint
unknown: required policy state is missing
Separating dimensions also makes disagreement diagnosable. One composite score can hide the fact that all raters agree on factuality and disagree only on style.
Humans need a protocol too¶
"Human-rated" is not a complete methodology.
At minimum, record:
rater expertise / locale
rubric_version
examples_or_anchors_version
blindness / model identity visibility
randomization policy
independent rating before adjudication
adjudication rule
item assignment
rater_id or stable pseudonym
Independence before adjudication¶
If H1 sees H2's decision before rating, we no longer have two independent observations.
A sound flow is:
item
→ H1 label
→ H2 label
→ store both
→ measure disagreement
→ adjudicate if the protocol requires a final label
Adjudication produces an operational reference. It should not erase the original disagreement, because that disagreement contains information about task or rubric ambiguity.
Blind identity when identity is not part of the criterion¶
If a rater knows which provider or model produced each answer, identity can influence the decision.
For output comparison, hide identity and randomize order unless the evaluation object explicitly requires knowing the source system.
Observed agreement: measure what actually happened first¶
For two evaluators producing categorical labels, observed agreement is:
p_o answers a simple question: on what fraction of items did they assign the same label?
It does not show where they disagree. Pair it with a confusion matrix or at least per-label counts.
Illustrative example:
Here p_o = (72 + 14) / 100 = 0.86.
This is illustrative arithmetic, not a benchmark of any model.
Cohen's κ: correcting expected agreement does not create a universal score¶
For two raters and nominal categories, Cohen proposed:5
where p_e is expected agreement from the raters' marginal label frequencies.
For the example above:
- judge:
pass=0.80,fail=0.20 - human:
pass=0.78,fail=0.22 p_e = 0.80\cdot0.78 + 0.20\cdot0.22 = 0.668κ = (0.86 - 0.668)/(1 - 0.668) ≈ 0.578
The difference between 0.86 and 0.578 is not a contradiction. They are different statistics.
Because p_e depends on the marginals, κ should not be treated as an absolute quality number that transfers cleanly across datasets with different label prevalence. Report at least N, label distribution, p_o, κ, and the confusion matrix.
For more than two raters, ordinal categories, or continuous scores, use an agreement measure appropriate to the design rather than forcing Cohen's κ.
Human agreement does not prove criterion validity¶
Two judges can agree perfectly while applying the wrong rubric.
For example, both humans and an LLM could prefer a fluent, long answer even when the actual requirement is "make no material claim beyond the supplied evidence."
The correct order is therefore:
product construct
→ rubric
→ calibration cases
→ independent human judgment
→ automatic judge
→ agreement + bias analysis
not the reverse.
The MT-Bench/Chatbot Arena paper showed that, under its protocol, strong LLM judges could approximate human preferences and explicitly studied position bias, verbosity bias, and self-enhancement bias.3 That is evidence that the approach can work under concrete setups, not a transferable guarantee for every domain, rubric, or judge model.
Variance: one judgment can hide instability¶
A model-based grader can vary because of sampling, infrastructure, or sensitivity to small prompt/context changes.
For a numeric score s_{i,r} for item i on repetition r, measure:
We do not need infinite repetitions. We do need to know whether a release gate changes because the same judge oscillates around the threshold.
For categorical labels, keep the observed distribution:
Silently collapsing this to pass destroys useful uncertainty information.
Separate judge variance from task disagreement¶
If the same judge changes its answer across repetitions, we have intra-judge instability.
If stable judges disagree with one another, the cause may be different policies, different capabilities, or rubric ambiguity.
Those are different failures and require different fixes.
Position bias: the minimum control is to reverse the order¶
For an A/B comparison, also run B/A.
Define swap consistency:
SC=1 means that swapping presentation order did not change semantic preference on the measured items.
It does not mean the preferences are correct.
The original MT-Bench/Chatbot Arena work identified position bias as one limitation of LLM-as-a-judge.3 Later studies show that its magnitude depends on the judge and task, so a single universal correction factor is not defensible.4
A production gate should fail if a release decision depends on a judge whose preference changes materially under order reversal.
Verbosity and style: control content before calling it bias¶
"The judge prefers long answers" is a hypothesis, not an automatic conclusion.
A longer answer may contain more correct information. Testing style bias requires controlled pairs where relevant content is equivalent or the transformation is explicitly designed.
Useful controls include:
- A/B ↔ B/A ordering
- Markdown ↔ plain text when formatting is not a criterion
- concise ↔ expanded without adding claims
- hidden model identity
- equivalent presentation of references or citations
If content changes, the experiment no longer isolates style.
Self-preference and judge family: cross generators and judges¶
The MT-Bench paper discussed self-enhancement bias: a judge can favor responses that resemble its own distribution.3
Do not assume that using "the strongest model" removes this risk.
A control matrix can cross:
If the ranking shifts systematically when the judge shares a family with one candidate, treat that as a risk signal and increase human review or use an explicitly designed judge panel.
Do not turn one benchmark's observation into a rule about every provider.
A reference can help and can also change the task¶
For problems with an objective answer, giving the judge a reference can reduce reasoning load.
But an overly narrow reference can also penalize valid alternative solutions.
Before introducing one, decide whether the task is:
- reference matching: one concrete solution is expected
- constraint satisfaction: multiple solutions are valid
- preference: quality is compared under a subjective rubric
The grader must match the construct.
Practical calibration: do not use the holdout to write the judge prompt¶
Build a calibration/dev bank where rubric, examples, and grader prompt may be iterated.
Then reserve an independent bank to confirm the judge.
calibration/dev
→ tune rubric + judge prompt
→ inspect disagreement
→ repair ambiguity
judge validation holdout
→ freeze judge version
→ measure agreement / bias / variance
→ accept, restrict or reject scope
If you inspect the holdout after every grader change and edit the prompt again, that bank is now participating in development.
This is the same development-leakage principle from Chapter 5.2, applied to the evaluator itself.
A judge acceptance gate¶
There is no universal κ, agreement, or SC threshold that makes a judge "good" for every system.
Thresholds should come from product error costs.
A reproducible gate can require:
judge_revision fixed
rubric_revision fixed
validation_set fixed and not used for tuning
per-stratum sample counts declared
raw agreement + confusion matrix
agreement statistic appropriate to label type
repeat stability near release boundary
A/B ↔ B/A consistency for pairwise judging
controlled style/length probes
human review of disagreements
known failure strata explicitly excluded or routed to humans
The output need not be binary judge good / judge bad.
It can be:
ACCEPT: automatic judging for low-risk support-quality items
RESTRICT: human fallback for safety/policy disputes
REJECT: not reliable enough for release gating
That scope must travel with the grader version.
Three concrete cases¶
Case A — Grounding in a RAG assistant¶
We want to know whether every material claim is supported by supplied evidence.
Design:
- deterministic checks for citation-ID presence and validity
- isolated LLM judge for claim↔evidence entailment
- calibration bank labeled by experts
unknownwhen evidence is insufficient- validation by strata: simple factual, multi-source synthesis, source conflict
- human review of false
passdecisions because they are the costliest error
We do not mix fluency into the same grounding score.
Case B — Comparing two support responses¶
We want the response that resolves the case better without violating policy.
First validate any machine-checkable policy actions deterministically. Then compare communication quality using both A/B and B/A.
If the judge frequently changes the winner when positions are reversed, silently averaging the two results is not a repair. Investigate the bias or route that region to humans.
Case C — Human review of high-risk answers¶
Two specialists independently rate a sensitive decision.
Store:
- original labels
- rationale or cited evidence
p_oand confusion matrix- κ only when the two-rater nominal design makes it appropriate
- final adjudication separately
If disagreement concentrates in one subgroup, inspect the rubric or context for that stratum rather than hiding it inside the global average.
What to record with every judge-based result¶
At minimum:
judge_type: deterministic | llm | human | hybrid
judge_model_revision: ...
judge_prompt_rubric_revision: ...
judge_sampling_config: ...
human_protocol_revision: ...
calibration_set_version: ...
validation_set_version: ...
item_stratum: ...
raw_judgments: ...
repetitions: ...
position_randomization: ...
agreement_metrics: ...
known_bias_probes: ...
adjudication_status: ...
Without this identity, repeating "the same eval" after changing the judge or rubric can measure a different decision policy.
Production implication¶
An LLM-as-a-judge should not enter a release gate merely because it is cheap, fast, or because an external benchmark says LLM judges can agree well with humans.
It should enter when that concrete judge version:
- implements a rubric that represents product risk
- has been compared with independent human decisions on relevant cases
- is stable enough around the operational threshold
- survives bias probes designed for its judging format
- declares unreliable strata and routes those cases to another mechanism
A robust architecture is often hybrid:
deterministic invariants
+ calibrated model graders for scalable semantic checks
+ targeted human review for ambiguity, high-risk cases, and judge maintenance
The central principle is:
a judge is another component of the measurement system: it needs a version, tests, calibration, limits, and observability.
The next chapter applies this discipline to the hardest evaluation object in the series: a complete agent trajectory with tools, recovery, efficiency, and policy compliance.
References¶
-
Anthropic Engineering, Demystifying evals for AI agents, 9 Jan 2026. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ↩↩
-
OpenAI API Reference, Graders. https://platform.openai.com/docs/api-reference/graders ↩
-
Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track. https://arxiv.org/abs/2306.05685 ↩↩↩
-
Lin Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge, 2024. https://arxiv.org/abs/2406.07791 ↩
-
Jacob Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement 20(1), 1960. https://doi.org/10.1177/001316446002000104 ↩