How to evaluate the evaluator
Freeze the judge version before testing it on a separate set. Accept, restrict or reject its use according to risk and evidence.
Links containing ?t= open the video at a specific second.
The ideas to retain
Define what "calibration" means first
In this chapter, judge calibration means an operational process: validate one concrete grader version against independent decisions and controlled cases before scaling it.
Choose the narrowest grader that answers the question
Do not use an LLM to judge a property that can be checked exactly.
Deterministic
Use deterministic checks when there is a verifiable invariant:
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — Choose the verifier the task needs.
When code can verify the result, start with a deterministic check.
Use a language-model judge for criteria requiring interpretation.
Escalate to people when risk or ambiguity requires it.
00:22 — A concrete rubric. The same evidence.
“Sounds good” does not define a correct answer.
Separate the claim, supporting evidence and rejection conditions.
Collect independent human judgments before adjudicating disagreements.
Illustrative example, not a benchmark result.
00:44 — Measure agreement. Do not call it truth.
In this example, human and model judgments agree on 16 of 20 cases.
That is 80% observed agreement, not 80% system quality.
They may also agree while applying the wrong criterion.
Illustrative example, not a benchmark result.
01:08 — Reverse the order. Keep the content.
Present the same response pair in both orders.
Check whether preference follows the answer or its position.
A preference reversal calls for investigation; it does not prove the cause alone.
Illustrative example, not a benchmark result.
01:30 — A single judgment can be unstable.
Repeat judgments while holding the case and rubric fixed.
Separate evaluator variation from disagreement about the task itself.
Record abstentions and differences by case type, not just an average.
Illustrative example, not a benchmark result.
01:52 — Calibrate on dev. Test separately.
Refine the rubric using calibration examples and human adjudication.
Freeze the judge version before testing it on a separate set.
Accept, restrict or reject its use according to risk and evidence.
