LLM-as-judge, human evaluation, and calibration
A judge is useful only when its criterion is calibrated against human evidence, disagreement is measured, and bias, variance, and stability are separated before using it as an acceptance signal.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
Define what "calibration" means first
In this chapter, judge calibration means an operational process: validate one concrete grader version against independent decisions and controlled cases before scaling it.
Choose the narrowest grader that answers the question
Do not use an LLM to judge a property that can be checked exactly.
Deterministic
Use deterministic checks when there is a verifiable invariant:
Key moments
Jump directly to a section
Next step
