--- title: LLM-as-judge, human evaluation, and calibration seo_title: LLM-as-judge, human evaluation, and calibration — video description: A judge is useful only when its criterion is calibrated against human evidence, disagreement is measured, and bias, variance, and stability are separated before using it as an acceptance signal. keywords: LLM-as-judge, human evaluation, grader calibration, inter-rater agreement, Cohen kappa, position bias, evaluator variance, AI evaluation date: '2026-09-12T00:00:00+00:00' robots: index,follow,max-snippet:-1,max-image-preview:large,max-video-preview:-1 hide: - toc - navigation video_watch_page: true ---
All videosOther topics0:36

LLM-as-judge, human evaluation, and calibration

A judge is useful only when its criterion is calibrated against human evidence, disagreement is measured, and bias, variance, and stability are separated before using it as an acceptance signal.

Links containing ?t= open the video at a specific second.

Video summary

The ideas to retain

01

Define what "calibration" means first

In this chapter, judge calibration means an operational process: validate one concrete grader version against independent decisions and controlled cases before scaling it.

02

Choose the narrowest grader that answers the question

Do not use an LLM to judge a property that can be checked exactly.

03

Deterministic

Use deterministic checks when there is a verifiable invariant:

Key moments

Jump directly to a section

  1. Define the rubric before asking for a verdict
  2. Compare judge and humans on the same cases
  3. Calibrate bias, variance, and disagreement before the gate
Next step