Evaluation, observability and reliability
The useful unit is a traceable turn: correlated evidence, taxonomy, and a production→eval→repair loop.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
A transcript is not the system's ground truth
A transcript is valuable evidence, but it only describes part of the path. Consider this turn:
The useful unit of analysis is the logical turn
A full session is too coarse for many decisions. A provider request is too narrow. A useful unit is usually a logical turn: from evidence that the user is trying to communicate something…
Observed, inferred, and judged are different kinds of evidence
Record where a signal came from:
Key moments
Jump directly to a section
Next step
