Evaluation and observability
Join business, turn, media and runtime evidence. A transcript does not prove the complete outcome.
Links containing ?t= open the video at a specific second.
The ideas to retain
A transcript is not the system's ground truth
A transcript is valuable evidence, but it only describes part of the path. Consider this turn:
The useful unit of analysis is the logical turn
A full session is too coarse for many decisions. A provider request is too narrow. A useful unit is usually a logical turn: from evidence that the user is trying to communicate something…
Observed, inferred, and judged are different kinds of evidence
Record where a signal came from:
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
The sentence is not all the evidence
The agent says “booked”. A text judge may consider that response correct.
But op_42 is still UNKNOWN, and the audio may remain in a cancelled queue.
Join business, turn, media and runtime evidence. A transcript does not prove the complete outcome.
One turn can have several attempts
T12 keeps its identity when one request fails and fallback uses another.
Attempts, operation and audio stay correlated without becoming new turns.
Separate observation, inference and judgement. A probable diagnosis is not a recorded fact.
The denominator changes the meaning
Of ten synthetic turns, three have an internal failure. Two recover without visible harm.
The internal rate is 3 out of 10; the visible-failure rate is 1 out of 10.
Recovery is 2 out of 3 internal failures. Do not mix requests with turns.
Deterministic checks do not need a judge
Require one effect per operation, no stale audio and evidence before declaring SUCCESS.
When an invariant fails, a high semantic score must not compensate for it.
Reserve judges for semantic or perceptual questions, keeping their criteria and version.
Replay the boundary that failed
Text replay keeps the words, but loses the exact timing of the interruption.
This failure needs audio or timed events and the playback queue’s state.
A local audio test does not certify the real network either. State which layers are excluded.
From incident to a useful regression
Isolate the smallest sequence that reproduces stale audio; do not copy the entire conversation.
Redact sensitive data and pin the input, version and expected outcome.
The test should fail before the mechanism is fixed and pass afterwards, at the right layer.
