From incident to repair
After release, measure recurrence and exposure over the agreed observation window. Close the loop with production evidence and an accountable owner.
Links containing ?t= open the video at a specific second.
The ideas to retain
Observability and evaluation answer different questions
Observability tries to reconstruct what happened in a running system.
Preserve a reproducible system identity first
A failure that cannot be tied to what was actually deployed is difficult to explain.
A useful trace preserves causality, not every byte
More data is not automatically better evidence.
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — An alert is not a diagnosis.
Telemetry shows that something happened, not necessarily why.
Reconstruct the sequence with versions, states and relevant responses.
Separate the observed symptom from an unproven cause.
Illustrative example, not a benchmark result.
00:22 — The same answer. Two writes.
The user received “refund complete”, but the backend records a duplicate.
The trace reveals a write, an ambiguous timeout and a retry without reconciliation.
The causal hypothesis concerns recovery, not answer style.
Illustrative example, not a benchmark result.
00:46 — One mechanism. Its neighboring cases too.
Group incidents that share the same failure mechanism.
Add a timeout before the write and a read-only operation.
The repair must fix the failure without breaking neighboring cases.
Illustrative example, not a benchmark result.
01:08 — Turn the incident into a reproducible test.
Remove sensitive data while preserving the conditions behind the failure.
Pin initial state, tool responses, versions and the verifier.
When evidence is insufficient, record the limitation; do not invent a cause.
01:30 — Test the repair. Not just the symptom.
Reproduce the failure on the old version before attributing improvement.
Change recovery, then rerun the failure case and neighboring cases.
Check that a grader change did not merely hide the failure.
Illustrative example, not a benchmark result.
01:52 — Green tests are not production verification.
Passing regression allows progress; it does not close the incident by itself.
After release, measure recurrence and exposure over the agreed observation window.
Close the loop with production evidence and an accountable owner.
