Evidence before expanding
A violation blocks; insufficient data keeps the rollout paused. Choose stages for the change and its risk; not every release needs them all.
Links containing ?t= open the video at a specific second.
The ideas to retain
Before exposing traffic, define the unit of change
A “candidate” should not simply mean “the new model.”
Shadow: observe the candidate before giving it authority over the response
Shadow traffic, also called mirroring, copies real requests to a candidate while the stable variant continues to serve the user.
Shadowing does not measure real user impact
The candidate response does not determine what the user sees. A shadow therefore cannot directly observe:
Jump directly to a section
Read the reviewed transcript
Visual-text transcript. This video has no narration.
00:00 — Three methods. Three different questions.
Shadow observes the candidate without serving its answer to the user.
Canary limits real exposure while checking operational risk.
A/B uses randomized assignment to study causal effects.
00:22 — Not serving the answer does not remove effects.
The stable version responds; the candidate receives a copy for observation.
Candidate writes must be isolated, suppressed or executed in a safe environment.
Shadow does not directly measure a changed user experience.
Illustrative example, not a benchmark result.
00:46 — Expose a little. Define when to stop.
This 5% is an illustrative starting exposure, not a universal threshold.
Predeclare limits for errors, latency and prohibited actions.
Insufficient evidence requires a pause; it does not mean PASS.
Illustrative example, not a benchmark result.
01:08 — Randomize the unit that preserves context.
A conversation with memory may need stable assignment by user.
Switching variants every turn mixes treatments and contexts.
Record assignment, exposure and the unit of analysis.
Illustrative example, not a benchmark result.
01:30 — Before the effect, validate measurement.
Check assignment, exposure, logging and metric integrity.
Investigate imbalances inconsistent with the design before reading lift.
An observed canary difference is not automatically causal.
Illustrative example, not a benchmark result.
01:52 — Expansion requires sufficient evidence.
Combine hard constraints, reliability and evidence for the current stage.
A violation blocks; insufficient data keeps the rollout paused.
Choose stages for the change and its risk; not every release needs them all.
