Online evaluation: shadow, canary, A/B, and regression gates
Online evaluation increases exposure under control: shadow observes without deciding, canary limits blast radius, A/B estimates effect, and guardrails or regression gates stop an unsafe promotion.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
Before exposing traffic, define the unit of change
A “candidate” should not simply mean “the new model.”
Shadow: observe the candidate before giving it authority over the response
Shadow traffic, also called mirroring, copies real requests to a candidate while the stable variant continues to serve the user.
Shadowing does not measure real user impact
The candidate response does not determine what the user sees. A shadow therefore cannot directly observe:
Key moments
Jump directly to a section
Next step
