Speculative decoding and prefix caching
Speculative decoding proposes several cheap tokens and the target model verifies them; prefix caching reuses prefill work when the prefix truly matches.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
Start from the latency budget again
For an autoregressively served request, we can keep the decomposition from chapter 4.1:
Prefix caching reuses state from a previously computed prefix
In an autoregressive Transformer, prefill produces K/V state for input tokens. If a later request begins with exactly the same prefix under the runtime's identity contract, that K/V state…
A hit only covers the state that is actually reusable
Let a request contain Lin input tokens and let Lhit tokens be covered by reusable KV state. Define a token-weighted reuse ratio:
Key moments
Jump directly to a section
Next step
