Prefix caching vs speculative decoding
The continuation must still be generated. A prefix hit does not retrieve a complete answer.
Links containing ?t= open the video at a specific second.
The ideas to retain
Start from the latency budget again
For an autoregressively served request, we can keep the decomposition from chapter 4.1:
Prefix caching reuses state from a previously computed prefix
In an autoregressive Transformer, prefill produces K/V state for input tokens. If a later request begins with exactly the same prefix under the runtime's identity contract, that K/V state…
A hit only covers the state that is actually reusable
Let a request contain Lin input tokens and let Lhit tokens be covered by reusable KV state. Define a token-weighted reuse ratio:
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
Reuse only the prefix
Two requests share four initial tokens but have different suffixes.
The second prefill can reuse those four tokens’ state if the cache is compatible.
The continuation must still be generated. A prefix hit does not retrieve a complete answer.
Identity determines the hit
Text can look identical while token IDs or the model revision differ.
The key must reflect actual dependencies and required access isolation.
An incomplete match does not authorise KV reuse. Correctness comes before hit rate.
A proposal is not a commit
The proposer drafts three candidate tokens: A, B and C.
The target accepts A and B, rejects C and produces D through the correct procedure.
The committed sequence is A, B, D. Unverified candidates must not become final output.
High acceptance is not enough
Three ordinary tokens cost twelve time units in this example.
Proposing and verifying costs nine in one case but eighteen in another with the same committed output.
Only the first saves time. Acceptance rate alone omits proposal and verification costs.
Retire the rejected branch
Speculative candidates can also reserve provisional KV state.
When C is rejected, its state cannot remain part of the committed prefix.
Rollback preserves A and B while discarding the invalid branch. Measure the cost of temporary state too.
Interleave the work
A large prefill can occupy an interval while other requests wait for decode.
Splitting it into chunks lets the scheduler interleave generation steps.
The policy changes interference and efficiency. Measure both; chunking does not necessarily accelerate every workload.
