Latency, Streaming and Product Design
TTFT, streaming and perceived-latency thresholds in reasoning models. RouteLLM, design patterns and production session-cost management.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
1. Perceived-latency thresholds
User-experience research has studied for decades how latency affects perception. Jakob Nielsen established a scale in the 1990s that remains relevant:
Dynamic routing: RouteLLM
A relevant pattern in this context is dynamic routing, formulated as a technique under the name RouteLLM (Ong et al., 2024): instead of applying the most capable (and slowest) model to…
2. Streaming and the perception of latency
Streaming—sending tokens to the client as they are generated instead of waiting for the complete response—is the most widely used tool for improving perceived latency without reducing…
Next step
