Benchmarking: cost, throughput, latency, and energy
An inference benchmark is comparable only when workload, measurement boundary, and hardware are fixed, and latency distribution, useful throughput, cost, and energy are reported.
Links containing ?t= open the video at a specific second.
Video summary
The ideas to retain
A benchmark is a protocol, not a scalar
An interpretable result needs a contract that fixes at least:
The workload should resemble the problem you are trying to solve
An average of 512 input tokens and 128 output tokens is not a distribution.
Request rate and concurrency describe different load models
A benchmark configured only with concurrency=64 commonly behaves like a closed loop: when one request completes, another fills the slot. A benchmark driven by a fixed or stochastic arrival…
Key moments
Jump directly to a section
Next step
