Measure the complete system
These are different experiments: the second can reduce arrivals precisely when latency increases.
Links containing ?t= open the video at a specific second.
The ideas to retain
A benchmark is a protocol, not a scalar
An interpretable result needs a contract that fixes at least:
The workload should resemble the problem you are trying to solve
An average of 512 input tokens and 128 output tokens is not a distribution.
Request rate and concurrency describe different load models
A benchmark configured only with concurrency=64 commonly behaves like a closed loop: when one request completes, another fills the slot. A benchmark driven by a fixed or stochastic arrival…
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
Arrivals or concurrency
Under open-loop load, requests arrive at a fixed rate even when the server slows down.
Under closed-loop load, each client waits for a response before sending another request.
These are different experiments: the second can reduce arrivals precisely when latency increases.
Divide by successful tasks
A run costs twenty-four euros and makes one thousand attempts.
Nine hundred and twenty pass the success criterion fixed in advance.
Cost per attempt is €0.024; per successful task, about €0.0261. Failures also cost money.
Energy accumulates power
An illustrative constant six hundred watts over three hours consumes 1.8 kWh.
Dividing that energy by 920 successful tasks gives about 1.96 Wh per task.
This is not a measurement: power and outcomes must refer to the same window and system.
Draw what you measure
GPU telemetry does not necessarily include CPU, RAM, networking and system losses.
A wall meter observes a different boundary from the accelerator sensor.
Comparing them without a boundary mixes quantities. Rated power is not measured consumption either.
Queuing changes the regime
In this illustrative M/M/1 model, the server processes five requests per second on average.
As arrivals approach capacity, mean response time grows as one divided by five minus lambda.
At four arrivals per second, it is one second. A low-load test does not describe saturation.
Preserve the variation
Three equivalent repetitions take 1.0, 1.2 and 1.1 seconds in this example.
Publishing only the fastest removes information about variation between runs.
Fix warmup, cache state and protocol. Compare distributions and retain errors, not just the best scalar.
