Precision and distribution
The error is 0.12. Lower precision requires task evaluation, not just memory accounting.
Links containing ?t= open the video at a specific second.
The ideas to retain
Quantization does not mean that the whole model has one precision
Common notation already carries important scope information.
The basic operation introduces representation error
For a simple symmetric quantizer, we can write:
Weight-only and weight-plus-activation quantization target different bottlenecks
Jump directly to a section
Read the reviewed transcript
This video has no narration. This transcript reproduces its on-screen text; it does not invent a spoken track.
Fewer precision levels
Quantisation approximates values with a scale and a limited set of levels.
At scale 0.25, the value 0.62 is represented as 2 and reconstructed as 0.50.
The error is 0.12. Lower precision requires task evaluation, not just memory accounting.
Payload is not total footprint
Eight billion parameters at sixteen bits require sixteen GB of ideal payload.
At four bits, that payload falls to four GB before scales and other costs.
Actual memory includes metadata, buffers and KV. This does not automatically require a GPU one quarter the size.
Three distinct precisions
Weights, activations and the KV cache are different objects.
A weight-only scheme changes weights; it does not automatically convert the other two.
Record every dtype and compatible kernel. A quantisation label does not describe the entire execution.
Splitting a layer communicates
Tensor parallelism distributes parts of a layer operation across devices.
Partial results must communicate and combine according to the chosen partition.
Memory per rank can fall, but the interconnect enters the critical path. Two GPUs do not guarantee twice the speed.
Pipeline stages also wait
Pipeline parallelism partitions layers across stages. A request traverses them in order.
Additional microbatches allow some work to overlap across devices.
Startup and drain bubbles remain. Fewer layers per device do not remove stage dependencies.
Replicas are not experts
Data parallelism distributes requests across complete model replicas.
Expert parallelism distributes MoE experts and routes activations between them.
These placements solve different problems. An expert partition does not exist in every dense model.
