Tools · Scaling · 15

Reallocate the same training compute between parameters and data.

Start from a model and training-token count, keep or scale the total budget, and inspect which allocation minimizes the loss predicted by the Hoffmann et al. fitted surface. Change the exponents to see how sensitive that conclusion is to the fit.

ConstraintC ≈ 6ND

Objectiveminimize loss

ReferenceChinchilla 2022

Limitpretraining ≠ full system

Current scenario

Parameters billionsDense model parameter count.

Training tokens billionsTokens seen during pretraining.

Target budget × current budget1× preserves total FLOPs; 10× multiplies the N·D product by ten.

Fit sensitivity

Parameter exponent α

Data exponent β

E, A and B remain fixed at the Hoffmann et al. published fit. Changing α or β is a pedagogical counterfactual, not a new empirical fit.

Copy scenarioExport JSONReset

Fixed budget

What changes when N and D compete for the same compute

—

Current compute—dense 6ND approximation

Target compute—after the multiplier

Optimal parameters—for this fitted surface

Optimal tokens—for this fitted surface

Allocation

Loss with the same N:D ratio: —

Loss at the optimum: —

Difference attributable to allocation inside this model: —

Tokens per parameter: — now · — at the optimum.

How the optimum scales

N_opt ∝ C^—

D_opt ∝ C^—

—

Method

The optimum is mathematical inside one empirical loss surface.

Two bottlenecks in the loss

The tool uses L(N,D)=E+A/N^α+B/D^β. More parameters reduce one term; more data reduce the other.

Compute as a product constraint

The approximation C≈6ND turns a fixed budget into a trade-off: if N grows, D must fall to keep the same training cost.

Visible derivation

Under that constraint, N_opt scales as C^β/(α+β) and D_opt as C^α/(α+β). Editable exponents expose how sensitive that conclusion is.

What it does not mean

This curve does not predict the performance of a complete modern model.

The original fit describes pretraining loss for dense Transformers in a specific experimental regime. It does not include post-training, MoE routing, data quality, architectural changes, inference, RL, tools or serving cost. The coefficients shown here are the rounded values printed in the paper, so an analytical optimum computed from them does not have to reproduce every table or fitted frontier exactly. Use it to understand the parameter–data trade-off and conditional compute optimum, not to quote a real training run.

Hoffmann et al. · 2022
Kaplan et al. · 2020
Constants: E=1.69 · A=406.4 · B=410.7 · α=0.34 · β=0.28
Sources reviewed: 2026-08-22