Pretraining: the law everyone quotes governs a number nobody optimizes anymore

Chinchilla's "20 tokens per parameter" was never a law. It was one constant from one fit, of one objective, over a sweep that topped out at 16B parameters — and the regime frontier models train in now has no public curve at all.

Understanding LLMs From the Inside — part 3 of 5.

← Previous: The forward pass: what actually happens to a token, and what it costs · Next: Post-training: where behavior comes from →

The sentence everyone quotes, and the four words in it that got lost

Start with the mechanism, because "scaling law" sounds like physics and is nothing of the kind. It is a curve fit. You train a lot of language models at different parameter counts on different quantities of text, you write down the final loss each one reaches, and you fit a power law to the resulting cloud of points. That's it. The famous version, Kaplan et al. on 23 January 2020, reported that loss "scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."

The word doing the quiet work there is loss. It means cross-entropy on text the model was not trained on: how surprised the model is, on average, by the next token in a held-out sample. Lower is better, it is measured in the same units regardless of scale, and it has nothing directly to do with whether the model can answer a question or write working code. Every scaling law in this literature is a statement about that one number.

Which sets up the allocation problem. A training run's cost is governed mostly by two knobs, how many parameters the network has and how many tokens you push through it, and a lab has a fixed budget of GPU-hours. Spend it on a big model that sees little text, or a smaller model that sees much more? "Compute-optimal" is the answer to exactly that constrained question: for a fixed training compute budget, the parameter/token split that reaches the lowest held-out cross-entropy. Not the best model. The lowest loss per FLOP spent training.

Hoffmann et al., submitted 29 March 2022, answered it by brute force: "training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens." Their fit said the two knobs should move together, in the sentence the field memorized: "for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled."

Then they did the thing that made everyone believe it. They trained Chinchilla at 70B parameters on 4× Gopher's data using the same compute budget as Gopher's 280B, and it "uniformly and significantly" beat Gopher, GPT-3 at 175B, Jurassic-1, and Megatron-Turing NLG at 530B, reaching 67.5% average on MMLU, "greater than a 7% improvement over Gopher," with a quarter of Gopher's parameters and correspondingly cheaper inference. A prediction from a fitted curve, tested prospectively at a scale above the fit, confirmed on downstream benchmarks. That is about as good as empirical evidence gets in this field, and it is why the received account of 2022 is that the industry had been building models too big and feeding them too little.

Now the trouble. The number people quote, 20 tokens per parameter, appears nowhere in that abstract. It is a constant read off the fitted frontier and the confirmation run, and a 2025 robustness study names it accurately as "the influential '20-to-1' heuristic." One point on one curve, remembered as a ratio because ratios are easy to remember.

The sweep also topped out at 16B parameters. The model designed from it was 70B; the models it was used to criticize were 175B and 530B. That is an extrapolation of roughly four to thirty times beyond anything measured, which is ordinary practice for a power law and also the precise property that turns a power law into a liability once the regime changes.

And the four words that fell off in transmission are the first four: for compute-optimal training. Everything the ratio asserts is conditional on caring about loss per unit of training compute. Hold onto that clause. The industry didn't.

The dependent variable was always cross-entropy, and cross-entropy is a compression rate

Kaplan et al., 23 January 2020, open by naming the measured quantity precisely: "We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude."

Cross-entropy on held-out text. Nothing else. Not accuracy, not helpfulness, not whether the thing can do your job.

That quantity has a clean interpretation. The next-token objective from Part 2's forward pass, averaged over a held-out corpus and converted from nats to bits, is the number of bits per token the model needs to encode text it has never seen. A model at 2.0 nats/token encodes that corpus at about 2.9 bits per token. Lower loss means shorter codes, which means the weights have absorbed more of the corpus's statistical structure. Pretraining is lossy compression of a text distribution into 16-bit numbers, and the scaling laws describe how the compression rate improves as you spend. A September 2025 paper makes the reframing explicit in its title, *Scaling Laws are Redundancy Laws*, which is a useful lens rather than a new result.

Two details in Kaplan's abstract are routinely skipped. First: "Other architectural details such as network width or depth have minimal effects within a wide range." The law is indifferent to most of what people mean by architecture. Second: "Simple equations govern the dependence of overfitting on model/dataset size." Overfitting is a term inside the original framework, not a later patch.

Now the break in the chain. GPT-3's *Language Models are Few-Shot Learners*, May 2020, reports downstream few-shot accuracy, a different measurement instrument from the loss curve that justified the run, and it concedes in the same abstract that the authors "identify some datasets where GPT-3's few-shot learning still struggles." The loss went where the law said. Which capabilities moved, and by how much, was measured separately and not predicted.

Chinchilla demonstrated an allocation, not a preference for data

Read Kaplan's conclusion and Hoffmann's side by side, because they share an objective and prescribe opposite things. Kaplan: "optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence." Hoffmann: "current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."

Both are minimizing cross-entropy per unit of training compute. The disagreement is about a fitted exponent, not a philosophy. That is worth sitting with, because it means the 2022 correction was a measurement dispute inside a shared framework, and measurement disputes have error bars. Epoch AI published a replication attempt on the Chinchilla fit on 17 April 2024, re-deriving the parametric estimates from the published figures. I have the fact of that paper but not its numerical verdict in front of me, so I am not going to characterize its conclusion; if you are about to make a budget decision on the strength of 20:1, read it yourself first. The relevant point stands either way: the constant was contested inside its own budget definition, by people who bothered to re-fit it.

What the Chinchilla result does not show is the more important list.

It does not show that data is more valuable than parameters in general. It shows that at a fixed training-compute budget, in the 70M–16B fitting range, the loss-minimizing point sits at a smaller model on more tokens than Kaplan's fit implied. Change the budget definition and the optimum moves.

It does not show that lower loss buys capability in general. It shows one instance where a loss-motivated reallocation moved MMLU by more than seven points at matched compute. One instance is a demonstration, not a transfer function.

It says nothing about serving cost as an objective. The abstract mentions that Chinchilla "uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage" as a consequence, listed after the loss result. In hindsight that sentence reads like the seed of what came next — but in the paper it is a side benefit, not a term being optimized.

And it says nothing about repeated data, because in 2022 nobody at the frontier needed to repeat.

The budget line moved from training FLOPs to served tokens

GPT-3's abstract advertises "175 billion parameters, 10x more than any previous non-sparse language model." That was the scaling axis of the compute-bound era: parameters, because data was assumed abundant. The Pile, released 31 December 2020, is the artifact of that assumption, 825 GiB across 22 subsets, presented as a large-scale pretraining corpus, which it was. How many tokens that is depends on the byte-per-token asymmetries Part 1 covered, which is itself part of why tokens-per-parameter is a slippery quantity to compare across labs.

Now do the arithmetic on something shipped. The DeepSeek-V3 technical report, December 2024, states its training set exactly: 14.8 trillion tokens, for a mixture-of-experts model with 671B total parameters and 37B activated per token. Against the active count, that works out to roughly 400 tokens per active parameter. Chinchilla's heuristic was 20. So: a factor of twenty past the compute-optimal ratio, chosen on purpose, in a run the authors report cost 2.788M H800 GPU hours. Use total parameters as the denominator instead and the ratio comes out near 22, sitting almost on top of Hoffmann et al., which tells you the MoE denominator is a live dispute rather than a settled measurement.

Then the dispute stops mattering. Meta's Llama 4 model card, for models released 5 April 2025, is the most recent frontier release whose pretraining token count this piece could verify from a primary source, and it prints the numbers in the summary table. Llama 4 Scout: 17B activated, 109B total, ~40 trillion pretraining tokens, 10M context window, August 2024 data cutoff, 5.0M H100-80GB GPU hours. Per active parameter, about 2,350 tokens. Per total parameter, still about 370. Both are past 20 by more than an order of magnitude, and no choice of denominator brings Scout anywhere near the curve Hoffmann et al. fitted. Maverick, the same 17B active against 400B total on ~22T tokens, lands near 1,300 and 55. With DeepSeek-V3 you could argue that picking a denominator decided whether the model was compute-optimal. With Scout the argument is moot on both counts.

Compare what a lab chooses to advertise. Llama 3's abstract, July 2024, leads with 405B parameters, a 128K context window, and quality the authors put alongside GPT-4 across a wide range of tasks. It gives no training token figure; that number lives in the paper body. Nine months later the token count is a column in Meta's own table, and the parameter count has split into two.

The reason the ratio moved is an accounting difference, not a scientific one. Training compute is paid once. Inference compute is paid on every served token, forever, and DeepSeek's own abstract puts efficient inference and cost-effective training first among its design goals, ahead of any capability claim. If you expect volume, spending extra tokens now to serve 17B or 37B active parameters later is straightforwardly rational, even though it buys worse loss per training FLOP than the frontier prescribes. One 2026 explainer states the operating logic without ceremony: scaling decisions are now about "making the best model that you can afford to serve."

The literature has a name for it. A March 2025 paper on distributed training notes public text exhaustion projected for 2028, "potentially accelerated to 2026 through excessive data reuse... (a practice termed overtraining)." Overtraining there means exactly training past the compute-optimal point, and by 2025 it is common enough to be listed in passing as a cause of corpus exhaustion.

So the reading instruction is: when a lab reports training tokens, divide by parameters, do it twice for a sparse model, and compare both to 20. A 370 or a 2,350 tells you the model was positioned against someone's inference bill, and that Chinchilla arithmetic says nothing about where it sits on any published curve.

The cost is data. The 2028 figure is Villalobos et al.'s median, with exhaustion very likely by 2032 and models by then using around 5e28 FLOP in training. Nathan Lambert's August 2024 dissent narrows it usefully: we are running out of open training data, a different and less tractable problem than running out of text.

For sparse models the denominator stays contested even so. The LifeArchitect Chinchilla page, updated around June 2026, notes that compute-optimal tracking of Hoffmann holds for MoE roughly in the 1B–10B active-parameter band, with higher granularity optimal at larger compute budgets. Scout's 17B active and DeepSeek-V3's 37B both sit outside it, which is why none of these ratios is a measurement of anything Hoffmann et al. actually fitted.

The strongest objection: overtraining is a scaling-law decision, not a departure from one

Here is the case against everything above, and it is good.

The framework was not abandoned, it was extended. Add an inference-cost term to the Chinchilla objective — total FLOPs over the model's serving lifetime, not just its training run — and the loss-minimizing allocation moves toward a smaller model on far more tokens, which is what shipped. On that reading, training far past 20:1 is a law being obeyed under a different budget rather than a law being ignored. Extending the objective is in-tradition: Kaplan's original framework already carried an overfitting term. I don't have a verifiable citation for the formalization in front of me, so take the mechanism and not anyone's coefficients.

The second half of the objection is that I am underselling loss-to-capability transfer. Chinchilla is the counterexample to my own scepticism. A reallocation motivated purely by a loss fit moved MMLU to 67.5% at matched compute with 4× fewer parameters. That is weak evidence for a general transfer function, but it is evidence, and "the bridge from loss to capability is unmeasured" would be too strong a claim.

So I'll narrow. The method survived. The published ratios did not, and the regime everyone now trains in has no public fit I can point to. Whatever the extended objective's coefficients are for frontier runs, they aren't published; what is public is a 2022 fit for a budget definition that, if the serving-cost framing is right, few frontier runs now optimize.

The 2026 literature adds a wrinkle I want to flag carefully. Lilian Weng's 24 June 2026 post reports that Lovelace et al. (2026) find excess parameters decay in value faster than repeated data, so under a data constraint you should buy epochs rather than parameters. If that holds it is a genuine inversion, and it would have been the wrong call under the 2022 fit. I could not resolve the underlying paper to a retrievable arXiv entry as of 26 July 2026, so treat it as a pointer to check rather than a result to build on. The same caution applies to anyone quoting a 2026 scaling number at you without a resolvable identifier.

What has no public curve, as of July 2026 — and the ablation that would settle it

Take inventory. There is a published fit for dense transformers at compute-optimal, from 2020 and 2022. There is a published fit for repeated data: Muennighoff et al. found that up to about four epochs of repetition is roughly as good as unique data, with no benefit beyond. There is a published fit for numerical precision, in Kumar et al.'s Scaling Laws for Precision at ICLR 2025.

There is no public fit for the regime frontier models actually occupy: far past compute-optimal, on partly repeated data, with sparse activation and two parameter counts. The absence may be proprietary rather than real. From outside, it is an absence.

What is public is a claim about the method, not the ratio. The International AI Safety Report 2026 technical supplement, from February 2026, states that scaling relationships have held across six orders of magnitude of model size, up to frontier systems costing hundreds of millions of dollars. Note what that asserts: the relationships hold. It says nothing about the allocation. Meanwhile a January 2026 paper on multi-hop reasoning restates the awkward part — different capabilities scale differently, so a single loss curve underdetermines which eval moves.

The falsifying ablation for this piece is straightforward. Match two models on held-out cross-entropy over a shared validation set while varying corpus composition. If downstream scores land in the same place, loss is sufficient and I am wrong. The Pile remains a good fixed corpus for the controlled repeat-epoch version of this, and the shape of the loss-versus-compute curve is reproducible on a ladder of 100M-parameter models by anyone with a few GPUs.

The test worth running this week is smaller. Score a candidate model's per-token loss on a held-out corpus drawn from your own product that cannot be in any crawl, then on a public benchmark set for the same task. A large gap favouring the public set is your contamination-and-distribution-shift signal, and it takes an afternoon with any eval harness. The Pile paper demonstrated the general failure in December 2020: GPT-2 and GPT-3 "struggle on many of its components, such as academic writing," while posting a perfectly respectable aggregate.

Pretraining sets what the weights compress. What the model then does with that compression is a separate training stage with its own objective and its own fitted curves — a survey page updated this week now catalogues power-law fits for reinforcement learning post-training as well. Part 4 takes that up.

---

The series

1. Tokens: the alphabet an LLM actually reads
2. The forward pass: what actually happens to a token, and what it costs
3. Pretraining: the law everyone quotes governs a number nobody optimizes anymore (you are here)
4. Post-training: where behavior comes from
5. Making it yours: distillation and fine-tuning a domain model