Post-training: where behavior comes from

RLVR looks like a new training paradigm; mechanically it's the same three-stage post-training recipe as 2022's InstructGPT, with a programmatic checker standing in for the reward model, and the reasoning it produces is paid for in inference-time tokens, not summoned from scale.

Understanding LLMs From the Inside — part 4 of 5.

← Previous: Pretraining: the law everyone quotes governs a number nobody optimizes anymore · Next: Making it yours: distillation and fine-tuning a domain model →

The graph that everyone points to: response length climbing across RL steps

A freshly pretrained model is not a product. It continues text. Ask it a question and a plausible completion is another question, or a list of similar questions, because that is what the training distribution rewarded. Everything you recognize as assistant behavior — answering rather than continuing, refusing, formatting, admitting uncertainty — is installed afterward, in a phase the field calls post-training, sitting on top of the pretraining and scaling story from Part 3.

The canonical recipe was written down by Ouyang and colleagues at OpenAI in the InstructGPT paper, arXiv:2203.02155, posted 4 March 2022. Three stages. Collect demonstrations from human labelers of what a good response looks like and fine-tune on them, supervised. Collect human rankings of model outputs and fit a reward model to those comparisons. Then optimize the policy against that reward model with reinforcement learning. The headline result is the one worth memorizing: on their own prompt distribution, human evaluators preferred outputs from the 1.3B-parameter InstructGPT to those from 175B GPT-3, a model with a hundred times the parameters. Behavior was cheaper to buy through post-training than through scale.

Now the received account of reasoning models, which is where DeepSeek's R1 paper comes in, first posted to arXiv on 22 January 2025 as arXiv:2501.12948. It describes a variant called R1-Zero: reinforcement learning applied directly to a base model, with no supervised fine-tuning stage first. No demonstration data anchors the policy before the reward signal starts pushing it around. And over the course of RL training, average response length climbs, steadily and by a wide margin, as the model produces longer chains of reasoning before committing to an answer. Nobody told it to do that. Nothing in the reward function mentions length.

The paper's most-quoted moment is a passage where, mid-generation on a math problem, the trace pauses, reconsiders an earlier step, and catches its own mistake before continuing. The authors call it an "aha" moment and present it as emergent self-verification, arising from optimization pressure rather than copied from any labeled example. Their framing in the abstract is that reasoning can be incentivized through pure RL, obviating the need for human-labeled reasoning trajectories, with the resulting model surpassing counterparts trained by conventional supervised learning on human demonstrations.

Take that claim seriously, because the narrow version of it is true and important: nobody wrote those reasoning traces. The work carries the weight the field gives it. It appeared in Nature (volume 645, pages 633–638, 2025), and the arXiv listing picked up a version-2 revision on 4 January 2026, most likely to align the preprint with the peer-reviewed text. The reference document for the entire reasoning-model category got a second pass a year after the original.

So the question is not whether the graph is real. It is what the graph is a graph of.

Same three stages as 2022, one part swapped

Pretraining and supervised fine-tuning minimise the same objective: cross-entropy on the next token. That is why the distinction confuses people. The difference is entirely in what you feed the loss and which tokens you let it score. Pretraining runs over scraped text where every token is a target, and the model ends up with a distribution over plausible continuations of arbitrary documents (part 3 covers how that scales). SFT runs over prompt-and-response pairs written by human labelers, and the loss is masked over the prompt so gradients flow only from the response tokens. You are no longer teaching the model what text looks like. You are teaching it what a reply looks like, conditioned on a request.

Nothing new enters the weights in the sense of facts. What changes is the shape of the conditional distribution: after SFT, given a question, the logits that used to spread mass across "and another thing worth asking is…" and "Posted by u/throwaway" now concentrate on a direct answer in the register of the demonstrations. This is why Ouyang et al.'s InstructGPT paper, published by OpenAI in March 2022 (arXiv:2203.02155), could report that human raters preferred the outputs of a 1.3B-parameter InstructGPT to those of 175B GPT-3, a model with a hundred times the parameters. The smaller model did not know more. It had been taught which of the things it already knew how to say were the ones a user wanted.

Stage one gets you format and compliance. It cannot get you better, because imitation is capped by the demonstrations: the model can only learn to produce responses as good as the ones the labelers wrote. To improve past that you need a signal that can rank two model-written responses, neither of which appeared in the training set. Humans can do that — ranking is much easier and more reliable than assigning absolute scores — but they cannot do it for the millions of samples an RL run consumes. So you train a stand-in.

A reward model is, mechanically, unremarkable. Take the SFT model, replace the token-prediction head with a head that emits a single scalar, feed it a prompt plus one completion, and read off a number. You train it on preference pairs: for each prompt, a completion the labelers ranked higher and one they ranked lower. The loss pushes the scalar for the preferred completion above the scalar for the rejected one, under the Bradley-Terry model of pairwise preference. It is a learned, differentiable, arbitrarily-queryable approximation of "would a labeler pick this one." Nothing more. Its numbers are meaningless in absolute terms and only weakly meaningful in comparison.

Then stage three, which is the part that actually looks like reinforcement learning. The loop, in the terms a programmer would write it: pull a prompt from the dataset, sample one or more completions from the current policy, score each with the reward model, and compute a gradient step that raises the log-probability of the tokens in the high-scoring completions and lowers it for the low-scoring ones. InstructGPT used PPO, which clips how far the updated policy's token probabilities may move from the sampling policy's on any one step, so a single freakishly-rewarded sample cannot yank the model somewhere strange. That is the whole update. Sample, score, reweight, repeat.

Left alone, this diverges. The reward model is a finite approximation, and a policy optimising against an approximation will find the places where the approximation is wrong: outputs that score 8.4 and read like broken liturgy. The standard defence is a KL penalty. You keep a frozen copy of the SFT model as a reference, measure the per-token KL divergence between policy and reference, and subtract it from the reward with some coefficient. High coefficient, the model barely moves and barely improves. Low coefficient, reward goes up and quality falls off a cliff. The DPO authors describe this same tension in their framing of RLHF as maximising estimated reward "without drifting too far from the original model" (arXiv:2305.18290, May 2023); their contribution was to reparameterise the reward so the optimal policy has a closed form, collapsing the whole loop into one classification loss over preference pairs with no sampling from the model during fine-tuning at all.

That slot in the pipeline turns out to be swappable in more than one direction. Anthropic's Constitutional AI paper (arXiv:2212.08073, December 2022) kept the RL stage and replaced the human labels with a model judging its own two samples against a written list of principles, training the preference model on AI comparisons instead of human ones. Same machinery, different label source.

RLVR fills the same slot with something that isn't a model. DeepSeek's R1 paper (arXiv:2501.12948, submitted 22 January 2025, later published in Nature 645:633–638) describes R1-Zero as RL applied directly to a base model with no supervised cold start, and the reward is a function you could write in fifty lines of Python: extract the final answer, compare it to the reference; run the generated code against unit tests, count passes; check the output has the required structure. Zero or one, deterministic, no gradient of its own. DeepSeek used GRPO, which samples a group of completions per prompt and normalises reward within the group to get advantages, so there is no separate value network to train either. Sample a group, grade with a script, reweight.

A checker can replace a learned reward model exactly where correctness is decidable and nowhere else. For a competition maths problem the checker is the ground truth, so there is nothing to hack: the policy cannot fool ==. For "is this refusal appropriately worded" there is no checker, which is why the preference-model pipeline hasn't gone anywhere for tone and safety behaviour. And the discipline this buys comes with the corresponding blindness. A programmatic grader on verifiable domains says nothing about whether the resulting behaviour holds under distribution shift into domains where no grader exists.

If you want the ablation that tests all of this, it is the one R1-Zero already half-runs: strip the imitation stage and the long chain-of-thought still appears, which tells you the behaviour comes from the optimisation pressure and not the demonstrations. The complementary experiment, replacing the checker with a random reward and seeing whether the reasoning traces survive, is the one to look for in anyone's paper before believing the mechanism story. RLVR is stricter grading. It is not a different kind of training.

Four papers, one lineage of reward-signal swaps

Once you see reward-signal-swap as the operative move, RLVR stops looking like an event and starts looking like the fourth entry in a table anyone could have drawn up in advance. InstructGPT (2022) established the baseline (arXiv:2203.02155): a reward model fit to human preference rankings, then policy optimization against it. Rafailov et al.'s Direct Preference Optimization, posted in May 2023 and last revised July 2024 (arXiv:2305.18290), keeps the same preference data but collapses the reward model into the policy itself through a closed-form reparameterization, turning the RL loop into a classification loss. The reward signal is still human preferences; what moved is where the arithmetic happens.

Anthropic's Constitutional AI paper, from December 2022 (arXiv:2212.08073), swaps the source of the preference labels rather than the mechanism: an AI model generates the preference comparisons instead of a human labeler, producing what the paper calls RL from AI feedback. The RL-against-a-reward-signal structure is untouched. Only the origin of the signal changed, from a person to a model trained to imitate a person's judgment.

RLVR is the next line in that same table: replace the learned or AI-generated reward model with a deterministic checker, a unit test suite or an answer-matching function that returns a scalar with no learned component at all. Four papers, four swaps to what generates the number the policy gets pushed toward, and the underlying optimization loop, sample a completion, score it, adjust the policy to make high-scoring completions more likely, hasn't changed shape since 2022.

In the open-weight world that swap has a specific algorithm attached to it, and it predates the reasoning-model discourse by a year. DeepSeekMath, February 2024 (arXiv:2402.03300), continued pretraining DeepSeek-Coder-Base-v1.5 7B on 120B math-related tokens and introduced Group Relative Policy Optimization, which the paper presents as a PPO variant that improves mathematical reasoning while reducing PPO's memory usage. The reduction comes from what GRPO throws away: the separate learned value network PPO carries as a critic. In its place you sample a group of completions for the same prompt, score each one, and use the group's own scores as the baseline each completion's advantage is measured against. The policy is pushed toward whichever samples did better than their siblings. DeepSeekMath 7B reported 51.7% on the competition-level MATH benchmark with no external toolkits or voting, which the authors describe as approaching Gemini-Ultra and GPT-4, and 60.9% with self-consistency over 64 samples. Note which of those two numbers came from training and which came from a runtime budget: 64 samples bought 9.2 points at no change to the weights. That algorithm is now the default rather than one lab's preference: a March 2026 analysis recasting GRPO's policy gradient as a U-statistic (arXiv:2603.01162) opens by calling it a cornerstone for scaling reasoning and derives a universal scaling law for choosing the group size, which is the sort of paper you only write about a method everyone is already running.

What that does not establish is that the group-relative baseline is what produces the long chain-of-thought behavior. DeepSeekMath measured GRPO on a 7B math-specialized model on MATH, and DeepSeek's R1 paper (arXiv:2501.12948, revised January 2026, published in Nature 645:633–638) reports the emergent self-verification and reflection patterns without, in its abstract, presenting a PPO-versus-GRPO comparison isolating them. That's the ablation I'd want: rerun R1-Zero-style training with a PPO critic and identical verifiable rewards, and check whether reasoning length still grows. If it does, GRPO is a memory-efficiency result and not an elicitation mechanism, and the credit belongs entirely to the checker. The theory leans that way already: the same U-statistic analysis finds GRPO asymptotically equivalent to an oracle policy gradient with access to a value function, which is a statement about asymptotic optimality rather than the empirical ablation, but it makes "the group baseline is what elicits reasoning" the harder position to hold. Circuit-level work from September 2025 (arXiv:2509.25758) does report that post-training adds functionally specialized attention heads, with distillation and SFT accumulating stable ones while GRPO iteratively activates, evaluates and prunes a few whose survival tracks fluctuations in the task reward, and it also finds that controllable "think on/off" models have no dedicated thinking heads at all: switching explicit reasoning off recruits a broader and less efficient set of compensatory heads.

Which is the interesting empirical question all along, and it was never "is RLVR a new paradigm." It's how much the quality of the reward signal matters once you swap it. A deterministic checker has a property none of the earlier three have: it cannot be fooled by fluent-sounding nonsense the way a learned reward model can, and it can be queried dozens of times per prompt without drifting, which is precisely what GRPO's group sampling requires. The bill for that property arrives as entropy: an August 2025 study of RLVR (arXiv:2508.14029) reports vanilla training raising Pass@1 while policy entropy collapses, shrinking generation diversity and capping Pass@k, which the authors treat as the upper bound on a model's reasoning, and their self-play problem-synthesis variant recovers 18.3 and 22.8 absolute points of Pass@32 on AIME24 and AIME25. That property, not novelty, is what made math and code the domains where this swap paid off first.

The strongest version of the counterargument: no anchor, no demonstration, and it still learned to check its own work

Set the "just a stricter grader" framing aside for a moment and take R1-Zero at full strength, because the fair version of the counterargument is genuinely harder to dismiss than the marketing version. R1-Zero was not fine-tuned on any human-written reasoning trace before RL began. The policy that entered training was the raw pretrained base model, and the only shaping force applied to it was reward on final-answer correctness plus a format constraint. Everything the paper documents afterward, the self-verification passages, the strategy-switching mid-solution, the rising response length, happened without a single labeled example of a human doing any of those things.

That is a stronger claim than "RL selected among behaviors already present in the training corpus," because it argues the selection process itself, run at a scale of RL steps well beyond what any RLHF budget from the InstructGPT era used, is sufficient to produce structured problem-solving behavior that no supervised signal ever pointed toward. If true at face value, RLVR isn't reward-signal engineering, it's a demonstration that verifiable-reward RL at sufficient scale can generate capability the pretraining-plus-imitation story doesn't predict. Proponents of this reading aren't being sloppy. They're pointing at a real experimental design choice, no SFT cold start, and a real reported outcome, behavior the paper says wasn't demonstrated anywhere in the model's supervised history. That deserves an answer, not a shrug.

What 'no labeled example' does not mean: the base model, not the checker, is doing the work

"No labeled example of self-correction" does not mean "no exposure to self-correction anywhere in training." The base model underneath R1-Zero was pretrained on a web-and-code corpus that contains enormous quantities of text in which people work through problems, catch their own errors, backtrack, and say some version of "wait, that's wrong" before continuing, exactly the pattern Part 3 of this series describes the pretraining objective as compressing at scale. RL against a correctness checker doesn't need to invent self-verification from nothing. It needs to notice that self-verification, already latent in the base model's distribution from pretraining, correlates with getting the right final answer, and then upweight it. Selection is a much cheaper explanation than invention, and it doesn't require RLVR to be a new capability class to explain the observation.

Whether that selection story fully accounts for everything reported in R1-Zero specifically is exactly the kind of ablation DeepSeek's paper would need to run to settle: what happens if you just sample the base model on the same problems with no RL at all, or run the identical RL recipe on a base model pretrained on a corpus scrubbed of worked-solution text. Neither test is described in the abstract I'm working from here, and I'm not going to claim a result that isn't in it — that gap is worth naming rather than papering over. What is documented, and predates RLVR by more than two years, is a comparable case that doesn't depend on R1-Zero at all.

Anthropic's Constitutional AI paper already showed chain-of-thought-style self-critique emerging from an RL loop trained against a model-generated preference signal, with zero human-labeled critique examples anywhere in the pipeline. The behavior showed up under RLAIF in 2022, using a learned reward model, not a programmatic checker. Whatever is producing structured self-correction under optimization pressure, it isn't specific to verifiable rewards. It's a property of RL-against-a-signal generally, applied to a base model whose pretraining corpus already contains the raw material.

The chain-of-thought gets longer because longer is being rewarded, and that's the whole trend

Go back to the graph the paper opens with. Response length rises across RL training steps because the reward function, correctness on math and code problems, rewards whatever policy gets to the right answer, and for the problems in R1's training set, longer deliberation before committing correlates with getting there. The model isn't discovering that reasoning is valuable in some abstract sense. It's discovering, through the same gradient-following process every RL loop uses, that spending more generated tokens before answering raises its score on this checker. The length curve is the optimization pressure made visible, not a separate finding sitting alongside it.

That reframing has a direct, checkable consequence: if a 2026 announcement claims a model's "reasoning improved," the first thing worth finding is whether the reported number is pass@1 on a held-out verifiable benchmark at a matched or reduced thinking-token budget, or whether it's an aggregate score with no budget control reported at all. The distinction is the whole ballgame. A gain that holds when you cap the model at, say, a quarter of its usual thinking tokens is evidence of trained capability. A gain that evaporates under that cap is evidence you bought more inference-time compute and called it reasoning. Any reader with API access and a token-limit parameter can run this test on a claim themselves this week; nobody needs to take the announcement's word for which one it is.

This is exactly the ablation nobody in the RLVR literature has published cleanly. The open question, as of mid-2026, is whether RL against verifiable rewards at the scale R1-Zero used produces generalization beyond what the checker directly rewards, or whether every reported gain decomposes into longer chains of thought bought with more tokens plus narrow skill on the exact problem class the checker can verify. A widely discussed paper from Apple researchers, "The Illusion of Thinking," argued around mid-2025 (the exact posting date isn't pinned down in the sources I have) for the second reading: accuracy on controlled puzzle tasks collapsed past a complexity threshold, and reasoning-token effort sometimes fell as problems got harder rather than rising to compensate, the opposite of what a genuinely scaling capability should do. A rebuttal followed disputing the measurement, arguing the collapse reflected output-format and token limits rather than an actual reasoning ceiling. That dispute hasn't been resolved by a public ablation that isolates RL-steps-at-scale from token-budget-at-inference as the source of reported gains, and I haven't seen one land as of this writing. By some accounts, at least one frontier lab began exposing a thinking-token budget as an explicit, user-controllable parameter sometime in 2025 rather than leaving it a hidden training artifact — consistent with the mechanism this piece describes — but I'm working from a research memo that itself flags the specific models, dates, and current terms as unverified, so treat that detail as a lead to check, not a settled fact. Until the underlying ablation exists, "reasoning improved" is a claim to check against a token cap, not a claim to take on faith.

---

The series

1. Tokens: the alphabet an LLM actually reads
2. The forward pass: what actually happens to a token, and what it costs
3. Pretraining: the law everyone quotes governs a number nobody optimizes anymore
4. Post-training: where behavior comes from (you are here)
5. Making it yours: distillation and fine-tuning a domain model