Making it yours: distillation and fine-tuning a domain model

Fine-tuning buys behavior and a price point, not knowledge — and the "Distillation" button in your training console is running a different, weaker mechanism than the paper that coined the word.

Understanding LLMs From the Inside — part 5 of 5.

← Previous: Post-training: where behavior comes from

The button says 'Distillation.' The paper it's named after says something else.

Two mechanisms, then the label problem.

Fine-tuning is continued training. You take a model that has already been pretrained, hand it a comparatively tiny set of input/output pairs, and run gradient updates until its next-token predictions lean toward the shape of those examples. The examples are gone at inference time; what survives is a nudged set of weights. It is cheap next to pretraining, and cheaper still with adapter methods like LoRA, which is why a small team can afford to do it at all.

Distillation, in the original sense, is a teacher-student setup. Geoffrey Hinton, Oriol Vinyals and Jeff Dean published "Distilling the Knowledge in a Neural Network" in March 2015 to solve a deployment problem: averaging the predictions of many models reliably improves accuracy, and serving an ensemble of large nets to a large number of users is too expensive to contemplate. Their answer was to train one compact student to reproduce the ensemble's output probabilities, softened with a temperature term so the small numbers the teacher assigns to wrong answers do not vanish into rounding. That is the load-bearing idea. The student learns which classes the teacher confused, and by how much, which is strictly more information than the teacher's final answer. They reported results on MNIST and a significant improvement to the acoustic model of a heavily used commercial system. Nine years of compression work, DistilBERT included, rests on that paper, and the standard account of why it works is basically correct.

Now open a fine-tuning console in 2026 and find the tab labeled Distillation. OpenAI shipped it at DevDay in October 2024: point the platform at a frontier model, run traffic through it, and the platform saves the completions as a labeled dataset. Fine-tune a cheaper model on that data, per OpenAI's own description of Model Distillation, built on what it calls Stored Completions. It works. It is genuinely useful, and I have no quarrel with anyone shipping on it.

But a stored completion is text. It is the one answer the teacher happened to sample, with the distribution behind it thrown away: no soft labels, no temperature, no logits. Mechanically this is supervised fine-tuning on teacher-generated data, which the 2015 paper would classify as the weaker case it was written to improve on. The feature is now more than a year and a half old, the borrowed word nine years older than that, and the conflation has hardened into default practice. What it costs you is not obvious from the UI, so it is worth working out precisely what a student trained on text can and cannot pick up from its teacher.

What Hinton, Vinyals, and Dean actually distilled: the logits, not the answer

The 2015 paper starts from a practical problem: an ensemble of models generalizes well but is too expensive to serve. Hinton, Vinyals, and Dean wanted to compress that generalization into something small enough to deploy without losing what made the ensemble good. Their answer was to train the student not on the teacher's hard label — the single class it predicted — but on the teacher's softmax output before it collapses into a decision: the full vector of probabilities across every class, raised to a temperature that flattens the distribution and exposes the relative weight the teacher assigned to answers it didn't pick.

That last part is the mechanism. A teacher classifying an image of a BMW might put 0.9 probability on "car," the correct label, but the ratio between its probability on "truck" and its probability on "carrot" carries information about how the teacher generalizes, something Hinton's paper describes as knowledge held in the small probabilities and which later commentary nicknamed dark knowledge. A student trained only on the hard label "car" never sees that ratio. A student trained on the temperature-raised distribution does, which is why Hinton, Vinyals, and Dean report compressing an ensemble into a single model on MNIST, and separately improving the acoustic model of a production speech system, at a size where the ensemble itself was no longer deployable.

The mechanism has a hard requirement built in: access to the teacher's logits, on the training examples, at inference time. Not its answer. Its distribution.

You can't distill logits through an API — so 'distillation' as sold is SFT on sampled text

Here's the mechanical problem with calling a stored-completions product distillation in Hinton's sense: an API does not return logits. It returns sampled text. Call GPT-4o and get back a paragraph, and you're seeing one draw from the model's output distribution, not the distribution itself. OpenAI's Model Distillation feature operates on exactly that draw: the text completions the teacher's API handed back, saved and reused as supervised fine-tuning data for a student. There is no temperature-raised softmax anywhere in that pipeline, because there is no softmax the customer can see. What you're running, mechanically, is supervised fine-tuning on synthetic text generated by a stronger model. That's a legitimate technique. It is not the technique from the 2015 paper, and the difference isn't cosmetic: SFT on sampled text teaches a student to reproduce the shape of an answer, not the teacher's uncertainty about the answers it didn't give.

The clearest acknowledgment of this limit came from inside the field, in June 2023. Orca, from Microsoft Research, was built to imitate GPT-4's "explanation traces" — step-by-step reasoning, not bare final answers — because its authors had diagnosed exactly this failure mode in prior imitation work: smaller models trained on a teacher's sampled outputs tend to learn the teacher's style, not its reasoning process. That's a description, from researchers doing the work, of what happens when you run sequence-level imitation without logit access. Orca's fix — richer traces, staged training, more of them — is real engineering. It's also an admission that black-box imitation starts from a weaker signal than the 2015 paper assumed you'd have.

Fine-tuning's actual job: buying behavior and a price point, not knowledge

Set distillation aside and ask what plain fine-tuning does, mechanically, and the answer is narrower than most pitches suggest. LoRA, introduced by Edward Hu and coauthors in June 2021, freezes the pretrained weights entirely and injects small trainable rank-decomposition matrices alongside them. Nothing about the base model's stored knowledge — the parameters shaped by pretraining, in the terms Part 3 of this series covers — gets touched. What gets trained is a low-rank correction to how the model routes and weighs what it already has. Hu and coauthors report cutting trainable parameters by 10,000 times relative to full fine-tuning of GPT-3 175B, while matching or beating full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3. A correction that small cannot be adding facts at any scale that matters. It's adjusting which behaviors the model already has get surfaced, in what format, under what instruction.

QLoRA, from Tim Dettmers and coauthors in May 2023, pushes the same logic further down the cost curve. It backpropagates through a model frozen and quantized to 4 bits, training only the LoRA adapters on top, which is how the paper reports fine-tuning a 65-billion-parameter model on a single 48GB GPU in 24 hours. That's not a knowledge-acquisition claim. It's a price-point claim: the same underlying capability, reachable on hardware costing a few dollars an hour instead of a cluster.

Both papers are ancient by this field's clock, and that is precisely the point. Low-rank adapters are still the default way to fine-tune an open-weight model in 2026, refined at the edges (rank choice, quantization format, which projections get adapters) rather than replaced, so read this as current practice and not a historical survey.

This is the load-bearing claim of the piece: fine-tuning changes what a model does, not what it knows. It buys a JSON-only output format, a terse support voice, a model that stops refusing a domain-legitimate question, a 7B model doing a 70B model's job at a fraction of the inference-time compute. What it does not reliably buy is a fact the base model never saw in pretraining. Ask a fine-tune about something that shipped after its pretraining cutoff and the adapters won't help. That's a retrieval problem wearing a training-time costume, and mistaking one for the other is the single most common failure in domain fine-tuning projects.

The purist's objection answered on its own numbers: Orca and Guanaco didn't need logits to win

Here's the honest problem with insisting on Hinton's definition: the results don't care. Orca-13B, trained entirely on sampled GPT-4 and ChatGPT text with no logit access at all, beat Vicuna-13B by more than 100% on Big-Bench Hard and 42% on AGIEval. Vicuna was itself a black-box imitation model, fine-tuned on shared ChatGPT conversations. Orca beating it by that margin, under the same black-box constraint, is a result achieved entirely inside the weaker mechanism this piece has been describing.

Guanaco is the harder case, because there's no teacher at all. Dettmers and coauthors report their QLoRA-tuned Guanaco model reaches 99.3% of ChatGPT's performance on the Vicuna benchmark using ordinary supervised fine-tuning on instruction data, in 24 hours on one GPU. No stored completions, no teacher API, no distillation of any kind. If the goal is a small model that scores well on a benchmark built around conversational quality, the mechanism-purist framing doesn't obviously buy anything the fine-tuning literature wasn't already delivering by 2023.

There's a further complication for anyone reaching for the 2015 paper as the standard: almost nobody building a domain model today has the access it requires. If the teacher is GPT-4o or Claude, you get an API and a text box, not a readable softmax, and no contract on offer changes that. Insisting on Hinton's original definition, in that world, risks being an academic distinction with no decision consequence. You were always going to be doing SFT on sampled text, because that's the only door open, and calling it by the wrong name doesn't change what happens when you run it.

That objection is real and should temper how confidently the mechanism claim gets stated. It doesn't survive contact with what happens as the gap between teacher and student widens, though, which is where the distinction stops being academic.

Why the distinction still decides what to do next: a working decision order

The place the mechanism gap should worry you is calibration, not benchmark score. Hinton, Vinyals, and Dean's argument for soft targets was that the ratios between a teacher's probabilities on wrong answers carry generalization information that a hard label destroys. A student trained on sampled text never sees those ratios, because one sample from a distribution is not the distribution. That predicts what Orca's own authors flagged: black-box students learn a teacher's style without inheriting its reasoning process, so the failure surfaces on inputs shaped differently from the training traces rather than on a benchmark built to resemble them. A model can post a strong BBH number, overfit to the benchmark's particular shape, and still fall over on a held-out case that needs exactly the calibrated uncertainty logit distillation transfers and sequence-level SFT structurally cannot copy.

The version of this that people actually deploy in 2026 is not the API product. DeepSeek's R1 report, out in January 2025, argues that reasoning ability can be induced by pure reinforcement learning with no human-labelled reasoning trajectories, producing emergent self-reflection and verification behaviour, and beating models trained by supervised learning on human demonstrations on verifiable work: mathematics, competition coding, STEM. The part relevant here is the last claim in the abstract, that the reasoning patterns a large model develops can be used systematically to improve the reasoning of much smaller ones. The distilled checkpoints shipped on Qwen and Llama backbones, sizes running from roughly 1.5B up to 70B, with open weights you can download and serve yourself. Set that against the scale of the models doing the teaching: the same group's V3 report describes a 671B-parameter mixture-of-experts with 37B active per token, pre-trained on 14.8 trillion tokens for 2.788M H800 GPU hours. That gap is the entire commercial argument.

Note what it does not show, because this is where the chapter's distinction earns its keep. DeepSeek trained the teacher, so they had its logits, and the reported transfer was still done by training students on the teacher's generated reasoning traces. The paradigm case of distillation-that-works in the current field is therefore the same sequence-level mechanism sold through stored completions, run by people who controlled the generator and worked in a domain where answers can be checked. The student inherits the output of the teacher's search, not the capacity to search: the RL loop that produced those traces stays with the teacher, and Part 4's material explains why that loop is not something a supervised objective reconstructs. Nor does the report demonstrate transfer where correctness is unverifiable, which is most enterprise work. And nobody has published the ablation that would settle how much soft labels were worth here: same prompts, same student, teacher logits against sampled text, everything else fixed. A lab that owns both the teacher weights and the student recipe is one of the few parties who could run it. As of July 2026 that comparison is an open gap, not a resolved footnote.

Which is why the synthetic dataset, rather than the training run, is where distillation is won or lost. The mechanics are mundane. Prompt the teacher, keep the completions, fine-tune the student on the pairs; everything that matters lives in what you prompted with and what you threw away. Orca's authors diagnosed the imitation work that preceded them as suffering from shallow teacher outputs, training sets that were small and homogeneous, and evaluation loose enough to overestimate the student. Their data-side fix was to collect GPT-4 explanation traces and step-by-step thought processes instead of final answers, with ChatGPT acting as an intermediate teaching assistant, then apply what they call judicious sampling and selection over a large and diverse pool. Dettmers and colleagues found the complementary result across the more than 1,000 models they trained for QLoRA: a small high-quality set beat bigger ones, well enough that a 65B student tuned for 24 hours on a single 48GB GPU reached 99.3% of ChatGPT's score on the Vicuna benchmark.

The failure mode to design against is inheritance. A teacher's confident mistake enters the training set as a target indistinguishable from a correct one, with no soft label left to record that the teacher was genuinely torn. Skew in the prompt distribution transfers the same way. Generate every example from one template and you have taught the student that template, then shipped it into traffic that looks nothing like it: distribution shift you manufactured yourself. Verify what is machine-verifiable (arithmetic that checks out, code that actually runs) and drop the traces where the teacher hedged rather than promoting hedges to ground truth. This is also why the R1 result travels less well than it looks. Verifiable domains let you filter the teacher's output automatically; a legal summarization pipeline does not.

Then the eval, and the trap most tuning projects walk into. Hold out ten percent of the teacher-generated set and the number you get measures how faithfully the student reproduces the teacher on prompts from the same generator. That is fidelity, not competence, and it will look excellent. Orca names the consequence outright: absent rigorous evaluation, you overestimate the small model's capability. The QLoRA authors went further, reporting that current chatbot benchmarks are not trustworthy for ranking chatbots at all, while separately finding GPT-4 judging to be a cheap and reasonable substitute for human raters. Hold both at once: use the judge, distrust the leaderboard. What to measure instead is a frozen set of real production prompts collected before you tuned anything, never generated by the teacher, labelled by someone who knows the domain, scored on the behaviour you were actually buying, which usually means format compliance and cost per call. Copy QLoRA's lemon-picked analysis too and go read the cases where your student loses. Note how Orca's own headline numbers depend on configuration: parity with ChatGPT on BBH, more than 100% over Vicuna-13B there and 42% on AGIEval, but a 4-point gap on professional exams with an optimized system message, trailing GPT-4 throughout. The system message is part of the eval harness, and a figure quoted without it is not comparable to yours.

So the order runs one direction, with the eval built first because nothing downstream is legible without it. If the failure is missing facts, that is retrieval, and no amount of fine-tuning reliably injects an absent fact. If the model knows the fact but expresses it wrong, refuses when it shouldn't, or costs too much per call, that is fine-tuning, and LoRA or QLoRA is almost always the right first attempt: cheap enough to fail fast, and in LoRA's case with no additional inference latency to pay for afterwards. Reach for a distilled reasoning model when the task is one whose answers you can check and a downloadable 7B or 32B student would let you stop paying frontier prices, which is precisely the shape of the case DeepSeek reported. Reach for building your own only after answering whether you have logit-level access to the teacher or only its sampled text. If the teacher isn't yours, it is sampled text. Budget accordingly: for the curation and filtering the synthetic set will demand, and for the Orca-style outcome where you buy the style and not the reasoning.

Test it yourself before you buy either claim

There's a cheap way to check whether a fine-tune added knowledge or just reshaped output, and it takes an afternoon. Ask your fine-tuned model, in the exact tone and format you trained for, about something you're certain postdates its base model's pretraining and never appeared in your training set. Confident and wrong confirms the adapters reshaped output without touching what the base model knows, exactly as the LoRA mechanism predicts. Then ask the same model to reason through a case shaped differently from your training examples. Hard degradation there is a black-box distillation ceiling, not a training bug, and retrieval fixes the first problem but not the second. Building the eval harness that separates those two failures, before scaling either fix, is worth more than another epoch of training.

---

The series

1. Tokens: the alphabet an LLM actually reads
2. The forward pass: what actually happens to a token, and what it costs
3. Pretraining: the law everyone quotes governs a number nobody optimizes anymore
4. Post-training: where behavior comes from
5. Making it yours: distillation and fine-tuning a domain model (you are here)