Distillation

Distillation takes a large, capable model (the teacher) and uses it to train a much smaller model (the student) that imitates its behavior on the task you care about. The motivation is economic: the big model is accurate but slow and expensive, while the student, once it has absorbed the teacher's behavior on your task, delivers most of that quality at a tenth of the cost or less. You are not compressing a file; you are transferring know-how from one network into another, and the transfer only covers what the teacher was asked to demonstrate.

From soft labels to sampled answers#

The original recipe, from Hinton, Vinyals, and Dean in 2015, trains the student to match the teacher's full output distribution rather than its single top answer. Those soft probabilities carry more signal than a hard label: when the teacher rates one answer 70% likely with a plausible 20% runner-up, it is revealing how it weighs the options, and the student learns that structure of mistakes and near-misses, not just the verdict.

Modern LLM practice mostly uses a looser, sequence-level version: prompt the teacher, collect its generated responses, and fine-tune the student on those prompt-response pairs as ordinary supervised data. The reason is access. Matching output distributions requires the teacher's token probabilities, which you only get from open-weight models you host yourself; an API teacher gives you text. So when people say "we distilled GPT-class quality into an 8B model," they almost always mean supervised fine-tuning on teacher outputs. The flagship public example is DeepSeek-R1: its team used roughly 800,000 curated samples from the large reasoning model to fine-tune students from 1.5B to 70B parameters, and the distilled 14B model beat much larger open models on math benchmarks.

Why it pays: the cost and latency math#

Work the numbers for a typical production task: 5 million requests a month, averaging 500 input and 200 output tokens each. On a frontier teacher priced around $3 per million input tokens and $15 per million output tokens, that is about $22,500 a month. A distilled small model priced around $0.15 and $0.60 lands near $1,000 a month, a roughly 20x cut, and a self-hosted 8B student on a single GPU can go lower still. Latency improves alongside: a small model has far fewer FLOPs per token, so it decodes several times faster and time to first token drops, which matters for anything interactive.

The one-time cost of building the student is small by comparison. Generating 100,000 teacher examples averaging 700 output tokens costs on the order of $1,200 at the prices above, less than two days of the teacher's production bill. Distillation also composes with model routing: ship the student for the 90% of traffic it handles well, and route the hard or low-confidence remainder back to the teacher.

Letting the teacher write the training set#

The reason distillation became the default play is that the teacher manufactures its own training data, sidestepping human annotation entirely. The working pipeline has four steps. First, assemble a prompt set that actually covers production: real logged queries, plus authored prompts for rare-but-important slices. Second, generate with the teacher, often asking for reasoning along with the answer, since reasoning traces transfer surprisingly well to students. Third, filter aggressively: deduplicate, drop malformed outputs, and use rejection sampling, where you generate several candidates and keep only ones that pass a check, whether that is a unit test for code, an exact-match answer for math, or an LLM-as-judge score for open-ended text. Fourth, fine-tune the student on what survives.

Filtering is where projects succeed or fail, because the student faithfully copies whatever the teacher got wrong. If 4% of teacher outputs contain a hallucinated policy detail, the student will learn to hallucinate that detail with confidence. Budget as much effort for curation as for generation.

The terms-of-service caveat#

Before pointing a teacher at a million prompts, read the provider's terms. OpenAI, Anthropic, and Google all prohibit using their model outputs to train models that compete with them, and this stopped being theoretical in early 2025 when OpenAI publicly alleged that DeepSeek had distilled its models. Whatever the legal merits, providers actively watch for distillation-shaped traffic.

There are two clean paths. Distilling within one provider's ecosystem is explicitly supported: OpenAI, for example, documents capturing a large model's completions and fine-tuning a smaller OpenAI model on them. And open-weight teachers avoid the issue structurally: DeepSeek-R1 ships under MIT and expressly permits distillation, and several other open-weight licenses allow it with attribution conditions. If the student will be a self-hosted competitive asset, get the license reviewed before generation starts, not after.

Test the student out of distribution#

A student learns to mimic, not to reason from first principles, so its competence ends roughly where the distillation set ends. On prompts resembling the training data it can sit within a point or two of the teacher; on inputs nobody generated examples for, it can fall off a cliff that the teacher would have handled gracefully, because the teacher's robustness came from breadth the student never saw.

That means your evals need two layers. An in-distribution held-out set tells you whether distillation worked at all. An out-of-distribution suite tells you whether you can ship: longer inputs than the training set contained, new product names, adversarial phrasings, other languages if users might send them, and last quarter's weirdest tickets. Measure the teacher-student gap separately on each slice; a 2-point gap in distribution and a 20-point gap out of distribution is a routing problem you want to discover before launch, not after. Re-run the OOD suite on a schedule, since production traffic drifts away from any frozen training set.

Practical takeaways#

Distill when a big model already solves your task and the bill or the latency hurts: let the teacher generate the data, filter hard, fine-tune the student, and expect a 10x to 20x cost cut on the traffic the student keeps. Check the teacher's license or terms before generating, prefer open-weight or same-provider teachers for anything competitive, and judge the student on out-of-distribution evals rather than the data it was trained to copy. Pair it with quantization and routing, and the student handles the bulk of traffic while the teacher remains the fallback for the hard tail.