Model Distillation

Model distillation is a way of training a much smaller model, the student, to do one task the way a large, capable model, the teacher, does it. The reason is money and speed. The teacher is accurate but slow and costly per request. A student that has absorbed the teacher's behaviour on your task can serve it for a small share of that cost. The student does not get a compressed copy of the teacher. It learns only from the examples the teacher was asked to produce, so its skill ends where those examples end.

From probabilities to sampled answers#

The original method, from Hinton, Vinyals and Dean in 2015, trains the student to match the teacher's full set of probabilities for each answer, not just its top choice. Those probabilities carry more than a single label does. A teacher that rates one answer 70% likely and a near miss at 20% is telling the student how close the second option was.

Most work with language models now uses a looser version. You prompt the teacher, collect the answers it writes, and fine-tune the student on those prompt and answer pairs as ordinary training data. The reason is access. Matching the full distribution needs the teacher's probability for every token, which you have only for a model you run yourself. A hosted API returns text, with at most a few token probabilities alongside. When a team says it distilled a frontier model into an 8B one, it almost always means fine-tuning on the frontier model's answers.

The best-known public case is DeepSeek-R1. Its team fine-tuned students from 1.5B to 70B parameters, built on Qwen and Llama, on about 800,000 samples, most of them written by the large reasoning model. The 14B student beat QwQ-32B-Preview, a model more than twice its size, on every metric the paper reports.

What the cost case looks like#

A worked example, with list prices that are illustrative rather than any one vendor's: a task with 5 million requests a month, each about 500 input and 200 output tokens. A teacher at $3 per million input tokens and $15 per million output tokens costs about $22,500 a month. A small model at $0.15 and $0.60 comes to about $1,000, around a twentieth of that, and a student you host yourself on one GPU can go lower. A small model also does far less arithmetic per token, so it answers faster, which matters for anything a user waits on.

Building the student costs less than running the teacher. Generating 100,000 teacher examples of about 700 output tokens each comes to around $1,200 at those prices, under two days of the teacher's production bill. Distillation also works with model routing: the student takes the traffic it handles well, and the hard or uncertain requests go back to the teacher.

The teacher writes the training set#

The teacher produces its own training data, so nobody has to label it by hand. The work runs in four steps. Build a prompt set that covers production: real logged queries, plus written prompts for rare cases that matter. Generate with the teacher, often asking for its reasoning along with the answer. Filter: remove duplicates and malformed outputs, and generate several candidates per prompt, keeping only those that pass a check such as a unit test for code, an exact match for maths, or an LLM judge for open text. Then fine-tune the student on what is left.

Filtering decides whether the student is usable, because the student copies whatever the teacher got wrong. A teacher that sometimes invents a policy detail produces a student that states the same detail with confidence. The consistency of the training set matters as much. On the trademark drafting platform, which is a fine-tune on attorney-written opinions rather than a distillation, Bigcircle first unified years of opinions into a single five-section template, because without it the model learns a dozen individual writing styles and averages them. Teacher outputs gathered from different prompts drift the same way.

Check the teacher's terms first#

Read the provider's terms before sending a teacher a million prompts. OpenAI's terms bar using output to develop models that compete with OpenAI. Anthropic's commercial terms bar using the services to train competing AI models. Google's Gemini API terms bar using the services to develop models that compete with them. The issue became public in January 2025, when OpenAI said it had seen some evidence that DeepSeek had distilled its models.

Two routes avoid the problem. An open-weight teacher can license distillation outright: DeepSeek-R1 is released under MIT, and its model card permits distillation for training other models. Distilling inside one provider is the other route, and it is closing at OpenAI. As of October 2026, OpenAI's fine-tuning guide still describes capturing a large model's answers and tuning a smaller model on them, but it also says the fine-tuning platform is winding down. New users cannot reach it, and existing customers can create training jobs until 6 January 2027 unless the base model shuts down first. If the student is meant to be an asset you own and host, get the licence reviewed before generation starts.

Testing the student where it was not trained#

A student copies the teacher's behaviour on the examples it saw, so its skill ends roughly where the distillation set ends. On prompts like the training data it can sit close to the teacher. On inputs nobody generated examples for it can fail badly, on cases the teacher would have handled, because the teacher's range came from training the student never saw.

Evals for a student therefore need two parts. A held-out slice of the distillation set shows whether the imitation worked at all. A second suite, built from inputs outside it, shows whether the student can ship: longer inputs, new product names, hostile phrasing, other languages if users send them, last quarter's oddest tickets. Measure the gap between teacher and student on each slice separately. A small gap on the first and a large one on the second means those requests should route to the teacher, and you want to learn that before launch. On the trademark platform, production logging compares the base and fine-tuned models on the same inputs, so a regression from a model change shows up as a score movement. A student and its teacher can be logged the same way, and the suite needs re-running as production traffic drifts away from the frozen training set.

Distil when a large model already does the task well and the bill or the wait is the problem. If the large model does not yet do the task well, a student will not either, and the work belongs in the prompt, the retrieval or the teacher's fine-tune first. Pair the student with quantization and routing, and keep the teacher as the fallback for the hard requests.