Synthetic Training Data

Synthetic data is data produced by a model and then used for training. It can be instruction examples, code tasks, reasoning traces, preference pairs, tool-call transcripts, or domain-specific question-answer sets. It is not automatically good. The whole craft is generating diverse candidates, filtering aggressively, and training only on the survivors.

Generation patterns#

Self-Instruct showed the basic loop: seed a small set of tasks, ask a model to invent more instructions and answers, filter them, and fine-tune. Modern variants add complexity prompts, personas, domain constraints, and teacher-student distillation. A strong teacher model can generate traces that make a smaller student behave better than its raw size suggests.

Quality beats volume#

The risk is training on bland, wrong, duplicated, or self-reinforcing text. Model collapse papers show that recursively training on generated data can erase distribution tails. Successful synthetic-data pipelines avoid that by grounding generation in real seeds, using verifiers, rejecting low-novelty examples, and mixing human or naturally occurring data back in.

Why it works in practice#

Phi-style models and frontier post-training recipes showed that curated synthetic data can be extremely effective. The reason is not that synthetic text is magical. It is that a data pipeline can target exactly the behaviors the model needs: step-by-step math, function calls, refusals, SQL repair, code tests, or customer-support style.

Practical takeaways#

Use synthetic data when you can specify and check the desired behavior. Keep provenance, watch licensing and model-provider terms, deduplicate hard, and maintain held-out evals made from real use. The best synthetic sets are small compared with web-scale pretraining but dense with the skill you want.