Teacher-Student Distillation: How It Works and When to Use It
Teacher-student distillation is the core mechanism behind knowledge distillation. A large, capable model (the teacher) generates training signal that a smaller model (the student) learns from, producing a compact model that captures the teacher’s expertise on a specific task.
It’s how you go from a 70B-parameter model running on a GPU cluster to a 1B-parameter model running on a single CPU, without giving up the accuracy that matters.
What is the core idea?
The insight behind teacher-student distillation is simple: a large model already knows how to solve your task. Instead of training a small model from scratch on hand-labeled data, you let the large model demonstrate the correct behaviour and train the small model to replicate it.
This is fundamentally different from traditional supervised learning. In supervised learning, you need humans to label every example. In distillation, the teacher model does the labeling, and it can generate far more training data, far more cheaply, than any human annotation team.
The approach dates to Hinton and colleagues in 2015, who showed a small network could be trained to match a large one’s behaviour rather than the raw labels. Stanford Alpaca later demonstrated the same idea at instruction-tuning scale, producing a usable dataset for a few hundred dollars.
How does the process work?
Teacher-student distillation follows a straightforward pipeline:
1. Define the task
Start with a clear task description and a small set of seed examples (as few as 10). This tells the teacher what kind of outputs you expect. They’re seeds rather than a training set, which is why how many examples you actually need is a much smaller number than people expect.
2. Teacher generates training data
The teacher model (something like GLM-5 or Qwen3 235B) takes your task description and seed examples, then generates hundreds or thousands of new input-output pairs. These synthetic examples form the training dataset.
3. Validate and filter
Not every teacher output is usable. Automated validation removes duplicates, off-topic examples, malformed outputs, and low-quality generations. This step is critical. Noisy training data leads to noisy student models.
4. Student learns from teacher
The filtered dataset is used to fine-tune a small student model (e.g., Llama 3.2 1B, Qwen3 0.6B, or SmolLM2 135M). The student learns to map inputs to outputs the same way the teacher does, but in a fraction of the parameters.
5. Evaluate
The trained student is tested against a held-out test set to measure accuracy, consistency, and any regressions compared to the teacher.
Why not just use the teacher?
In production the teacher’s strengths become liabilities:
| Dimension | Teacher (70B+) | Student (1B–3B) |
|---|---|---|
| Latency | 500ms–2s per request | 20–100ms per request |
| Cost per 1M tokens | $1–10 | $0.01–0.10 |
| Infrastructure | Multi-GPU cluster | Single GPU or CPU |
| Privacy | Often requires API calls | Runs fully on-prem |
| Reliability | Variable (prompt-dependent) | Deterministic on narrow tasks |
The student isn’t better than the teacher at everything. It’s better at the one thing you need, while being dramatically cheaper and faster to run. The size of that cost gap depends on how you serve the student, which is the subject of cost per million tokens, local vs API.
When does teacher-student distillation work best?
Distillation excels in specific conditions:
- Well-defined tasks. Classification, extraction, QA, tool calling. The clearer the expected output format, the better the student learns.
- Narrow domains. You’re not trying to build a general-purpose assistant. You need a model that does one thing reliably.
- Scale matters. You’re running thousands or millions of inferences per day, and cost or latency is a constraint.
- Limited labeled data. You have a handful of examples, not thousands. The teacher bridges the data gap.
When does it not work?
Distillation is less effective when:
- The task is vaguely defined. If you can’t clearly describe what a good output looks like, the teacher will generate noisy data and the student will learn noise.
- You need open-ended creativity. Distillation compresses knowledge, which means the student trades breadth for depth. That trade is one of several reasons for not using a small language model at all.
- The teacher can’t do the task. The student can’t exceed the teacher’s capability on the training distribution. If the teacher gets it wrong, the student will too. Running teacher evaluation first catches this before you spend a training run on it.
How do you choose a teacher and student?
Teacher selection
Pick the most capable model that reliably handles your task. Bigger isn’t always better. What matters is that the teacher produces high-quality outputs for your specific domain.
Common teacher choices:
- GLM-5. Strong general-purpose teacher, and the best performer in our own five-teacher comparison.
- Qwen3 235B. Excellent for multilingual and reasoning tasks.
- DeepSeek R1. Strong on tasks requiring chain-of-thought reasoning, though the catalog marks it unavailable for tool calling.
Student selection
Choose based on your deployment constraints:
| Model | Parameters | Use case |
|---|---|---|
| SmolLM2 135M | 135M | Ultra-low latency, edge devices |
| Qwen3 0.6B | 600M | Balance of size and quality |
| Llama 3.2 1B | 1B | Solid general-purpose student |
| Llama 3.2 3B | 3B | Complex tasks needing more capacity |
How well does the student actually do?
On narrow tasks distilled students routinely match or beat their teacher. A fine-tuned Qwen3-0.6B beat its 120B teacher by 29 points on smart-home tool calling, and a 350M LFM2.5 beat the same teacher on two of three tool-calling benchmarks. Can a small model beat its teacher has the full comparison.
The performance gap is smallest on well-defined tasks like classification and extraction, and largest on open-ended generation tasks. For most production use cases, the student is more than good enough.
How do you get started?
With distil labs, teacher-student distillation takes minutes, not weeks:
- Describe your task and provide 10–50 seed examples.
- Select a teacher and student model.
- The platform generates synthetic data, validates it, trains the student, and evaluates the result.
- Deploy your student model to a cloud endpoint or download it for on-prem use.
You don’t need ML infrastructure, training scripts, or a data labeling pipeline. The supported models catalog lists every teacher and student you can pick from, and knowledge distillation explained covers the underlying concept in more depth.
The first thing to do is run teacher evaluation on two or three candidate teachers and compare them on your own held-out set. Everything downstream depends on that result.