Model Distillation Tutorial: From LLM to Deployable SLM

Model distillation is the process of transferring knowledge from a large, expensive language model into a small, efficient one. The result is a model that’s 10–100x smaller, runs on commodity hardware, and matches the original on your specific task.

What you’ll build

By the end of this tutorial, you’ll have:

• A teacher model (e.g., GLM-5) generating high-quality training data
• A validated synthetic dataset tailored to your task
• A fine-tuned student model (e.g., Qwen3 1.7B) that runs anywhere
• A clear understanding of how each step works

Prerequisites

You don’t need ML expertise. You do need:

• A clear idea of what task you want the model to perform
• 10–50 seed examples showing input-output pairs
• A test set of 50–250 examples to evaluate the result, per how big should your test set be

Step 1: Define your task

Every distillation project starts with a task definition. Write a plain-language description of what your model should do:

“Classify incoming customer support tickets into one of five categories: billing, technical, account, shipping, or general.”

Then gather your seed examples. Each example is a messages conversation with a user turn (input) and an assistant turn (expected output):

{"messages": [{"role": "user", "content": "I was charged twice for my subscription"}, {"role": "assistant", "content": "billing"}]}
{"messages": [{"role": "user", "content": "The app crashes when I try to upload a file"}, {"role": "assistant", "content": "technical"}]}
{"messages": [{"role": "user", "content": "How do I change my email address?"}, {"role": "assistant", "content": "account"}]}

Ten to fifty examples is enough to get started. Focus on covering the range of inputs your model will see in production.

Step 2: Choose your teacher model

The teacher model generates the synthetic training data your student will learn from. Pick the most capable model you can afford for this step. It only runs during training, not in production. If you’re choosing from a hosted catalog, which teacher model to pick compares the usual candidates head to head.

Teacher Model	Parameters	Strengths
GLM-5	744B (40B active)	Strong general-purpose, good instruction following
Qwen3 235B	235B	Excellent reasoning, multilingual
DeepSeek R1	671B (MoE)	Deep reasoning, chain-of-thought. Not available for tool-calling tasks

The teacher doesn’t need to be perfect. It just needs to be better than random on your task: the validation step will catch mistakes.

Step 3: Evaluate the teacher

Before generating training data, confirm the teacher can actually do your task. Run it against your test set and measure accuracy. This scored run is a teacher evaluation, and it takes minutes where training takes hours.

This step catches problems early. If the teacher struggles with your task, you need to either:

• Improve your task description
• Provide better seed examples
• Choose a more capable teacher

A teacher accuracy of 80%+ is a good starting point. Student models routinely match or exceed the teacher after distillation because they benefit from the concentrated, validated training set.

Step 4: Generate synthetic data

This is the core of the distillation pipeline. The teacher model generates hundreds or thousands of new examples based on your task description and seed data.

A good generation pipeline uses mutation strategies to ensure diversity:

• Topic mutation. Vary the subject matter across examples.
• Complexity mutation. Mix simple and difficult cases.
• Length mutation. Vary input and output length.

Each generated example is validated automatically. Invalid, duplicate, or low-quality examples are filtered out. A typical pipeline generates 500–2,000 usable examples from just 10 seeds.

Step 5: Choose your student model

The student model is what you’ll deploy to production. Choose based on your constraints:

Student Model	Parameters	Best For
SmolLM2 135M	135M	Edge devices, ultra-low latency
Qwen3 0.6B	600M	Balance of speed and accuracy
Llama 3.2 1B	1B	General-purpose baseline
Llama 3.2 3B	3B	Complex tasks needing more capacity
Llama 3.1 8B	8B	Maximum accuracy, still far smaller than the teacher

Smaller models are faster and cheaper to run. Start small and only scale up if accuracy isn’t sufficient. In practice what size model you need is set by your hardest deployment constraint rather than by the task.

Step 6: Fine-tune the student

Train the student model on your validated synthetic dataset. Key configuration:

base:
  task: classification
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
tuning:
  num_train_epochs: 4
  use_lora: true
  learning_rate: 0.0002
synthgen:
  generation_target: 1000
  teacher_temperature: 0.6

LoRA (Low-Rank Adaptation) is the default training method. It’s faster, uses less memory, and produces results comparable to full fine-tuning for most tasks.

Training typically takes 30 minutes to a few hours depending on the dataset size and student model.

Step 7: Evaluate the student

Compare your fine-tuned student against the teacher on your held-out test set. Metrics to track:

• Accuracy. Does the student produce correct outputs?
• Consistency. How stable are outputs across similar inputs?
• Latency. How fast is inference compared to the teacher?
• Cost. What’s the per-request cost reduction?

On narrow tasks distilled students routinely match or beat their teacher while running orders of magnitude faster. Can a small model beat its teacher collects the published margins.

Step 8: Deploy

Fine-tuned SLMs are small enough to run almost anywhere:

• Serverless API. Deploy behind an endpoint for easy integration.
• On-premises. Run on your own infrastructure for data privacy.
• Edge devices. Models under 3B parameters run on mobile hardware and laptops.

Each route wants the weights in a different shape, and deploying a fine-tuned small language model covers what to hand each one.

Your deployed model processes requests in milliseconds, costs a fraction of API calls to frontier models, and keeps all data under your control.

Common pitfalls

Starting with too little evaluation data. If your test set is too small or unrepresentative, you won’t know whether distillation worked.

Skipping teacher evaluation. If the teacher can’t do the task, the student won’t learn it. Always validate the teacher first.

Over-generating without validation. More data isn’t always better. A thousand validated examples outperform ten thousand noisy ones.

Choosing a student that’s too small. Start with a 1B–3B model. You can always compress further once you’ve validated the approach.

Putting it all together

The full distillation pipeline looks like this:

1. Define task + gather seed examples.
2. Select and evaluate a teacher model.
3. Generate and validate synthetic training data.
4. Fine-tune a small student model.
5. Evaluate against your test set.
6. Deploy to production.

With distil labs, this entire pipeline runs from a single configuration. Describe your task, provide your examples, and the platform handles teacher evaluation, data generation, training, and evaluation automatically.

For background, the technique was introduced by Hinton and colleagues in 2015, and DistilBERT demonstrated it on a production language model.

Reference material while you work: supported models for the teacher and student catalog, metrics for reading the evaluation output, and teacher-student distillation for the concept behind the pipeline.

Start at Step 1 with a task you already send to a frontier model. Step 3 tells you within a single run whether the rest of the pipeline is worth building.