Distillation vs Pruning
Pruning removes weights, heads, or layers from a trained network and keeps whatever survives. Distillation leaves the original alone and trains a separate small model to reproduce its behaviour. Pruning starts from the large model’s structure; distillation starts from the target model’s structure.
How do the two techniques compare?
The difference is what the output model inherits: architecture in one case, behaviour in the other.
| Pruning | Distillation | |
|---|---|---|
| Starting point | The large model’s weights | A small model’s checkpoint |
| Operation | Delete parameters | Train on teacher-derived data |
| Output architecture | A subset of the original | Independent of the teacher |
| Family constraint | Output is the same family | Student and teacher can be unrelated |
| Typical size change | Bounded by how much you can cut | 10–100x, set by the student you chose |
| Task scope | General capability, degraded | Narrow task, often improved on it |
| Recovery step needed | Yes, retraining after cutting | No, training is the whole method |
The family constraint is the practical one. Pruning a 15B model gives you a smaller version of that model; it can’t give you a Qwen. Distillation, in the sense Hinton and colleagues introduced, is indifferent to that, and does the student need the same architecture as the teacher covers why text-level distillation transfers behaviour rather than structure.
What does pruning actually remove?
It removes either individual weights or whole structural units, and the distinction determines whether you get a speedup.
Unstructured pruning zeroes individual weights wherever they’re least important. It produces a sparse matrix that’s smaller on disk but not faster on ordinary hardware, because dense matrix multiply doesn’t care that some entries are zero. You need sparsity-aware kernels to convert it into throughput.
Structured pruning removes entire attention heads, MLP channels, hidden dimensions, or layers. The result is a genuinely smaller dense model that runs faster everywhere. Sheared LLaMA is the reference example: targeted structured pruning of LLaMA2-7B to 1.3B and 2.7B by “removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner”, combined with dynamic batch loading during the retraining that follows.
That retraining isn’t optional. Cutting a trained network breaks it, and something has to put it back together.
Why do pruning pipelines end in distillation anyway?
Because the recovery step after cutting is a distillation step. The published pipelines say so in their own titles.
NVIDIA’s Minitron work is titled Compact Language Models via Pruning and Knowledge Distillation for that reason. The method combines depth, width, attention and MLP pruning with knowledge distillation as the retraining objective, and derives 8B and 4B models from an already-pretrained 15B model “using up to 40x fewer training tokens per model compared to training from scratch”.
Sheared LLaMA reports the same shape of result: 1.3B and 2.7B models from LLaMA2-7B “requiring only 3% of compute compared to training such models from scratch”.
So the comparison is really between a fresh small model trained on teacher behaviour and a surgically reduced copy of the large model restored with a distillation objective. Both end with distillation; they differ in where they start.
Which one produces a better production model?
For a bounded task, distillation into an off-the-shelf small model wins by a clear margin, because you aren’t trying to preserve general capability at all. What size model you need covers picking that off-the-shelf student.
Pruning aims to keep a model broadly useful at a smaller size. Task-specific distillation aims to make a small model excellent at one thing and accepts that it’s worse at everything else. When the deployment target is one job, the second objective is the one that matches.
The measured version: in the distil labs 12-model benchmark, a fine-tuned Qwen3-4B-Instruct-2507 matched or exceeded its 120B-parameter teacher on 7 of 8 tasks, beating it by 19 points on SQuAD 2.0. No pruning of a 120B model to 4B would produce that outcome, because pruning can’t exceed its source on the source’s own distribution.
Pruning wins when you need a general model that’s smaller, such as a compressed base checkpoint for other people to build on, which is exactly what the Minitron and Sheared LLaMA papers were producing.
Which should you reach for?
Reach for distillation if you’re shipping an application. Reach for pruning if you’re producing a base model.
Choose distillation when:
- You have one task, or a small set of them
- You want to pick the student independently on size, licence and tool-calling support, per the supported models catalog
- You have seed examples, or a corpus a teacher can generate them from
- Your target is on-prem, edge, or CPU-only
Choose pruning when:
- You need a general-purpose model in a size nobody publishes
- You have pretraining-scale compute and data available
- Preserving broad capability is the requirement, not task accuracy
The techniques also stack in the other direction. A pruned base model is still a base model, and can be distilled for a task afterwards. For the third compression option that changes neither architecture nor behaviour, see distillation vs quantization; for the mechanics of the distillation side, how to distill a large language model and the model training docs.