Learn
Practical guides to fine-tuning, distillation, and deploying small language models.
Knowledge Distillation for LLMs: Compress GPT-4 into a 3B Model
Learn how knowledge distillation lets you compress the capabilities of massive language models like GPT-4 and Llama 70B into small, deployable models with 1B–8B parameters, without sacrificing accuracy on your task.
Teacher-Student Distillation: How It Works and When to Use It
Learn how teacher-student distillation transfers knowledge from a large language model to a small, efficient one. Understand the training process, when it makes sense, and how to get started.
What Is Catastrophic Forgetting?
A fine-tuned model that nails its new task but has lost the general abilities it arrived with has forgotten catastrophically. Here's how to spot it, fix it, and when to accept it.
What Is Overfitting in Fine-Tuning?
Overfitting is when a fine-tuned model memorises its training examples instead of the pattern behind them. Spot it from the held-out gap and fix it with data before hyperparameters.
When Does Distillation Fail?
Distillation fails on under-specified tasks, inconsistent labels and open-ended outputs. Ranked causes, the fix for each, and the teacher evaluation step that catches most of them early.
Why Did My Fine-Tuned Model Get Worse?
Your tuned student scores below the base model or the teacher on its own task. Work through the metric artefacts first, then the ranked causes, then the re-run.
Why Your Model Passes Eval But Fails in Production
Diagnosing the gap between a good held-out score and bad real-world behaviour: filtered test sets, per-turn scores read as per-session reliability, and evaluation data drawn from the same place as training data.
Epoch, Batch Size and Learning Rate Explained
The three hyperparameters that decide how a fine-tuning run proceeds, what each one is called in a distil labs config, and which of them is worth changing first.
What Are Logits?
Logits are the raw per-token scores a model emits before any softmax, the quantity temperature divides and the quantity classical distillation trains against.