What Is Teacher Evaluation?
Teacher evaluation is a scored run of the teacher model against your held-out test set, executed before any student training starts. It answers two questions: can a large model solve this task at all, and what accuracy should the student be aiming at? It takes minutes. Training takes hours.
What does teacher evaluation measure?
It measures the teacher’s accuracy on the same test set your student will later be scored on, using the same metrics. Nothing is trained and nothing is generated. The teacher is prompted with your task description and a small number of few-shot examples, and its answers are graded.
The teacher evaluation docs describe two distinct jobs it does:
| Job | What it tells you |
|---|---|
| Feasibility check | Whether a large model can solve your task at all, before you spend a training run finding out |
| Performance benchmark | A first approximation of the accuracy your trained student can reach |
The metrics you get back depend on the task type. Question answering and classification return LLM-as-a-Judge, Exact-Match and ROUGE-L; tool calling returns tool_call_equivalence, binary_tool_call and staged_tool_call. The metrics guide covers how to read each one.
Why does it run before training instead of after?
Because a failure here is cheap and a failure after training isn’t. The student learns to reproduce the teacher’s behaviour on synthetic data the teacher generates, so a teacher that can’t do the task produces training data that teaches the wrong thing.
There’s a second reason. When the teacher scores badly, the cause is usually the task description rather than the teacher. The docs list revising the task description, improving example quality, checking for dataset inconsistencies and confirming the task is well-defined as the four things to try, and only the last of those is about model choice. Teacher evaluation is therefore the first place a badly specified task becomes visible, and it’s much easier to fix a job description than to debug a trained model.
What score is high enough to proceed?
There’s no universal threshold, because the number that matters is whether the teacher clears the bar your application needs. In practice, teams set a task-type threshold and treat it as a gate. In the trace-training walkthrough, a teacher evaluation scoring 0.808 on LLM-as-a-Judge was recorded as PROCEED (0.808 ≥ 0.70 tool-calling threshold).
Two calibrations are worth holding on to:
- The teacher score is a strong expectation, not a hard ceiling. In our 12-model benchmark a fine-tuned Qwen3-4B matched or beat its GPT-OSS-120B teacher on 7 of 8 tasks, and beat it by 19 points on SQuAD 2.0. Can a small model beat its teacher goes into why.
- A low score isn’t automatically a reason to swap teachers. Fix the task specification first. Teacher choice is the last lever, not the first.
What do you change when the teacher fails?
Change the inputs before you change the model. The order that works is: sharpen the task description, then improve the seed examples, then check the test set isn’t itself the problem, then consider a different teacher.
If you do reach for a different teacher, which teacher model should you pick has a measured five-teacher comparison on a shared test set. The two constraints that actually bind are tool-calling support and the reasoning-family temperature band, both covered in the config reference.
Every command names the entity it acts on, and the evaluation itself hangs off a seed dataset:
| Command | What it does |
|---|---|
distil teacher-evaluation create-from-seed-dataset <seed-dataset-id> |
Start the evaluation, and print the evaluation ID |
distil teacher-evaluation status <teacher-evaluation-id> |
Show whether the job is still running |
distil teacher-evaluation metrics <teacher-evaluation-id> |
Show scores by evaluation ID |
distil teacher-evaluation download-predictions <teacher-evaluation-id> |
Download per-example predictions as JSONL |
Download the predictions. The aggregate score tells you whether to proceed; the per-example file tells you why the teacher missed, and that’s what you act on. Read twenty of the wrong predictions before you change anything.
Two config values shape the run itself: evaluation.num_few_shot_examples defaults to 1, and evaluation.llm_as_a_judge_model_name defaults to openai.gpt-oss-120b. If you’re comparing evaluations across time, keep both fixed, since changing the judge changes the scale.
Related terms
- Teacher model. The large model whose behaviour is being distilled, and the thing being scored here.
- Student model. What gets trained afterwards, and what this score sets expectations for.
- Held-out test set. The data teacher evaluation runs on; if it’s contaminated, this number means nothing.
- LLM-as-judge evaluation. The default metric for generation tasks, and the subject of an active research literature.
- How to choose a teacher model. What to do when this gate fails on capability rather than specification. The larger mixture-of-experts teachers such as GLM-5 are the usual next step.