Evaluation is the part of the pipeline that decides whether anything else was worth doing, and it is where most of the avoidable mistakes happen. Not because the metrics are hard, but because the test set is usually built after the fact, from the same distribution as the training data, and therefore cannot detect the failures that matter.
Evaluate the teacher first
Before generating a single training example, check that the teacher model can actually solve your task. This takes minutes and it is the highest-value check in the whole pipeline.
The reasoning is mechanical. The student learns to reproduce the teacher’s behaviour. If the teacher is 75% accurate on your task, the student’s ceiling is roughly 75%, and you will spend a training run and an evaluation cycle discovering something you could have known upfront. A teacher that fails here is a signal to reconsider the task framing or the teacher choice, not to proceed and hope.
Metrics follow the task type
There is no universal metric, and picking the wrong one produces confident nonsense.
Classification has clean answers: accuracy when classes are balanced, F1 when they are not. Accuracy on a dataset that is 95% one class will read 95% for a model that has learned nothing.
Extraction and structured output want exact match, sometimes relaxed to normalised match. Tool calling wants schema validity plus argument correctness. A call with the right function name and wrong arguments is a failure, not a partial success.
Free-text generation is the genuinely hard case. Overlap metrics like BLEU and METEOR were built for machine translation and correlate poorly with quality on generation tasks; they are treated as legacy here for that reason. LLM-as-judge evaluation is the more useful approach, with the caveat that you are now trusting one model’s opinion of another’s.
Why good scores fail in production
The usual cause is that the test set was drawn from the same distribution as the training data. Both are clean, both are representative of what you imagined, and neither resembles what users actually send.
The fix is unglamorous: hold out real examples before training, including the messy ones. If your training data is synthetic and your test set is also synthetic, your evaluation is measuring internal consistency rather than performance.
Comparing the fine-tuned student against the base model on the same test set is the other check worth running. It tells you what the training actually bought, which is a different question from whether the final number looks good.
Where this cluster goes
The articles below cover teacher evaluation, which metric fits which task, sizing a test set, building one that catches real failures, reading the base-versus-fine-tuned comparison, and the twelve-model benchmark that produced much of the data cited across this site.