Metrics

Aggregated scores come back as the *_performance object of a metrics command. Per-example predictions are a separate download (CLI reference).

Metrics by task

Suite	Tasks	Metric keys
Question answering	question-answering	rouge, binary, llm-as-a-judge, llm-as-a-judge-reference-free
Conversation	chat-completion, chat-completion-agentic	rouge, tool_call_equivalence, binary_tool_call, staged_tool_call, llm-as-a-judge, llm-as-a-judge-reference-free

All metrics are on a 0-1 scale.

• llm-as-a-judge-reference-free. An LLM verdict on whether the prediction solves the problem, from the full prefix and the prediction alone: good = 1, bad = 0, or a graded score from 0 to 1. The judge never sees the reference, so a wrong reference doesn’t lower the score. This is the default primary metric when the teacher wrote the reference answers.
• llm-as-a-judge. An LLM verdict on whether the prediction matches the reference answer: good = 1, bad = 0, or a graded score from 0 to 1. Read it next to the reference-free score: a prediction marked bad against the reference but good without it points at the reference row, not the model.
• binary. Exact string equality, ignoring differences in whitespace. The faithful measure for a label or a set; for free text, a cross-check only.
• rouge. Text overlap. A secondary signal.
• tool_call_equivalence. Exact match, except that an argument set to its schema default counts as equal to omitting it.
• binary_tool_call. Strict name and argument equality.
• staged_tool_call. 0.25 per stage: one call on each side, then the name matches, then the argument keys match, then the arguments match. A score of 0.5 means the right tool with the wrong arguments.

Both judge metrics see the full prefix, including the question, and follow the job description’s llm_as_a_judge_instructions. By default the judge gives a pass/fail verdict, and the metric is the fraction of predictions marked good. When the instructions ask for a number from 0 to 1, the judge gives a graded score, and the metric is the mean score (Graded score). An unparseable verdict counts as 0.

For the conversation suite, a predicted turn can carry text, tool calls or both. The tool-call metrics compare parallel calls positionally, so the right calls in the wrong order score 0, and extra or missing calls are penalized. A turn with no calls on either side scores 1 on the tool-call metrics, which keeps them meaningful for tool-free rows; the judge metrics carry the text.

A metric a run didn’t compute comes back as null rather than being absent, so filter those out before averaging.

Primary metric per task

One metric summarizes a run; the others help explain it. The default depends on who wrote the test set’s answers, for every task:

The test answers come from	Primary metric
You (gold labels)	llm-as-a-judge, scored against the reference
The teacher (relabeled or synthetic rows)	llm-as-a-judge-reference-free; switch to llm-as-a-judge when the teachers score near 1 on it

Use binary exact match only where exact string equality is the faithful measure: a label (Classifying into labels) or a set. Never use it for free text or code, where correct answers differ in form. You can name a different primary metric for your task.

Every score in Reading a score (teacher, base student, tuned student, production model and closed) is the primary metric you use.

Reading a score

A score on its own says little: 0.72 is close to a teacher at 0.75 and far from a teacher at 1.00. Read it against the reference points of the same test set:

Reference point	Where it comes from	What it tells you
Teacher	teacher_performance from teacher evaluation	The ceiling. The student learns from the teacher.
Base student	base_model_performance from distil slm metrics	The floor: what the student does untrained.
Tuned student	tuned_model_performance from distil slm metrics	What training produced.
Production model	teacher_performance from a teacher evaluation run with the production model as the teacher	The model you’re replacing: the floor the student has to clear.

The production model’s score is its answers scored with the same metrics as the student, against the same references. It exists only when the production model is in the teacher catalog; when it isn’t, judge the student on the absolute score.

The fraction of the gap between base student and teacher that training closed, with each term read on the primary metric:

closed = (tuned - base) / (teacher - base)

A closed of 1 means the student matches the teacher, above 1 that it beats the teacher’s score on this test set, and 0 that training changed nothing. When teacher - base is small, the ratio moves a lot for small changes in the scores.

With an empty test split there are no scores: the model is unmeasured.

Variation between runs

The judge runs at non-zero temperature, so the same model on the same test set scores differently from run to run. A difference between two scores means something only when it’s larger than that variation. Measure it for the test set you use, for example by running the same evaluation again. Changing the judge model or its instructions changes what every score measures, so scores from different judges can’t be compared.

The predictions file

Every metrics command has a matching download that writes the per-example detail. Failure patterns are found here, not in the aggregate.

distil teacher-evaluation download-predictions <teacher-evaluation-id>
distil slm download-predictions <slm-id>

Each command writes a JSONL file to the current directory, one test example per line, with the prompt, the reference completion, the prediction and that example’s score under each metric name. Group the rows the model got wrong by your task’s failure modes: a class, a format, a rule.

Check the judge before you trust a low score: sample predictions the judge marked bad. If they look correct, the judge instructions need fixing before anything else (Job description).