Model iterations

A trained model that isn’t good enough is improved by running part of the build loop again with one change. The test set stays locked, so every score stays comparable, and the last model’s predictions show what to change. Everything upstream of the step you change is reused: a new teacher evaluation, a top-up of the train set or a training run all point at a dataset already in the chain.

One iteration has five steps:

 1 Read the scores
   ▼
 2 Inspect the predictions
   ▼
 3 Find the step the errors come from
   ▼
 4 Change one setting in that step
   ▼
 5 Run from that step through training ─────► back to 1

New traffic recorded by the serving endpoint is a different kind of iteration: it goes round the whole loop with a new dataset (The next iteration).

1. Read the scores

A trained model’s score is read against four others, all measured on the same test set:

• The teacher’s score from teacher evaluation, the upper limit of what the student can learn from it.
• The base student’s, the untrained student, which training starts from.
• The production model’s, from its teacher evaluation, when it was evaluated (The production model as the baseline).
• The previous iteration’s.

Scores are on the task’s primary metric (Primary metric per task). The verdict comes from closed, the fraction of the base-to-teacher gap the student closed (Metrics):

closed	Verdict	Next
≥ 0.8, and above the production model	Deploy candidate	Serve it
0.4 - 0.8	Retune	Step 8 of the loop: train again on the same dataset with another student or tuning settings
< 0.4	Iterate	The teacher’s knowledge isn’t reaching the student: this page

A difference is meaningful only when it is larger than the run-to-run variation of the evaluation, which you measure by running the same evaluation twice (Variation between runs). A change inside that noise is neither progress nor a regression.

Every training run scores the base student again, so closed moves with the noise in both the tuned and the base score. Use one base score for the project: the first run’s, or the mean of the runs so far. Count a target as reached only when closed still clears it with the tuned score lowered by the noise.

2. Inspect the predictions

Download the trained student’s predictions and the teacher’s predictions on the same test set. Each command writes a JSONL file to the current directory:

distil slm download-predictions <slm-id>
distil teacher-evaluation download-predictions <teacher-evaluation-id>

Match rows between the two files on the last user message, since one conversation can produce several evaluation rows, one per assistant turn. For each row the student got wrong, note what it answered, what the reference says, and whether the teacher got it right.

Group the errors into named failure modes: specific scenarios such as “refunds for annual plans”, “dates in European format” or “calls search when lookup was needed”, not “low score”. Count each mode. A mode with one example says little; a mode with many examples is a pattern.

3. Find the step the errors come from

Each failure mode points at the earliest step that can change it. Cheapest first: training (nothing regenerates), then the train set, then the teacher, then the test set (every earlier score becomes stale).

What the predictions show	Likely cause	What to change
The teacher is right; the errors have no pattern and the gap to the teacher is small	capacity or too little training	Model training: the student, epochs, learning rate
The teacher is right; the student is wrong on a slice the training data barely covers	coverage	A top-up with a mutator for the slice
The teacher is right; the student is wrong the same way everywhere (format, a misread rule)	the training data teaches it wrong	Generation instructions or a stronger teacher (Synthetic data generation); relabeling when the relabeled rows carry the mistake (Relabeling)
The teacher is wrong where the student is wrong	the teacher cannot do the task	Teacher evaluation with another teacher, then the train set
The judge marks correct answers bad, or wrong answers good	the measure is wrong	The judge instructions or model; every teacher evaluation runs again and every earlier student score is stale
The reference answer itself looks wrong, or the teacher and the student are wrong in the same way	wrong test labels	Correct the rows (Changing the test set)
The test set has no rows for a failure you see in production	the test set cannot measure it	Add rows from traces that show it (Changing the test set)

Download the dataset the model trained on (distil dataset download) before blaming the student: a failure mode can be looked for in the rows it learned from.

4. Change one setting

Changing one setting per iteration shows what that setting did. Several training runs in parallel (different students or epochs) still count as one change to training.

• Model training: the student (a larger one has more capacity, at a higher serving cost), the epochs and learning rate, the LoRA rank, the evaluation length. Nothing regenerates.
• The train set: a top-up on the dataset your best model was trained on is the default: new mutators or instructions, and train_generation_target set to the rows to add. When the synthetic rows were bad, generate again on the parent of the bad run (the relabeled dataset, which keeps the relabeled rows). When the relabeled rows carry the mistake, relabel again on the test-set dataset, then generate.
• The teacher: another teacher is checked by teacher evaluation first; then the train set is built again on the test-set dataset.

task_description stays matched to the prompt your production system runs (Job description), so it is not one of the settings an iteration changes.

Changing the test set

The locked test set is in every dataset after it, and the traces that could extend it are only in the datasets before the expands used them. So a change goes through an upload that keeps the train rows:

• Wrong reference answers. Download the dataset your best model was trained on, correct the rows in test.jsonl, and upload the directory with distil dataset create --data. The train rows come along; the upload has no parent on the platform, so note the jump yourself.
• Rows the test set lacks. The same upload, with traces.jsonl replaced by traces that show the failure (new ones, or leftovers downloaded from the root dataset), then Building the test set on it. Relabeling draws a fresh sample of the traces, so it adds rows; it never corrects existing ones.

Either way the test rows changed: every earlier score is stale. Re-run the teacher evaluations, then train your best settings on the new dataset to get a comparable score before changing anything else. No command scores an existing model on new rows.

5. Run

Run the step you changed and every step after it, through training. The change is an override on a dataset already in the chain (How the platform works), except when the iteration uploads a new dataset. Keep a ledger: one row per iteration with the hypothesis, the change, the dataset trained on, the SLM and its score, and whether the change was kept. A change that didn’t help is reverted: the next iteration starts from the best iteration’s settings and dataset, not from the last one’s.

The agent skill runs this loop for you and decides what to change in each iteration.

The next iteration from new traffic

Once the student serves traffic, the serving endpoint’s records are the next iteration’s traces. They hold the student’s or the fallback’s answers, which aren’t labels: the test-set and train-set stages relabel them with your chosen teacher. If your own production answers are the ground truth, upload them as labeled rows instead. To retrain on the records:

1. Create a traces object from the endpoint, or download, filter and convert the records (Endpoint records to traces). Filter on metadata.source for the answers the next model should learn from.
2. Create a new dataset from the new traces with the current test.jsonl and your best dataset’s train.jsonl included (distil dataset create --traces --test --train; download the traces object first on the direct route and keep observation_format: langfuse). The test set stays the same, and the train set is topped up rather than rebuilt. The old dataset’s leftover traces aren’t carried.
3. Keep the test set as it is, or extend it with the new traces (Building the test set). Extending makes every earlier score stale; it is the right call when the failures you see in production aren’t in the test set.
4. Teacher evaluation only when the test set changed; with the same rows, the teacher and the baseline scores still hold. Then the train set and training as before. With the same test set, the new model is read against the previous one.