Base Model vs Fine-Tuned Model: How to Compare Them

Score the base model on the held-out test set first, train, then read the two numbers side by side with the teacher’s score between them. The gap is what fine-tuning bought. Skipping the first measurement is the most common reason a training run can’t be justified afterwards.

What you need before you start

Three things, all of them settled before you train.

Requirement	Detail
A held-out test set	The same file for both measurements, no exceptions
A base model score	Taken before training, on that file
Identical scoring	Same metric, same judge model, same few-shot count

If you’re training from production traces the platform does the first two for you: the upload-traces pipeline evaluates the original model that generated the traces on the test set as part of processing. For a hand-curated dataset you take the base measurement yourself.

Step 1: Score the base model on the test set

For a trace-derived seed dataset, the number is already there once processing finishes:

distil seed-dataset status <seed-dataset-id>
distil seed-dataset metrics --output json <seed-dataset-id> | jq .base_model_performance

distil seed-dataset metrics reports the base model’s performance on the generated or supplied test set, under base_model_performance. Poll seed-dataset status first. The metrics field reads null until processing has succeeded.

Write the number down somewhere outside the platform. You’re going to quote it in a comparison months later, and you want the metric and judge model recorded next to it.

Step 2: Train the student on the same seed dataset

Nothing here should change the data. Generation writes the training set from that seed dataset, and training reads what generation produced.

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

distil slm status <slm-id>

Resist the temptation to improve the test set between the two measurements. A test set edited after you saw the base score is no longer a fair comparator, and neither number means what you think it does. If the test set is wrong, fix it and re-take the base measurement.

Step 3: Read the two numbers with the teacher between them

Three scores, not two. The training docs frame a successful run as the student landing “reasonably close to the teacher model (typically within one standard deviation)”, and the base score tells you how far it travelled to get there.

Pattern	Reading
Base low, student ≈ teacher	The expected result. Fine-tuning did the work
Base low, student well below teacher	Under-trained, or the synthetic data missed the test distribution
Base already ≈ teacher	The base model could do the task. Check whether you need to train at all
Student below base	Something is wrong. See why did my fine-tuned model get worse

The third row is a real outcome and it’s worth taking seriously before spending a run. Is fine-tuning worth it and does base model accuracy predict fine-tuned performance both address it directly.

Step 4: Diff the predictions, not just the scores

Aggregate scores tell you whether to ship. Per-example predictions tell you what changed.

distil seed-dataset download-traces-predictions <seed-dataset-id> --file-name base.jsonl
distil slm download-predictions <slm-id> --file-name tuned.jsonl

Both files are JSON Lines. Join them on the input and split the result into four buckets: both correct, both wrong, base wrong and tuned right, base right and tuned wrong. That last bucket is worth an hour of your time. A fine-tuned model that has lost capabilities it previously had is exhibiting catastrophic forgetting, and no aggregate score will surface it.

What the published gaps look like

For calibration, these are real base-to-fine-tuned deltas from distil labs benchmarks. All were measured on held-out sets never seen during training or synthetic data generation.

From our 12-model benchmark, Qwen3-4B-Instruct-2507 against a GPT-OSS-120B teacher:

Benchmark	Base	Fine-tuned	Teacher
TREC	0.51	0.93	0.89
Docs	0.64	0.84	0.82
Ecommerce	0.75	0.90	0.88
HotpotQA	0.88	0.93	0.93
Mental Health	0.78	0.82	0.81
Roman Empire QA	0.65	0.80	0.75
SQuAD 2.0	0.26	0.71	0.52
Banking77	0.87	0.89	0.92

From the platform benchmark, a Llama3 3B student prompted versus trained: PII redaction 0.54 to 0.87, HotpotQA 0.80 to 0.95, pizza tool calling 0.0 to 0.70, git tool calling 0.03 to 0.95.

Two patterns are worth extracting. Tool calling shows the largest gaps because a prompted small model frequently can’t produce a schema-valid call at all, so a base score near zero is normal rather than a measurement error. And the gap shrinks where the base model was already competent: Banking77 moved two points, SQuAD 2.0 moved forty-five.

Making sure the comparison is fair

Five things that quietly invalidate a base-versus-tuned comparison:

• Different few-shot counts. evaluation.num_few_shot_examples defaults to 1 and tuning.num_few_shot_examples_student defaults to 0. A base model prompted with examples and a student evaluated without them aren’t being asked the same question. The config reference has both.
• A test set that moved. Regenerating traces between runs produces a new test set. Compare within one seed dataset.
• A different judge. Switching llm_as_a_judge_model_name rescales everything, as LLM-as-judge evaluation explains.
• A gap inside the noise. Judge scores in our published work carry standard deviations up to 0.020, and reference-free ratings around ±0.03.
• A test set too small to resolve the gap. Check it against how big should your test set be before quoting the delta.

When all five hold, the base-to-tuned delta isolates the one variable you changed, which makes it the number worth quoting. Record the metric, the judge model and the test set size alongside it.