What Is a Held-Out Test Set?

A held-out test set is data reserved from evaluation and never used to fit the model. In a distillation pipeline the definition has to be stricter than usual: the data must be unseen by training and by synthetic data generation, because the teacher generating your training set is a second path for the answers to leak in.

What makes a test set held out?

Two properties. It’s disjoint from everything that influenced the model’s weights, and it was reserved before anyone looked at it.

The conventional split has three parts, and distil labs uses all three under different names:

Split	Where it lives	What it’s for
Training	train.jsonl	Seed examples the teacher expands into synthetic data
Validation	tuning.train_eval_split, default 0.2	A slice of the training data used during the run to watch for overfitting
Test	test.jsonl	The held-out set every reported score is measured on

train_eval_split and test.jsonl are frequently confused, and they do different jobs. The validation slice is carved out of training data and the model’s training loop sees its distribution; the test set isn’t carved out of anything and is touched once. Reporting the validation number as your result is a quiet way to overstate a model. The data preparation overview lists both files as required for a reason.

How does contamination happen in a distillation pipeline?

Through the teacher, which is the path standard advice doesn’t cover. Ordinary fine-tuning has one contamination surface: examples appearing in both train and test. Distillation has three.

• Direct overlap. The same example in train.jsonl and test.jsonl. Easy to check, easy to fix.
• Generation overlap. The teacher generates roughly 10,000 synthetic examples from your seed data. If a test question was in the seed set, near-duplicates of it are now in the training data. This is why our benchmarks describe test sets as “never seen during training or synthetic data generation”. The second clause is doing real work.
• Filter-induced easiness. When the platform builds a test set from traces, that set passes through the same relevance and coherence filters as the training seeds. It isn’t contaminated in the strict sense, but it’s systematically easier than production traffic. Why your model passes eval but fails in production covers what that does to you.

Contamination isn’t an exotic failure. The GPT-3 paper devoted a section to measuring benchmark overlap in its own training corpus, and Dodge and colleagues found examples from standard NLP benchmarks sitting inside the C4 web corpus. If it happens at that scale with deliberate effort, it will happen in a pipeline where nobody checked.

Where does the held-out set come from on this platform?

One of two places, and they give you very different test sets.

You supply it. Put test.jsonl in your data directory for a minimal dataset, or pass --test when uploading traces. When you do, trace_processing.num_traces_as_testing_base is ignored and the platform generates nothing.

The platform builds it. If you upload traces without a test file, the trace pipeline selects traces, runs them through the same filtering and relabelling as everything else, and uses the result. Convenient, and it inherits every bias in the filter.

A worked example of doing it properly: in our traces benchmark, 34 conversations were reserved as a shared test set first, leaving 327 clean traces as the canonical training source. Every scenario in that study (including five deliberately corrupted ones) was then scored on the same reserved set. The reservation happened before the corruption, which is the only ordering that produces comparable numbers.

What breaks when the held-out set leaks?

Every downstream decision, silently. A contaminated test set doesn’t throw an error; it returns a high number.

• Teacher evaluation stops being a gate. If the teacher has effectively seen the answers, a passing score no longer tells you the task is solvable. What is teacher evaluation depends entirely on the test set being clean.
• The overfitting check inverts. The point of comparing training and held-out scores is that they diverge when the model memorises. With leakage they converge, and memorisation reads as generalisation. See what is overfitting in fine-tuning.
• Base-versus-tuned deltas inflate. The base model never saw the leaked data; the tuned model did. The gap you attribute to training is partly memorisation. Base model vs fine-tuned model comparison only works on clean data.
• Model selection goes wrong. Choosing between candidates on a contaminated set selects for whichever memorised hardest.

How to keep it clean

Four practices, roughly in order of effort.

1. Reserve before you process. Split the raw data first. Everything after that point (deduplication, filtering, relabelling, generation) happens to the training side only.
2. Draw few-shot examples from the training split only. Our platform benchmark states this explicitly for its teacher baseline: k-shot examples come from the training split, never test. A test example used as a prompt exemplar is contamination.
3. Let deduplication help. synthgen.validation_similarity_threshold defaults to 0.95 and removes generated data too similar to seed data. It’s aimed at generation diversity, but it also limits how closely synthetic examples can shadow seed items. See the config reference.
4. Hash and check. Before every reported number, confirm no exact or near-exact input appears on both sides. It takes a minute and it’s the only check that catches the case where someone helpfully added a failing test example to the training data.

Related: how big should your test set be for sizing, building a test set that catches real failures for composition, and is your fine-tuned model good enough for what to do with the score once it’s trustworthy.