Relabeling and synthetic data

A dataset grows by four expand operations: relabel traces into the test or the train split, or generate synthetic rows into the test or the train split. Each one reads a dataset and writes a new one with rows added to one split and the traces it used removed; the parent is unchanged (The expand operations). This page is the mechanics of the two kinds of operation. What to aim for is on Building the test set and Building the train set.

The four commands share one shape. Each prints the new dataset’s ID, and takes --config and --job-description overrides and --smoke:

distil dataset relabel-traces-test <dataset-id>
distil dataset relabel-traces-train <dataset-id>
distil dataset generate-synthetic-data-test <dataset-id>
distil dataset generate-synthetic-data-train <dataset-id>

Poll the new dataset until it reads Success, and read its log when it fails:

distil dataset status <new-dataset-id>
distil dataset logs <new-dataset-id>

A smoke run is the same job with the count or the target forced to 128, on a datasets_from_datasets_smoke_post credit. Never continue from a smoke child: its config carries the smoke values. The full run, and every expand after it, goes on the smoke’s parent.

Relabeling

The job takes trace_processing.num_test_relabelled or num_train_relabelled traces from the dataset (shuffled and deduplicated first), filters them for relevance, has the teacher rewrite the assistant answers, validates the rows, drops any that duplicate the other split, and writes them into the split. The trace_processing section controls it (Config file). The settings to choose deliberately:

observation_format. Has to match the shape of your traces.jsonl (Traces).
teacher_model_name and relabelling_committee_models. Who rewrites the answers. It defaults to base.teacher_model_name; a non-empty committee list has several models propose a rewrite and the teacher pick. On the test split these are the reference answers, so they define what every score measures.
relabel (default true). With relabel: false the production answers pass through unreviewed, and your student learns to imitate the model it is meant to beat. If your production answers are the ground truth, don’t relabel them: upload them as train.jsonl and test.jsonl rows instead. When the teacher then disagrees with them, write the conventions your labels follow into task_description (Job description) rather than replacing the labels. If relabeled answers come back worse than the originals, change the teacher or use a committee instead.
relevance_filtering (default false). Set it true to have an LLM score each trace and drop the ones low on relevance or coherence. That costs an LLM pass over every trace, and min_relevance_score and min_coherence_score apply only when it is on.
trace_processing_instructions in the job description. Guidance for the rewrite only, for something unusual about your traces (Job description).
compress_job_description. Set it true when your task description is long enough to overwhelm the filtering model.

With base.enable_thinking: true, every relabeled assistant turn carries reasoning: reasoning found in the trace is kept when it is correct and leads to the answer, and missing or bad reasoning is written by the teacher. With it off, reasoning is removed from the traces (Reasoning models).

Three behaviors to expect:

The job needs the traces it is asked for. It fails when the dataset holds fewer distinct traces than the count, with the message Relabelling 200 traces into the test split needs that many distinct traces, and the dataset has 120. Lower the count or add traces. A smoke fails the same way below 128 traces: run without --smoke then.
There is no floor on the rows. Filtering, relabeling, validation and the overlap check each drop traces, and whatever survives is written, zero rows included. A count far below the traces used usually means the traces don’t match the task type, for example multi-turn conversations for a single-turn task.
The remaining traces are written back in openai_messages form, without their system prompts unless base.variable_system_prompt is on, whatever the parent’s observation format.

Synthetic data generation

The teacher generates rows from the rows already in the split (the examples), the job description, the mutators and the traces as context, then validation and deduplication keep the ones that pass. The target is synthgen.test_generation_target or train_generation_target; generation runs in batches of generation_iteration_size, so the result can land a little above it. What the job reads from the dataset:

The split’s rows are the examples. With an empty split the job runs zero-shot, and the in-context exemplar counts are lowered to the rows there are. No rows then fix the format or style, so read the output closely. A train split an earlier run split into per-turn rows (clean_training_targets or enable_thinking on a multi-turn task) is refused as examples: generate from the dataset before that run.
The other split is held out. New rows that duplicate it are dropped, so generating into test never leaks train rows, and the reverse.
The traces are the context when synthgen.use_traces_as_context is true (the default): T + min(T, 1000) of them for a target of T, or all that are left, and they leave the dataset. Below min(T / 4, 10) traces the job generates without context and leaves the traces alone. Count the traces before setting the target, because this run can empty the pool for every expand after it.

What controls the generated data

task_description defines what a correct answer is. It also feeds evaluation and the judge, so it stays fixed and matched to your production prompt. Three things control the data:

synthetic_data_generation_instructions in job_description.json: a constant added to every generation call that describes the inputs to write (formats, domains, register, noise). It changes every generated row the same way, so use it when the whole set is off in the same direction.
Mutators in synthgen.mutators: the dimensions the data is spread over. Each generation call samples one value per mutator, so they set the mix of the data rather than every row. Use them for coverage: slices that are missing, rare, or over-represented against production.
The synthgen config: how much is generated, whether the traces serve as context, how much survives validation and how much each call writes. These change the volume and cleanliness of the data, not what it is about.

With base.variable_system_prompt on, the teacher also writes a system message for every example. Steer what those say with a mutator (System prompts).

Mutators

synthgen.mutators holds one entry per dimension you want to vary:

synthgen:
  mutators:
    - name: complexity
      values: [simple, medium, complex]
    - name: topic
      values:
        - "about billing disputes: refunds, double charges, invoice errors"
        - "about account cancellation: downgrades and win-back attempts"
        - "about technical support: setup problems and integration errors"

None are applied by default: with no mutators, every example comes from the same prompt.

Field	Required	Notes
`name`	Yes	The dimension. Unique across mutators
`values`	No	The values, each a plain string. Leave it out to have the teacher detect them, which requires a `description`
`method`	No	`fixed` (default) samples from the target distribution. `adaptive` measures what survived validation and samples more of the values that fall behind
`target_distribution`	No	`uniform` (default), `match_seed` (the proportions the teacher finds in the split’s rows), or one weight per value, such as `[5, 3, 2]`
`description`	No	What the dimension means. Required when `values` is left out; read by the classifier with `adaptive` and `match_seed`
`auto_detect_sample_size`	No	Rows from the split the teacher reads when detecting values. Default `50`
`beta`	No	`adaptive` only, 0 to 1, default `0.9`. How far sampling may move away from the target

A value is one string. Put any detail after a colon and quote the whole value, so YAML reads a string: - "simple: minimal reasoning required" is a value, while - simple: minimal reasoning required is a mapping and is rejected.

Each mutator renders one line in the generation prompt. With the first example above, one call’s prompt can contain:

Important! Make sure to follow those guidelines:
Generated examples should be simple
Generated examples should be about billing disputes: refunds, double charges, invoice errors

Every line is Generated examples should be followed by the value, so phrase values to finish that sentence (about billing disputes, not billing).

Use one mutator per independent dimension, 1-2 mutators of 3-10 values each. To skew the mix, give one weight per value: values: [English, French] with target_distribution: [3, 1] asks for English in about three calls out of four.

Letting the teacher pick the values. If you know the dimension but not its values, leave values out and describe the dimension:

synthgen:
  mutators:
    - name: topic
      description: which part of the product the customer is asking about

The teacher proposes the values once, at the start of the run, from your job description and a sample of the split’s rows. The values are printed in the run log; copy them into values to edit them or to get the same values on every run.

Adaptive mutators. A fixed mutator never checks what came back: the teacher can ignore a directive, and validation can drop one value’s examples more than another’s. With method: adaptive, the teacher labels the surviving examples every synthgen.mutator_update_frequency batches (default 5), and sampling moves toward the values that fall behind. Each labeled example is one teacher call, so this makes generation slower. Always write the description for an adaptive mutator.

Tools. For chat completion with tools, add a mutator with one value per tool, so every tool is covered (Chat completion).

Classification. One mutator with a value per label spreads the data over the labels (Classifying into labels). Keep any other mutator away from the label: a value that hints at a severity or a category can lead the teacher to the wrong label, while style, length and phrasing are safe. Check the labels of the generated rows after the smoke run.

Configuring generation

The full table is in Config file. The ones to set deliberately:

validation_max_total_length. The character cap on one row. Set it to the longest combined length you expect: the default of 30,000 rejects longer uploaded rows and filters out longer generated ones.
generation_in_single_call and the exemplar counts. Lower them when rows are long; each multiplies the prompt and output size of a teacher call.
validation_similarity_threshold. Deduplication against the rows already in the split. Lower it if generation produces near-duplicates.
output_is_json: true. When answers must be valid JSON. Question answering only.
base.enable_thinking. For a reasoning student, set here; see Reasoning models.

LLM validation

Set synthgen.validation_instructions to have an LLM check each generated example against your own criteria. For example, to check how the assistant ends a call:

synthgen:
  validation_instructions: |
    Reject examples where the assistant hangs up without saying goodbye.

Write the checks you want in validation_instructions. They do not have to appear in the job description. The judge receives your instructions, the job description and the full generated example, including its system prompt, tool calls and images.

Choose the judge with evaluation.llm_as_a_judge_model_name; when unset, it uses base.teacher_model_name. The generation and validation models can differ.

The check runs only during synthetic generation, after structural and diversity filtering. It adds one judge call per surviving example, using the pipeline’s configured request parallelism. Rejected examples and invalid judgments are dropped without rewriting them. Relabeling, target cleanup and dataset loading are unchanged. Omit the field or set it to null to disable this check. An empty string is invalid.

Checking what came back

Download the new dataset and its parent, which is free, and diff the split to isolate the new rows: nothing in a row says whether it was uploaded, relabeled or generated.

distil dataset download -d output <new-dataset-id>
distil dataset download -d parent <dataset-id>
wc -l output/train.jsonl parent/train.jsonl

Compare the count against the target first. Falling short means validation filtered heavily, usually on length or format. A run that adds no valid rows for 8 iterations in a row stops there, and its logs list the most common validation errors of the last batch. Then check three things on the new rows:

Form. Each row parses, carries the required output format and respects the stated constraints. A malformed row is a defect whatever the rate. A handful of answers shows whether the teacher understood the task; a low rate of wrong labels from the teacher is tolerated by the student. For relabeled rows, read the trace and its row side by side: the answer should be correct, and better than the original where they differ.
Distribution. Compare the new rows against the examples and the traces along the dimensions that matter for your task: length, topic coverage, style and register, class balance. Watch for filtering that dropped whole categories.
Requested slices. Count each slice your mutators or job description asked for: a mutator value is a request and can produce nothing. A smoke makes about 32 mutator draws at the default generation_in_single_call of 4, so check this on the smoke only when the mutator grid has at most 16 cells; with more, check it on the full run.

If the data isn’t good

Problems in the data show up in training only as a score that stops improving, after a full training run. A new expand costs one credit, while a training run on bad data costs a training credit and the full training time. So fix it first, with a new run on the same parent:

Relabeled answers worse than the originals: a stronger teacher or a committee, or trace_processing_instructions for a quirk of the traces. Irrelevant or incoherent traces in the output: relevance filtering.
Generated rows that break a rule or the format again and again: describe that slice and its correct handling in synthetic_data_generation_instructions, or use a stronger teacher (Supported models). Scattered label errors are noise the student tolerates. For classification, check whether a mutator causes the mislabeling.
Rows to fix by hand: download the dataset, correct the rows, and upload the directory as a new dataset with distil dataset create --data <dir>. This spends a datasets_post credit; the upload has no parent on the platform.

Building the test set and Building the train set say when to run which operation, and with what target.