Turn Production Traces Into Training Data

Export your logs to traces.jsonl, add a job description and a config, then run distil traces upload followed by distil seed-dataset create-from-traces. The platform deduplicates, filters, relabels and validates them into a train/test split you can fine-tune on. Training from traces is an experimental feature.

What you need

Three files, in one directory, plus an optional fourth.

File	Format	Required	Purpose
traces.jsonl	JSONL	Yes	One production trace per line
job_description.json	JSON	Yes	What the model should do
config.yaml	YAML	Yes	Task, student model, trace_processing parameters
test.jsonl	JSONL	No	Your own curated test set

If you omit test.jsonl, the platform builds one from your traces by running a slice of them through the same filtering and relabelling pipeline. Supplying your own is worth the effort, since it’s the only evaluation set you can fully vouch for.

Step 1: Export traces in a supported format

Each line of traces.jsonl is one conversation. The default observation_format is openai_messages: an object with a messages array following the OpenAI chat completion format, optionally with tools and response_format.

{"messages": [{"role": "system", "content": "You are a helpful banking assistant."}, {"role": "user", "content": "What is my account balance?"}, {"role": "assistant", "content": "Your current account balance is $1,234.56."}]}

If your observability stack is Langfuse, set observation_format: langfuse and export observation objects with id, input, and optionally output and metadata instead. Trace formats compared covers the trade-off between the two.

Multi-turn conversations are preserved in full. A single exchange is simply a two-turn conversation, so you don’t need to flatten anything.

Step 2: Write the job description

This is the normative half of the signal, and it defines correct independently of whatever your current system happens to do. Minimum viable version is a single field.

{
  "task_description": "You are a helpful banking assistant that answers customer questions about their accounts, transactions, and banking products."
}

Task types that need more get more: classes_description for classification, tools for tool calling. The per-task fields are in the data preparation guides, and writing a job description for synthetic data covers how to write one that actually produces good generations.

Step 3: Set the trace processing config

Pick the task, the student, and a teacher.

base:
  task: classification
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5

synthgen:
  teacher_temperature: 0.6

trace_processing:
  observation_format: openai_messages
  relabel: true

Two things to know. Reasoning teachers (the GLM, Kimi, MiniMax, DeepSeek and GPT OSS families) require synthgen.teacher_temperature between 0.5 and 0.7; anything outside that range is a validation error. And trace_processing has its own teacher_model_name, used for relevance filtering and for picking the best relabelled answer from the committee. Full parameter list in the config reference.

Step 4: Upload the trace files

distil traces upload --data ./traces
# Output: Prepared traces created. ID: <traces-id>

This only stores the files as a PreparedTraces resource. Nothing is processed yet, which is deliberate, since it lets you reprocess later without re-uploading. You can also pass --traces, --job-description, --config and --test individually instead of using directory mode.

Step 5: Process them into a seed dataset

distil seed-dataset create-from-traces <traces-id>
# Output: Processing started. Seed dataset ID: <seed-dataset-id>

This is where the pipeline runs: deduplication and train/test seed splitting, relevance and coherence scoring against your job description, committee relabelling of the seed conversations, and validation of the rewritten output. Leftover traces beyond the seed budget become unstructured context for generation.

Processing typically takes several minutes.

Step 6: Check the baseline before you train

distil seed-dataset status <seed-dataset-id>
distil seed-dataset metrics <seed-dataset-id>

Because the platform evaluates the model that produced your traces against the new test set, seed-dataset metrics gives you the number your fine-tuned model has to beat. That baseline is the most useful artefact of this whole step, and it exists before you spend anything on training.

For per-example detail, distil seed-dataset download-traces-predictions <seed-dataset-id> writes out the base model’s predictions as JSONL.

Step 7: Train on the processed data

The processed traces are already a seed dataset, so there’s nothing to download and nothing to upload again. Each remaining stage reads the id of the stage before it.

distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

Run teacher evaluation first, because it tells you whether the teacher can solve the task while changing your mind is still cheap. Generation and training are the two stages that cost real money.

Verifying it worked

Three checks, in order.

First, processing succeeded at all: distil seed-dataset status reports failure explicitly, and the pipeline errors rather than proceeding if it produced fewer than 20 examples. Second, the splits look sane: download them and read a few rows, checking that the relabelled assistant turns say what you’d have said. Third, the trained student beats the baseline from step 6 on the same test set.

If the student doesn’t beat the baseline, the problem is usually upstream of training. Why did my fine-tuned model get worse and when does distillation fail are the two places to look.

Reprocessing without re-uploading

Because staging and processing are separate, you can retry with different parameters against the same stored traces:

distil traces list
distil seed-dataset create-from-traces --config ./config.yaml <traces-id>

Each run produces a new seed dataset, so earlier attempts stay available for comparison. This is the cheapest way to test whether a looser min_relevance_score or a different relabelling committee helps. See how much traffic before traces are useful.