Writing a Job Description for Synthetic Data Generation
job_description.json is where you define what correct looks like. It’s a JSON file with a task_description plus task-specific fields, and it does more work than any other input: it steers generation, it’s the yardstick for relevance filtering, and it tells the judge how to score.
What a job description is and isn’t
It’s a specification, not a prompt. Both routes into the platform need one: the minimal dataset path and the trace upload path use the same file in the same format.
| It carries | It doesn’t carry |
|---|---|
| What the model should do | Few-shot examples (those go in train.jsonl) |
| The exact output format | Domain background (that goes in unstructured.jsonl) |
| Edge-case rules and tie-breakers | Model or hyperparameter choices (those go in config.yaml) |
| Task-specific schemas, classes, tools | Anything about your infrastructure |
The reason this file matters disproportionately is the split between signal types. Your examples and traces carry distributional signal: what inputs look like. The job description carries the normative signal of what a right answer is. Generation combines the two, and it’s the only place the second one lives. Why every path ends in synthetic data sets out the mechanism.
Step 1: Start from your existing prompt
If you’re already running an LLM feature, its system prompt is the fastest first draft. It already encodes the vocabulary, the constraints, and the edge cases you learned the hard way.
{
"task_description": "You are a helpful banking assistant that answers customer questions about their accounts, transactions, and banking products."
}Then edit it in one specific direction: a system prompt is written to steer a model at inference time, while a job description is written to define a task. Remove the politeness scaffolding and the “you are an expert” framing. Keep the rules.
Step 2: State the output format exactly
This is the highest-leverage sentence in the file, because format violations are what validators reject and what makes a model unusable downstream.
{
"task_description": "Extract the requested information from the provided invoice text. Return only the specific value asked for, without additional explanation. If the information is not found, respond with 'Not found'."
}Note the three things that short paragraph does: it says what to extract, it forbids commentary, and it specifies the behaviour for the failure case. Unspecified failure cases are where synthetic datasets go inconsistent: half the generated examples will apologise and half will return an empty string, and the student learns both.
Step 3: Add the task-specific fields
Each task type expects different additional fields, documented per task in the data preparation guides.
| Task | Additional fields |
|---|---|
question-answering |
none required |
classification |
classes_description |
tool-calling-closed-book |
tools |
multi-turn-tool-calling-closed-book |
tools |
Tool schemas use the OpenAI function-calling format. Get them exactly right, because the schema is what lets generation ignore wrong function names that appear in noisy traces. That mechanism is worth 25.9 percentage points under schema drift in our traces benchmark.
For classification, write classes_description so the boundaries between classes are explicit. “Complaint” and “Query” overlap in ways that only show up once a teacher has generated four thousand examples that straddle them.
Step 4: Describe the input, not only the output
synthetic_data_generation_instructions is easy to skip and expensive to skip. It tells the teacher what to invent. Unlike the fields in step 3 it isn’t task-specific: every task type accepts it, and it’s read during generation and nowhere else.
{
"task_description": "Extract the requested information from the provided invoice text. Return only the value asked for.",
"synthetic_data_generation_instructions": "Each input is one invoice text followed by a single question about that invoice. The invoice text carries an invoice number, a date, a vendor name, line items, a subtotal, tax and a total amount. Vary the vendors, the number of line items and the field the question asks for.",
"llm_as_a_judge_instructions": "Compare the predicted answer to the reference answer for the given question. Output 'good' if the prediction matches the reference value or is semantically equivalent, otherwise output 'bad'."
}Without it, generated inputs drift toward whatever the teacher considers a prototypical example of the domain. With it, they look like your data. This is the same lever WizardLM exploits when it evolves instructions along controlled axes rather than sampling freely, and the reason Stanford Alpaca needed a seed pool at all.
Step 5: Write the judge instructions
llm_as_a_judge_instructions is optional and usually worth adding. It defines what counts as a correct answer when correctness isn’t literal string equality, which is most of the time for question answering.
Be specific about tolerance. Is $540 equivalent to 540.00? Is a longer but correct answer good or bad? Whatever you decide becomes the metric you optimise against, so decide deliberately rather than letting a default judge decide for you. See metrics for how the scores are computed.
Step 6: Keep it short enough to filter with
On the trace path there’s a constraint that doesn’t exist elsewhere: the job description is used to score every seed trace for relevance. A very long description can overwhelm the filtering model.
The config has a documented escape hatch, trace_processing.compress_job_description, off by default, which has the teacher compress the description before relevance filtering. Turning it on is a workaround, but writing a tighter description is the actual fix.
Checking whether it’s good enough
Run teacher evaluation and read the result before training anything. A teacher that scores badly on your task with your description is telling you the description is ambiguous far more often than it’s telling you the task is hard.
Then read twenty generated examples. If they’re all subtly the same, your description is over-constrained. If a third of them are off-format, it’s under-specified. Either way the file is cheaper to change than a training run. Reprocess with distil seed-dataset create-from-traces --job-description ./job_description.json <traces-id> and compare. Turn production traces into training data and fine-tune with 20 examples cover the surrounding steps.