Job description

job_description.json defines your task in words. The teacher reads it when it’s evaluated and when it writes training data, and the judge reads it when it scores a prediction.

This page is the full field reference. For what to put in it when you’re preparing a particular kind of input, see Traces or Dataset overview.

Unknown fields are rejected at validation, so the fields below are the whole surface.

Fields by task type

Field	Question answering	Chat completion
task_description	required	required
tools	-	optional / required
llm_as_a_judge_instructions	optional	optional
synthetic_data_generation_instructions	optional	optional
trace_processing_instructions	optional	optional

The chat completion column covers chat-completion, where tools is optional, and chat-completion-agentic, where it’s required.

There’s no system-prompt field: task_description is rendered into the system prompt the model trains and serves with, so write it as one. With base.variable_system_prompt on, each row carries its own system prompt instead (System prompts).

What each field feeds

task_description

Goes into every teacher prompt: evaluation, generation and judging. It defines what a correct answer is.

Derive it from the system prompt of the system you’re replacing, with the same care: the output format with an example, the include and exclude rules, the edge cases. With base.variable_system_prompt on, write only what every application’s system prompt has in common (System prompts).

Examples in it come from your train split or are written for this purpose, never taken from test.jsonl. Keep them few and short: they become part of the system prompt the model trains and serves with.

Because it also feeds the judge, it defines what “correct” means everywhere at once, so keep it fixed once the test set is locked and scored. Before that, and whenever the teacher misses conventions your labels follow (format, label names, length), write those conventions into it: it is often the biggest lever on teacher quality. A change after the first teacher evaluation needs a new teacher evaluation, since earlier scores stop being comparable.

tools

A list of tool schemas in the OpenAI function-calling format, with unique names. chat-completion-agentic needs at least one tool. chat-completion may omit the field (or pass null or []) for a tool-free conversational model; data with tool calls and no declared tools fails validation (Chat completion).

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Look up the current weather for a city",
        "parameters": {
          "type": "object",
          "properties": { "city": { "type": "string" } },
          "required": ["city"]
        }
      }
    }
  ]
}

Parameter schemas can declare default values. The tool_call_equivalence metric treats an argument left at its default as equal to omitting it.

Every tool call in your training and test data validates against these schemas.

llm_as_a_judge_instructions

What the judge is told when it scores a prediction, for both judge metrics (Metrics) and for the RLVR reward (Configuration). State the criteria: what a correct answer must contain, and what to ignore, such as ordering, whitespace or paraphrasing.

The judge writes its score inside a <score> tag. It can give a pass/fail verdict or a graded number.

Pass/fail verdict (default)

Unless the instructions ask for something else, the judge answers <score>good</score> or <score>bad</score>, and the prediction scores 1 or 0. A shape that works:

Output ‘good’ if the prediction correctly completes the task, otherwise output ‘bad’.

followed by the criteria that decide it. Use this mode when an answer is either correct or not. The aggregated metric is then the fraction of predictions marked good.

Graded score

When the instructions ask for a number, the judge can give partial credit. Ask for a number from 0 to 1 and show the exact format:

Check the prediction against these four criteria: (1) … (2) … (3) … (4) …. Output the fraction of criteria met as a number between 0 and 1 inside the score tag, for example <score>0.75</score>.

Use this mode when an answer can be partly correct, for example when it is checked against several independent criteria. It also gives RLVR a better reward when the student passes few rollouts: with a pass/fail verdict most groups of rollouts all score 0, and RLVR has nothing to rank inside the group. The aggregated metric is then the mean graded score, which can’t be compared with a pass rate from earlier pass/fail runs.

How the score is read

• The judge must give exactly one <score> tag. A reply without a tag is read as a whole; a reply with several tags is unparseable.
• good, yes, true, correct, pass, valid and 1 read as 1. bad, no, false, incorrect, wrong, fail, invalid and 0 read as 0. Case doesn’t matter.
• Any other value is read as a number and clamped to the 0-1 range. A score of 8 from a 1-10 scale reads as 1, so always ask for 0-1. Percentages (75%) and fractions (3/4) are unparseable.
• An unparseable reply counts as 0 in evaluation. In RLVR, the rollout is left out of the reward.

Vague criteria make every number noisy, and teacher, base and tuned scores all come from this judge. The judge sees the full prefix, including the question.

synthetic_data_generation_instructions

Extra guidance injected into every generation prompt, and read during generation only. Use it to describe what the generated inputs should look like: formats, domains, register, noise. For a reasoning student, see Reasoning models.

Because it is read during generation and nowhere else, it changes the inputs without redefining the task. See Synthetic data generation.

trace_processing_instructions

Task-specific guidance added to the instructions the teacher follows when it relabels traces, into the test split or the train split. It does nothing in a dataset without traces.

Use it when the edits have to respect something unusual about your traces:

This is a live phone call; preserve the caller’s interruptions and any cut-off utterances verbatim.

See Relabeling.