The Task Types for Training a Small Language Model
A task type is the training recipe you declare in config.yaml, and there are three: question answering, chat completion, and agentic chat completion. It sets the shape of your training rows, the metric you’re scored on, and which student models you can use.
What are the three task types?
They are the three recipes listed in the task selection guide, each with an exact string you put in base.task.
| Task type | base.task value |
What the model produces |
|---|---|---|
| Question answering | question-answering |
One answer for one input: an extraction, a transformation, a JSON document, or a label from a fixed set |
| Chat completion | chat-completion |
The next assistant turn of a conversation: text, tool calls, or both |
| Agentic chat completion | chat-completion-agentic |
Tool calls whose results come back to the model, ending in an answer |
The task type describes the shape of the input-output pair, not the subject matter. Summarisation, rewriting, structured extraction, classification and answering over a retrieved passage all land on question-answering, because each is one input and one answer. Anything that is a conversation, with or without tools, is one of the two chat completion types.
Three familiar jobs map onto these as follows:
- Classification is question answering whose answer is the label. The labels and their definitions go in
task_description, ending with “Answer with the label only.”, and one mutator with a value per label keeps the synthetic data spread over the labels. Classifying into labels has the recipe. - RAG is question answering with the retrieved passage inside the user message. There is no separate open-book type; see open-book vs closed-book QA for the trade-off between retrieval and knowledge in the weights.
- Tool calling is chat completion with tools declared in the job description. Pick
chat-completionwhen the model only emits the calls and the next message is from the user. Pickchat-completion-agenticwhen the model reads the tool results and decides what to do next.
How does the task type change your data files?
It changes what each JSONL line contains. Every task uses the same dataset directory (config.yaml, job_description.json, and the optional traces.jsonl, train.jsonl and test.jsonl), but the rows differ enough that a file prepared for one task won’t train another.
| Task type | Extra field in job_description.json |
Shape of a train.jsonl line |
|---|---|---|
| Question answering | none required | messages: one user turn, one assistant turn with the answer or label |
| Chat completion | tools (OpenAI function schemas), optional |
messages: user and assistant turns; an assistant turn carries content, tool_calls, or both |
| Agentic chat completion | tools, required |
messages: the full loop, including tool role messages that carry the results |
Tool calls in your rows use arguments as a real JSON object, while the schemas in job_description.json follow OpenAI’s function-calling format. Getting those two backwards is the most common data-prep failure on tasks with tools. The chat completion and agentic chat completion pages show complete rows.
Why does the task type restrict which models you can use?
Because emitting tool calls requires structured-output behaviour that not every base model has. Whenever the job description declares tools, the supported models catalog limits students to the Qwen3, Qwen3.5, Qwen3.6, Qwen3.8, Llama 3, LFM2/LFM2.5, FunctionGemma, and Gemma 4 families, and limits teachers to those marked tool-calling capable. A chat-completion job with no tools, and every question-answering job, can use any model in the catalog.
So pick the task before the model. Choosing Gemma 3 1B for a classifier and later deciding it should also route function calls means starting over with a different student. The restriction is enforced at validation time, not degraded gracefully.
How is each task type scored?
By a metric matched to the output shape, which is why the task type also fixes what “accuracy” means for you. Question answering is scored with binary exact match, ROUGE-L, and two LLM-as-a-judge scores; distil labs recommends the judge for free text because it credits a semantically correct answer with different wording, and binary for classification because the label is the whole answer. The research behind the judge approach is surveyed in this arXiv paper.
The chat completion types add the tool-call metrics: tool_call_equivalence, binary_tool_call, and staged_tool_call, which awards quarter-credit for one call on each side, the right function name, the right parameter keys, and an exact match. That staged breakdown is genuinely useful during development: a score of 0.5 means the model picked the right function and filled the arguments wrong, which is a different bug from a score of 0.25.
When do two task types both look right?
More often than you’d expect, and the tie-breaks are worth knowing.
- Classification versus free-text question answering. If the output is a closed set of labels, make the label the whole answer and score with
binary. Letting the model answer in prose throws away accuracy and makes the result harder to measure. - Question answering versus chat completion. If every request is independent and answered in one turn, use question answering even if your application wraps it in a chat UI. Chat completion is for conversations whose later turns depend on earlier ones.
- Chat completion versus agentic. If users say “actually, make that Tuesday” and the model has to act on what a tool returned, you need the agentic type. Training on conversations that stop at the tool call won’t get you there.
When you’re still unsure, which task type should you pick walks the decision row by row.
Related terms
A student model is the small model being fine-tuned; a teacher model generates and validates the synthetic training data. Both are explained in what is a student model and what is a teacher model. Teacher evaluation is the pre-training check that a large model can solve your task at all. Synthetic data generation is what expands your few dozen seed examples into thousands. See generating synthetic training data for how that step works.