What Is Question Answering as a Training Task?
Question answering is the distil labs task type for any input-to-text problem where the output is a targeted answer rather than a label or a function call. You declare task: question-answering, and the model learns to locate or generate the specific value asked for instead of paraphrasing the whole input.
How does the question answering task work?
Each training example is a two-turn conversation: a user turn holding the input and the question, an assistant turn holding the expected answer. Everything the model needs sits in that user turn. Plain question-answering has no separate context field and no retrieval step.
The question answering data preparation guide uses invoice extraction as the worked example, and the shape is instructive:
{"messages": [{"role": "user", "content": "Invoice #1234 from Acme Corp dated 2024-01-15. Total: $540. What is the total amount?"}, {"role": "assistant", "content": "$540"}]}The answer is $540, not a sentence about the invoice. That terseness is the point: you’re training the output contract as much as the extraction. If your downstream code expects a bare value, every seed answer should be a bare value.
This task type covers far more than literal questions. The docs frame it as the fit for transforming text “by addressing implicit questions about its content”, which is where summarisation, rewriting, and structured extraction land. The name describes the input-output shape, not the subject. Two worked recipes show the breadth: text-to-SQL emits a query and PII redaction emits a JSON blob, and both declare question-answering.
What goes in the job description?
One required field and two optional ones, and both of the optional ones matter more than the word “optional” suggests.
| Field | Required | What it does |
|---|---|---|
task_description |
yes | What the model should do and how to format output, including the fallback for missing values |
synthetic_data_generation_instructions |
no | What the input looks like, so the teacher can fabricate realistic new inputs |
llm_as_a_judge_instructions |
no | The rubric the judge applies when scoring predictions |
task_description is where you pin the output contract: “Return only the specific value asked for, without additional explanation. If the information is not found, respond with ‘Not found’.” Without a stated fallback, the model will invent one, and inventing is the failure mode you’re training away.
llm_as_a_judge_instructions is worth writing even though it’s optional, because it decides what counts as correct. A rubric that accepts semantic equivalence and a rubric that demands the exact reference string will grade the same model very differently.
Which question answering variant do you actually need?
One task type, question-answering, covers three shapes. What differs is what the user turn carries and where the knowledge comes from.
| Shape | base.task |
Where the knowledge comes from |
|---|---|---|
| Question answering over the input | question-answering |
The input text itself: a document, an invoice, a ticket |
| Open-book QA (RAG) | question-answering |
Passages your retriever supplies, placed inside the user turn with the question |
| Closed-book QA | question-answering |
The model’s weights, learned during training; the user turn carries only the question |
The distinction between the first two is where the text comes from, not how the row looks: in both, the text sits in the user turn. For RAG, build the rows the way your retriever presents results, distractor chunks included, so the model learns to answer from the right passage and ignore the rest. Open-book vs closed-book QA covers the choice between retrieval and knowledge in the weights.
How is a question answering model scored?
With three metrics that disagree on purpose. The metrics reference recommends LLM-as-a-judge as the headline number.
- LLM-as-a-judge. A large model grades the prediction against the reference and the rubric. Best when many phrasings are correct; the research behind the approach is surveyed in this arXiv paper.
- Exact match. 1 if the strings match, 0 otherwise. Honest and harsh, useful when there’s one correct phrasing.
- ROUGE-L. Longest-common-subsequence overlap. Rewards reuse of reference wording and favours longer answers.
Read them together. Low exact match with a high judge score means the answers are right but paraphrased, which is a formatting problem rather than an accuracy problem. All three low means the task itself is under-specified. Revisit task_description before you revisit the model.
Related terms
Teacher evaluation runs a large model over your test set before training to check the task is solvable at all; see teacher evaluation. Unstructured data is in-domain text that steers synthetic generation toward realistic inputs. Mutators (synthgen.mutators in the config file) push the teacher to generate different kinds of question rather than a thousand variations of the easiest one. For related reading, what is a small language model covers the students, and when does distillation fail covers what goes wrong when a QA task is too broad.