Train an SLM for Intent Detection

Intent detection is a classification task where the difficulty is concentrated in a handful of adjacent intent pairs. A model that gets the clear cases right and the boundaries wrong isn’t usable, so build the whole recipe around the boundaries.

Intents blur where your backend doesn’t

An intent taxonomy that reads cleanly on a whiteboard usually contains three or four pairs that no human can separate reliably. “Card not working” and “Card declined”. “Change my address” and “Update my details”. Users don’t phrase requests along your product’s seams.

The fix isn’t more examples. It’s deciding, once and in writing, which side of each boundary an ambiguous utterance falls on, and encoding that decision where both the teacher and your annotators can see it.

Scale matters here too. Intent sets are typically large: BANKING77, a standard benchmark for this task, has 77 intents in a single banking domain. At that width, adjacent-pair confusion is the dominant error mode and overall accuracy hides it.

Step 1: Derive intents from actions, not from utterances

Write down what your system can actually do: the API calls, the workflows, the handoffs. One intent per distinct downstream action. If two candidate intents trigger the same action with the same parameters, they’re one intent, however differently users phrase them.

This inverts the usual process, which starts from a cluster analysis of past messages and produces intents nobody’s backend can serve. Starting from actions gives you a taxonomy that’s mutually exclusive by construction, because two actions are either the same call or they aren’t.

Step 2: Mine the boundary utterances from real logs

Pull real user messages, not invented ones; turning production traces into training data is the same move applied to a whole pipeline. Then sort them into three piles per intent: unambiguous, ambiguous, and out of scope. The ambiguous pile is your seed set’s centre of gravity.

Pile	How many seeds	Why
Unambiguous	5–10 per intent	Establishes the intent’s core meaning
Ambiguous between two intents	3–5 per boundary pair	Teaches the model where the line is
Out of scope	20+ total	Prevents confident misrouting of nonsense

Twenty or so examples per intent is enough for the platform to expand from (the classification data preparation guide sets that expectation), but the composition matters far more than the total.

Step 3: Encode the boundary in the class descriptions

classes_description is read by the teacher during synthetic generation, so an adjudication rule written there propagates into thousands of generated examples.

{
  "task_description": "Classify the customer utterance into exactly one intent.",
  "classes_description": {
    "card_payment_declined": "A specific payment or purchase was refused. The customer references a transaction that failed. Use this even if they also say the card 'isn't working'.",
    "card_not_working": "The card fails generally: not read at terminals, physically damaged, not activated. No specific declined transaction is referenced.",
    "out_of_scope": "The utterance is not a request this assistant can serve: chit-chat, unrelated products, or unintelligible input."
  }
}

“Use this even if they also say the card ‘isn’t working’” is the adjudication rule. Every boundary pair deserves one sentence like it.

Step 4: Create the seed dataset, evaluate the teacher, train

Your data directory becomes a seed dataset. Teacher evaluation and generation both read its id, and training reads the training dataset generation produced.

distil seed-dataset create --data ./data
# Output: Upload successful. Seed dataset ID: <seed-dataset-id>

distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>

distil teacher-evaluation status <teacher-evaluation-id>

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

distil slm status <slm-id>

Your config.yaml:

base:
  task: classification
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5
evaluation:
  num_few_shot_examples: 1
synthgen:
  teacher_temperature: 0.6
  num_negative_exemplars_per_generation: 4

num_negative_exemplars_per_generation shows the teacher examples from other classes while it generates for the current one. Raising it above the default of 2 helps most on a wide intent set, because it makes the teacher generate examples that are deliberately distinguishable from their neighbours. The config reference lists the rest.

Step 5: Read the confusion pairs, not the average

Download the predictions and build a swap matrix.

distil slm download-predictions <slm-id>

Expect the errors to be concentrated. On our 12-model benchmark, a fine-tuned Qwen3-4B reached 0.89 on Banking77 against a teacher’s 0.92, the one benchmark of eight where the student fell short, and within margin of error. Compare that with TREC on the same run: 0.51 base, 0.93 tuned. A wide, semantically adjacent intent set is the hard end of classification, and the base model already scoring 0.87 on Banking77 shows how little headroom fine-tuning has to work with there. The method behind those figures is written up in how we benchmarked 12 small language models.

When a pair keeps swapping, you have three options in order of cost: sharpen the adjudication sentence, merge the two intents, or accept the confusion and disambiguate downstream with a clarifying question.

Step 6: Ship with an out-of-scope class and a deploy check

distil deployment create-from-slm <slm-id>
# Output: Deployment started. Deployment ID: <deployment-id>

distil deployment endpoint <deployment-id>

endpoint hands back the URL and the API key for a hosted, OpenAI-compatible endpoint. Test the out-of-scope class deliberately before you route anything. A classifier with no escape hatch assigns your closest intent to every input, including “what’s the weather”, so send anything labelled out_of_scope to a fallback response rather than to an intent handler. The serving choices themselves are covered in how to deploy a fine-tuned small language model.

When intent detection should be tool calling instead

When the downstream action needs arguments. “Transfer £200 from checking to savings” carries an amount and two accounts; a label alone throws them away and you end up writing a second extraction step. Use tool calling, where the function name is the intent and the arguments are the slots, and see voice assistant command routing for that shape end to end. If your intents genuinely carry no parameters, classification stays the cheaper and more measurable choice, and what is text classification covers the task type.