Train an SLM for Support Ticket Triage

Triage is classification, a question-answering task whose answer is the queue name, and the model is only ever as good as the taxonomy it’s trained on. Spend your first day on the class definitions and your second on the CLI. A taxonomy two humans can’t apply consistently won’t produce a model that can.

The taxonomy is the model

Most triage projects fail before any training runs, because the queue names that grew organically in a ticketing system aren’t a taxonomy. They overlap, some are workflow states rather than categories, and several exist because one team lead created them.

Three properties make a taxonomy trainable:

• Mutually exclusive. Every ticket has exactly one correct queue. “Billing” and “Refunds” overlap; “Billing - refund request” and “Billing - invoice query” don’t.
• Decidable from the ticket text alone. If assigning a queue requires opening the customer’s account, the model can’t learn it from text and neither can the teacher.
• Balanced enough to seed. A class you can’t find twenty examples of is a class you should fold into its neighbour.

Run the cheap test first: take fifty resolved tickets, have two people label them independently, and measure how often they agree. Below roughly 90% agreement, fix the definitions rather than proceed. That disagreement is the ceiling on any model you train.

Step 1: Write the taxonomy as label definitions

job_description.json carries the whole taxonomy. The task_description lists every queue with a definition the teacher reads when fabricating synthetic tickets, so write each one for a new hire, not for a slide. End with the output rule.

{
  "task_description": "Assign each inbound support ticket to exactly one triage queue.\n\nQueues:\n- billing_refund_request: The customer asks for money back for a completed charge: refunds, chargebacks, duplicate payments. Not questions about what a charge was for.\n- billing_invoice_query: The customer asks what a charge was for, or wants a copy or correction of an invoice. No money is being requested back.\n- account_access: The customer cannot sign in: password resets, MFA lockouts, SSO failures, disabled accounts.\n\nAnswer with the queue name only."
}

Notice the negative clauses. “Not questions about what a charge was for” does more work than three extra examples, because it tells the teacher where the boundary runs. This text is also the system prompt the model trains and serves with, so it’s the only place the labels live. Classifying into labels has the full file spec.

Step 2: Seed the confusable pairs, not the obvious ones

Twenty examples per class is enough to start, in line with how many examples you actually need, but which twenty decides the outcome. Obvious tickets teach the model nothing it won’t infer from the class description. The tickets sitting on a boundary carry most of the value.

For each pair of classes that a human hesitated over during your agreement test, add two or three examples on each side. Keep the label strings byte-identical to the queue names in task_description; nothing checks this for you.

{"messages": [{"role": "user", "content": "I was charged twice for the same order in March, please sort this out"}, {"role": "assistant", "content": "billing_refund_request"}]}
{"messages": [{"role": "user", "content": "There's a £14 line on my invoice I don't recognise, what is it?"}, {"role": "assistant", "content": "billing_invoice_query"}]}

Add real tickets as traces.jsonl if you have them, one conversation per line and no labels needed. It costs nothing and it teaches the teacher how your customers actually write (abbreviations, product nicknames and all).

Step 3: Create the seed dataset, then check the teacher can apply your taxonomy

Teacher evaluation doubles as a taxonomy audit. It scores a large model on your held-out tickets using your class descriptions, before you spend a training run.

Your data directory becomes a seed dataset, and every later stage hangs off that one id.

distil seed-dataset create --data ./data
# Output: Upload successful. Seed dataset ID: <seed-dataset-id>

distil seed-dataset status <seed-dataset-id>

distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>

distil teacher-evaluation status <teacher-evaluation-id>

Your config.yaml:

base:
  task: question-answering
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5.3-low-thinking
synthgen:
  teacher_temperature: 0.6
  mutators:
    - name: queue
      description: which triage queue the ticket belongs to
      target_distribution: match_seed
      values:
        - "a ticket whose correct queue is billing_refund_request"
        - "a ticket whose correct queue is billing_invoice_query"
        - "a ticket whose correct queue is account_access"

zai.glm-5.3-low-thinking is a reasoning teacher, so the temperature must sit between 0.5 and 0.7. The mutator has one value per queue, so the teacher generates for every queue in turn instead of over-generating the easy ones; target_distribution: match_seed follows the queue mix of your seed tickets, and leaving it out gives an even split. Qwen3-1.7B is a reasonable default student for a taxonomy of a few dozen classes; drop to 0.6B if the queue count is small and latency matters.

If the teacher scores badly, download its mistakes and read them: distil teacher-evaluation download-predictions <teacher-evaluation-id>. Where the teacher confuses two classes, those two classes need either a sharper boundary or a merge.

Step 4: Train, then read the per-class errors

Training is two jobs, not one. The teacher writes the synthetic tickets into a training dataset first, and the student then trains on that.

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

distil slm status <slm-id>
distil slm download-predictions <slm-id>

The aggregate binary score, the exact match on the queue name, tells you very little here. Group the predictions by true label and look at what the errors have in common:

Error pattern	What it means	Fix
Two classes swapped symmetrically	The boundary is undefined	Add a negative clause to both definitions
One class absorbs its neighbours	Its definition is too broad	Narrow it, or merge the neighbours into it
One rare class is never predicted	Too few seeds, or it’s missing from the mutator	Add seeds; check the mutator has a value for it and count its rows in the generated data
An answer that isn’t a queue name	The output rule is weak	Make “Answer with the queue name only” the last line of task_description
Errors spread evenly, no pattern	Genuinely hard task	Try a larger student before touching the taxonomy

Fix the description, add four seed tickets on the boundary, retrain.

For scale: our email labelling model trained Qwen3-0.6B on 10 classes from 154 seed examples expanded to 10,000 synthetic ones, and moved from 38% for the base model to 93%, matching its teacher. That run used openai.gpt-oss-120b, the teacher it actually set.

Step 5: Deploy and route

distil deployment create-from-slm <slm-id>
# Output: Deployment started. Deployment ID: <deployment-id>

# once distil deployment status reports JOB_SUCCESS:
distil inference-endpoint create-from-deployment --name ticket-triage \
  --fallback-model "openai/gpt-4.1-mini" <deployment-id>

That puts a hosted, OpenAI-compatible inference endpoint in front of the deployment, and the fallback model answers whenever the deployment can’t. Link an API key to it with distil inference-endpoint link-api-key and call https://inference.distillabs.ai/v1 with the endpoint name as the model. To serve the model yourself instead, distil slm download --destination ./model <slm-id> writes the weights next to a generated model_client.py. Query it with that client, since it carries the trained system prompt. llama-cpp is one runtime under it: see local deployment for the choice, and running a fine-tuned SLM with llama.cpp for that one in detail.

Wire the label straight into your routing rules. Because the output is a bare label string, there’s nothing to parse, which is the practical reason to train a classifier rather than prompting a general model for a sentence.

When triage should not be one classifier

If your queue assignment depends on the customer’s plan tier or open order state, that’s a business rule, not a classification. Predict the topic from text and apply the rule in code. Mixing them produces a model that’s wrong whenever the account data changes.

If triage has to extract a value as well as pick a queue, you want tool calling instead. And if you’re choosing a student, Qwen3 1.7B vs Gemma 3 1B for classification covers the trade-off; what is text classification covers the task itself.