Train an SLM for Voice Assistant Command Routing

In voice, latency chooses the model. A cascaded assistant has roughly 500–800ms per turn before the conversation stops feeling natural, ASR and TTS have already spent most of it, and what remains sets the size of the model that routes commands. Train multi-turn-tool-calling-closed-book against that budget.

Design backwards from the latency budget

Conversational UX research places the threshold for natural-feeling interaction at 500–800ms end to end. Our voice assistant analysis measured the cascade against that ceiling:

Stage	Typical latency
ASR (speech to text)	~200ms
Brain (cloud LLM)	500ms–1.2s depending on provider
Brain (fine-tuned Qwen3-0.6B, self-hosted)	40–100ms local, ~200ms with network
TTS (text to speech)	~75ms

With a cloud brain the pipeline is over budget before TTS starts; the brain stage alone consumes over 70% of total processing time. With a locally served sub-1B student it fits with room to spare, which is consistent with the latency you can expect from an SLM on ordinary hardware. That’s the whole architectural argument, and it’s a latency argument, not a cost one (though the cost follows).

Fix your budget first, subtract ASR and TTS, and only then pick a student. Everything below assumes that ordering.

Step 1: Fix the output contract

The SLM must never generate user-facing text. It emits one structured tool call per turn (intent plus slots) and a deterministic orchestrator renders every spoken response from templates.

That constraint does three things: it bounds latency (no long generations), guarantees well-formed output (the schema is checked), and keeps brand voice out of the model’s hands. It also means a 0.6B model is sufficient, because it never has to write, only to classify and fill slots.

Step 2: Write the tool schema for the operations you actually have

One function per backend operation, with the slots as typed parameters and enum wherever the value set is closed. The banking assistant in the reference implementation covers 14 operations: balance checks, transfers, card cancellation, fraud reporting and the rest.

{
  "type": "function",
  "function": {
    "name": "transfer_funds",
    "description": "Move money between two of the customer's own accounts",
    "parameters": {
      "type": "object",
      "properties": {
        "amount": {"type": "number", "minimum": 0.01},
        "from_account": {"type": "string", "enum": ["checking", "savings"]},
        "to_account": {"type": "string", "enum": ["checking", "savings"]}
      },
      "required": ["amount", "from_account", "to_account"]
    }
  }
}

Constrain hard. Every enum you write removes a class of ASR mis-transcription the model could otherwise route into.

Step 3: Seed conversations with real speech artefacts

Voice input isn’t typed input. Seed data written from a product spec misses the disfluencies real callers produce.

Include, deliberately: filler words and self-corrections (“um I need to like cancel my uh debit card”), slots arriving across turns (“I want to transfer money” → “200 dollars from checking to savings”), and mid-conversation intent changes (“actually, what’s my checking balance first?”). About 50 example conversations covering the workflow is enough to start, which sits in the usual range for how many examples you actually need.

Each JSONL line holds the whole conversation in one messages array, ending with the assistant call to be learned. The multi-turn data preparation guide has the format; note that arguments is a real JSON object and each assistant turn carries exactly one call.

Step 4: Train with a tool-calling-capable teacher

Your conversation files become a seed dataset, and each later stage names the id of the one before it.

distil seed-dataset create --data ./data
# Output: Upload successful. Seed dataset ID: <seed-dataset-id>

distil teacher-evaluation create-from-seed-dataset <seed-dataset-id>
# Output: Teacher evaluation started. Teacher Evaluation ID: <teacher-evaluation-id>

distil teacher-evaluation status <teacher-evaluation-id>

distil training-dataset create-from-seed-dataset <seed-dataset-id>
# Output: Synthetic data generation started. Training Dataset ID: <training-dataset-id>

distil slm create-from-training-dataset <training-dataset-id>
# Output: Training started. SLM ID: <slm-id>

distil slm status <slm-id>

base:
  task: multi-turn-tool-calling-closed-book
  student_model_name: Qwen3-0.6B
  teacher_model_name: zai.glm-5
synthgen:
  teacher_temperature: 0.6

Both constraints from the supported models catalog bite here. The student must come from the Qwen3, Qwen3.5, Llama 3, LFM2/LFM2.5, FunctionGemma, or Gemma 4 families. Qwen3-0.6B qualifies, Gemma 3 doesn’t. The teacher must be ticked for tool calling; zai.glm-5 is, and as a reasoning teacher it needs teacher_temperature in 0.5–0.7. For the wider comparison see which teacher model you should pick.

Step 5: Measure the brain stage in isolation

Score accuracy with tool_call_equivalence, then measure latency separately with ASR and TTS removed. A blended end-to-end number hides which stage regressed.

Accuracy compounds across turns, so a per-call figure understates the risk: at 90.9% per turn, a three-turn conversation completes correctly about 75% of the time. On the banking taxonomy a fine-tuned Qwen3-0.6B reached 90.9% against its 120B teacher’s 87.5%, with the untuned base model at 48.7%. Those runs used GPT-oss-120B as teacher. Two other students have been measured on the same banking task: LFM2.5 350M at 95.9% and FunctionGemma 270M at 90.86%. See multi-turn tool calling explained for why the multiplication matters.

Step 6: Deploy behind a deterministic orchestrator

distil slm download --destination ./model <slm-id>

slm download writes the weights, the config and a generated model_client.py. Serve them on the same host as the orchestrator, because the budget in step 1 assumed a local hop rather than a network one.

The orchestrator owns everything the model doesn’t: checking whether required slots are filled, asking a templated clarifying question when they aren’t, executing the call, and rendering the response. That division is what makes a sub-1B model production-viable. The residual errors land as “could you repeat the amount?” rather than as a wrong transfer. The VoiceTeller reference implementation is a working example of the pattern.

When a voice SLM isn’t the answer

When the assistant has to hold open-ended conversation. Everything above depends on a bounded set of operations with typed slots; open-ended generation still favours larger models, and no amount of fine-tuning changes that. It’s one of the clearer cases of when not to use a small language model.

Also reconsider if your operations are still changing weekly, because each change is a retrain. And if some turns are genuinely hard, a cascade that defers to a larger model keeps the small model on the fast path. For student sizing under a hardware ceiling, see which SLM fits in 4GB of VRAM.