Frontier accuracy for self-hosted conversational agents
distil labs builds the model that powers your conversational agent in hours, matching frontier models even on difficult conversations. On hardware you control, as weights you own.
30M+ people use models trained on distil labs today
No off-the-shelf model fits your conversational agent
Self-hosted open models are not accurate enough on your task. Frontier APIs are accurate, but you rent them: off-site, priced per token and changed on someone else's schedule. The model your agent needs has to be built, and in-house that takes a specialist team and months per agent. With distil labs, one engineer ships it in hours.
How it works
Four steps from the data you have to a model you own. The auto-research agent runs them; you review the numbers.
- 1
Start with the data you have
Your harness spec (system prompt, tools, schemas, policies), a few manual examples, and the production traces you are already cleared to use. Only the harness spec is required: no protected records, no new legal approval. Built for teams whose data cannot leave the building.
> Build a small model for our support agent from this repo: the harness spec in harness/, the 24 manual examples and the cleared traces in traces/. Go ahead. ⏺ Bash(distil dataset create --data support-agent/) ⎿ Upload successful. Dataset ID: ds-7f3a harness spec · 24 examples · 1,900 tracesYour agentLLMauto-research agentyour model v1new traffic · new base models - 2
An auto-research agent builds your model
It builds a test set from your harness, simulates conversations to generate training data including the rare cases, trains the model, evaluates it against your frontier baseline and repeats until parity. Hours of training, minutes of your time.
> Train a model that matches our frontier API on the test set. Three runs at most, keep a log of every iteration. Go ahead. ⏺ Bash(distil dataset relabel-traces-test ds-7f3a) ⎿ ds-8c21 · 312 test conversations ⏺ Bash(distil dataset generate-synthetic-data-train ds-8c21) ⎿ ds-9d04 · 8,400 simulated conversations, 23% off script ⏺ Bash(distil slm create-from-dataset ds-9d04) ⎿ slm-2b7e · SFT, then RL · llm-as-a-judge 0.86 vs frontier 0.91 ⏺ Run 1 misses topic switches. Run 2: more of them, 4 epochs. ⎿ slm-4c19 · llm-as-a-judge 0.91 vs frontier 0.91 ✓ parity- 1Relabel traces
- 2Simulate conversations
- 3Post-train (SFT, then RL)
- 4Evaluate vs the frontier
- repeat until parity
- 3
Deploy and evaluate
Deploy the model on distil labs inference, in your VPC or on-premise, and point your agent at it, with the frontier API as fallback for the conversations you still want to rent. Score it on your own evals and on live traffic, side by side with the frontier API, and keep it when it matches.
> Deploy slm-4c19 behind our support endpoint, frontier as fallback. ⏺ Bash(distil deployment create-from-slm slm-4c19) ⎿ Deployment started. Deployment ID: dep-5a1c ⏺ Bash(distil inference-endpoint create-from-deployment --name support --fallback-model "openai/gpt-5" dep-5a1c) ⎿ Endpoint Name: support-Qk3bZ1 Endpoint: https://inference.distillabs.ai/v1/chat/completionsYour agentLLMauto-research agentyour model v1new traffic · new base models - 4
Keep improving: new traffic, new base models
Once cleared, production conversations flow back in. The same auto-research agent reruns the loop on its own and ships the next version in about 80 minutes of training, whenever your agent changes, a weak spot shows up or a better open base model is released.
> Two weeks of production traffic are cleared. Retrain on them. ⏺ Bash(distil traces create-from-inference-endpoint support-Qk3bZ1) ⎿ tr-c4e9 · 12,300 conversations since v1 ⏺ Bash(distil dataset create-from-traces --config config.yaml --job-description job_description.json tr-c4e9) ⎿ ds-a310 ⏺ Bash(distil slm create-from-dataset ds-a310) ⎿ slm-6e02 · v2 · SFT, then RL · about 80 minutesYour agentLLMauto-research agentyour model v1new traffic · new base models
How it works
Step 1 of 4
Start with the data you have
Your harness spec (system prompt, tools, schemas, policies), a few manual examples, and the production traces you are already cleared to use. Only the harness spec is required: no protected records, no new legal approval. Built for teams whose data cannot leave the building.
> Build a small model for our support agent from this repo: the
harness spec in harness/, the 24 manual examples and the cleared
traces in traces/. Go ahead.
⏺ Bash(distil dataset create --data support-agent/)
⎿ Upload successful. Dataset ID: ds-7f3a
harness spec · 24 examples · 1,900 tracesStep 2 of 4
An auto-research agent builds your model
It builds a test set from your harness, simulates conversations to generate training data including the rare cases, trains the model, evaluates it against your frontier baseline and repeats until parity. Hours of training, minutes of your time.
> Train a model that matches our frontier API on the test set. Three
runs at most, keep a log of every iteration. Go ahead.
⏺ Bash(distil dataset relabel-traces-test ds-7f3a)
⎿ ds-8c21 · 312 test conversations
⏺ Bash(distil dataset generate-synthetic-data-train ds-8c21)
⎿ ds-9d04 · 8,400 simulated conversations, 23% off script
⏺ Bash(distil slm create-from-dataset ds-9d04)
⎿ slm-2b7e · SFT, then RL · llm-as-a-judge 0.86 vs frontier 0.91
⏺ Run 1 misses topic switches. Run 2: more of them, 4 epochs.
⎿ slm-4c19 · llm-as-a-judge 0.91 vs frontier 0.91 ✓ parityStep 3 of 4
Deploy and evaluate
Deploy the model on distil labs inference, in your VPC or on-premise, and point your agent at it, with the frontier API as fallback for the conversations you still want to rent. Score it on your own evals and on live traffic, side by side with the frontier API, and keep it when it matches.
> Deploy slm-4c19 behind our support endpoint, frontier as fallback.
⏺ Bash(distil deployment create-from-slm slm-4c19)
⎿ Deployment started. Deployment ID: dep-5a1c
⏺ Bash(distil inference-endpoint create-from-deployment --name support
--fallback-model "openai/gpt-5" dep-5a1c)
⎿ Endpoint Name: support-Qk3bZ1
Endpoint: https://inference.distillabs.ai/v1/chat/completionsStep 4 of 4
Keep improving: new traffic, new base models
Once cleared, production conversations flow back in. The same auto-research agent reruns the loop on its own and ships the next version in about 80 minutes of training, whenever your agent changes, a weak spot shows up or a better open base model is released.
> Two weeks of production traffic are cleared. Retrain on them.
⏺ Bash(distil traces create-from-inference-endpoint support-Qk3bZ1)
⎿ tr-c4e9 · 12,300 conversations since v1
⏺ Bash(distil dataset create-from-traces --config config.yaml
--job-description job_description.json tr-c4e9)
⎿ ds-a310
⏺ Bash(distil slm create-from-dataset ds-a310)
⎿ slm-6e02 · v2 · SFT, then RL · about 80 minutesWhat you get
Build your own model in hours, without post-training experience
An auto-research agent turns your harness and examples into curated training data and delivers the first model at frontier accuracy within hours, then retrains automatically when your agent changes or a better open base model is released. No model goes live until you approve it.
Nobody on your team needs to have post-trained a model before.
- Post-training in house, per agentmonths
- Your first model with distil labshours
- Each retrain after thatabout 80 minutes
Frontier accuracy on difficult conversations
Your model is as accurate as frontier models even on rare events such as users interrupting or switching topics, because we generate synthetic training conversations for the uncommon cases that production traffic rarely contains.
A fine-tune on call logs learns the calls that went well. Simulated conversations cover the ones that go off script, and a failure your evals flag is fixed in the next model.
Own the model, not a subscription to one
You own the weights, so the model runs on-premises or in your VPC under your security and compliance rules and changes only when you decide. It encodes how your agent works and gets better with every retrain.
Because it is 100 times smaller than a frontier model, it costs 20% of a frontier API.
What the alternatives cannot do
| Frontier model in your VPC | Self-hosted open model | distil labs | |
|---|---|---|---|
| First model in hours | rented, not built | months of ML time | hours of training |
| Accurate on rare conversations | at frontier price | generic model | matches the frontier |
| No protected data needed | nothing to train | tuning needs your traces | harness and a few examples |
| Yours to move and retrain | rented, vendor cadence | yours, retrain by hand | yours, retrained in 80 min |
Ship your first conversational agent model in hours
Bring your harness and the data you are cleared to use. We show you a model that matches the frontier API on your eval, including the difficult conversations, before any protected data is involved.
Wondering what happens to your harness? Security and data handling
What Our Customers Say
We needed a small model that could power our product on an IBM P11, entirely on-premises. distil labs’ fine-tuned models allowed us to ship a self-contained solution where the SLM and our graph platform coexist on the same hardware. For customers in regulated industries, this means AI-powered query generation with complete data privacy – nothing ever leaves their environment.
David J. Haglin
Co-Founder and CTO at Rocketgraph
The collaboration with distil labs shows what’s possible when you fine-tune small language models and deploy them on IBM Power. It is not just about the performance–it’s about delivering private, accurate AI that’s ready for enterprise scale.
Unnikrishnan Rajagopal
Director at IBM Power
The distil labs platform accelerated the release of our cybersecurity-specialized language model, KINDI, enabling faster iterations with greater confidence. As a result, we ship InovaGuard improvements sooner and continuously boost investigation accuracy with every release.
Samir Bennacer
Co-Founder and CTO at Octodet
From our blog
How Knowunity used distil labs to cut their LLM bill by 68%
Knowunity, an edtech startup processing hundreds of millions of AI requests monthly, used distil labs to train a custom small language model that cut inference costs by 68% while improving classification accuracy from 81% to 93%.
Why training on production traces fails (and what to do instead)
Training directly on production traces doesn't work as well as you'd expect. We tested across five scenarios and synthetic data from traces scores up to 26 percentage points higher in accuracy.
The 10x Inference Tax You Don't Have to Pay
Benchmarking fine-tuned small language models (0.6B-8B) against 10 frontier LLMs across 8 datasets shows that task-specific SLMs match or beat frontier models at 10-100x lower inference cost.