How Much Can You Save Replacing an LLM API?
Knowunity, an edtech company processing hundreds of millions of AI requests a month, cut inference cost by roughly 68% on selected classification tasks. Accuracy went up too, from 81% to 93%. That’s one customer’s measured outcome on one workload, not a general promise, but it’s a real production number rather than a projection.
How much did Knowunity save?
About 68%, measured on production traffic over several days rather than in a synthetic benchmark. The case study compares a distilled student against the proprietary small model it replaced on a subject-classification task where the label set changes by country.
| Model | Classification accuracy | Latency p50 / p95 / p99 | $/1M requests |
|---|---|---|---|
| Gemini 2.5 Flash Lite | 81% | 0.49 / 0.81 / 1.28s | $109 |
| Custom distil labs model | 93% | 0.27 / 0.59 / 0.66s | $35 |
Their CTO Lucas Hild put it as “roughly 68% for selected tasks without sacrificing quality”. The accuracy column shows quality moved the right way, not sideways. The teacher used for distillation was Qwen3-235B; the student is a sub-8B model. Read the numbers with their scope: selected tasks, at that company’s volume and utilisation.
Note the figure was revised. The case study originally reported 50% and was updated in February 2026 to 68% after an improved model and an optimised deployment. Savings aren’t a fixed property of the swap.
What savings are published elsewhere?
A wide range, because the baseline you’re replacing matters more than the model you replace it with.
| Study | Baseline | After | Reported gap |
|---|---|---|---|
| Knowunity | $109/1M requests | $35/1M requests | ~68% |
| Inference tax benchmark | $45–$6,241/1M requests | $3/1M requests | 10–100x |
| Voice assistant | $120–$1,300/1M turns | ~$5/1M turns | 25–260x |
Knowunity’s 68% is the smallest of these, because they were already on one of the cheapest cloud models available. Teams replacing a premium frontier model see multiples, teams replacing an efficient small API see percentages. All three are February–March 2026 snapshots of API pricing, which moves.
How was each figure measured?
Differently, which is why they shouldn’t be averaged.
- Knowunity: production usage over several days. Benchmarks ran at up to 10 requests-per-second bursts with an 8-second failure ceiling, deliberately below their real >130 RPS so cloud quotas didn’t distort the comparison; the benchmarking setup is public.
- Inference tax benchmark: measured API token usage over 3 runs per frontier model, against sustained vLLM throughput on one H100 node at about $2.40/hr.
- Voice assistant: modelled from an ~800-input / ~100-output token turn against published API rates.
The pricing asymmetry runs through all three: we bill GPU uptime, most API providers bill tokens. Comparing them requires assuming a utilisation level, and every figure above assumes a high one.
What determines where you land?
Four things, roughly in order of impact.
- What you’re replacing. The dominant variable. Coming off Claude Opus is a different arithmetic from coming off Gemini Flash Lite.
- Utilisation. Below roughly 30% a dedicated GPU stops being the cheaper option at all.
- Task shape. Structured tasks distil well. Free-form generation doesn’t. See when does distillation fail.
- Prompt length. Fine-tuning moves instructions into the weights, so long system prompts you stop sending are a saving on every request forever.
What would make your saving smaller?
Low volume, a broad task, or an already-cheap baseline, plus the costs that sit outside the per-request rate.
Training is a one-off you have to amortise, serving needs someone to own it, and a model that drifts needs retraining. None of that appears in a $/1M requests column. Hidden costs of running your own models covers the full ledger, and calculating the ROI of a task-specific model turns it into a break-even volume for your own workload. If the answer is that your task isn’t narrow enough, is fine-tuning worth it is the honest prior question.