The cost argument for small language models is straightforward at the surface and more interesting underneath. A model with a hundredth of the parameters costs dramatically less per token to run. The interesting part is when that saving is real, and when it is eaten by things the headline number ignores.
Where the saving comes from
Inference cost scales with the compute needed per token, which scales with model size. A 1B model serving a task that a frontier model was handling isn’t a marginal improvement in unit economics. It’s a different order of magnitude, and for a workload running millions of requests that gap is the entire business case.
The saving compounds where the task is narrow and high-volume: classification, extraction, routing, redaction. These are exactly the workloads where a frontier model is most overqualified, and where teams most often default to one anyway because it was the fastest thing to prototype with.
What the headline number leaves out
Self-hosting replaces a per-token bill with a fixed one. GPUs cost money whether or not requests arrive, so the break-even depends on utilisation, and a self-hosted deployment at 5% utilisation can be more expensive than the API it replaced.
There’s also engineering time (deployment, monitoring, updates, on-call) which is real even though it doesn’t appear on an invoice. And there’s the one-off cost of training itself, which is small relative to ongoing inference for any serious volume, but is not zero.
The honest framing is that self-hosting wins clearly at sustained volume, loses at low or spiky volume, and that the crossover is worth calculating with your actual numbers rather than assumed.
Latency has a price too
Slow inference costs money in ways that do not appear in a cost-per-token comparison. It costs conversion in interactive products, it costs GPU-hours in agent loops that make several sequential calls, and it forces architectural workarounds (caching layers, speculative execution, fallback paths) that carry their own engineering cost.
A model that is five times faster removes work from the rest of the system, and that saving is real even though it is harder to attribute.
Calculating it for your case
The inputs are: request volume, tokens in and out per request, current per-token API pricing, the hardware needed to serve your chosen student at your latency target, and the utilisation you can realistically expect. That is enough for a defensible estimate, and the calculation is usually decisive in one direction or the other rather than close.
Where teams have run this properly, the results have been substantial. One edtech customer cut inference cost by roughly 68% on selected classification tasks, and accuracy went up rather than down, from 81% to 93%. That’s one customer’s measured outcome on one workload, not a general promise.
Where this cluster goes
The articles below work through inference cost per million tokens, the self-host versus API break-even, what fine-tuning itself costs, GPU requirements, the hidden operational costs, and a worked ROI calculation.