What Does It Cost to Run a Small Language Model?
Roughly $3 per million requests. That’s the figure distil labs measured for fine-tuned students between 0.6B and 8B parameters, served on one H100 node at about $2.40 per hour. You’re buying GPU uptime, not tokens, so the per-request number falls as you push more traffic through the same card.
What does a fine-tuned SLM cost per million requests?
Three dollars, at full utilisation on a single GPU. The inference tax benchmark put fine-tuned students against ten frontier APIs across eight datasets and reported cost alongside accuracy rank, where rank 1.0 means first place on every task.
| Model | $/1M requests | Avg rank |
|---|---|---|
| Fine-tuned SLMs (0.6B–8B) | $3 | 3.2 |
| GPT-5 nano | $45 | 9.2 |
| Gemini 2.5 Flash Lite | $75 | 7.4 |
| Grok 4.1 Fast | $160 | 7.8 |
| GPT-5 mini | $222 | 8.4 |
| Gemini 2.5 Flash | $313 | 3.5 |
| Claude Haiku 4.5 | $1,331 | 6.0 |
| Claude Opus 4.6 | $6,241 | 2.5 |
The nearest model on quality, Gemini 2.5 Flash at rank 3.5, cost over 100x more per million requests than the fine-tuned students that outranked it. Frontier prices are a February 2026 snapshot. Check OpenAI, Anthropic and Google for current rates before reusing them.
What does it cost per token instead?
Around $0.04 per million tokens for a 4B student on hosted inference, and $0.08 for a 9B. The pricing page quotes those as effective blended rates (three input tokens per output token) at sustained load on one H100.
Per-request and per-token are different denominators for the same GPU bill. Which one is honest for you depends on your payload: a classification call with a short label output and a document-summarisation call can cost the same per request and differ tenfold per token.
How were these numbers measured?
By dividing GPU time by measured sustained throughput, not by estimating. The benchmark served every student through vLLM on one H100 node and recorded the ceiling.
| Metric | Text2SQL 4B student |
|---|---|
| Max sustained RPS | 222 |
| p50 / p95 / p99 latency | 390ms / 640ms / 870ms |
| GPU memory | 7.6 GiB |
| Requests per day at ceiling | over 19 million |
At 222 requests per second one card clears more than 19 million requests a day, which is where the $3 comes from. Note the memory column: a Qwen3-4B student used 7.6 GiB of an 80 GB card. The H100 was chosen for prefill throughput on prompt-heavy work, not because the weights needed the room. See which SLM fits in 4GB of VRAM for the weight arithmetic.
What changes the number?
Utilisation, mostly. Everything else is second order.
- Utilisation. A GPU you rent and don’t saturate still bills. The benchmark reports full utilisation and notes the conclusions survive a pessimistic 10% assumption; the pricing page puts the threshold for a dedicated GPU at roughly 30% utilisation.
- Task shape. Tool-calling workloads got 2–4x fewer requests per dollar than classification or question answering in the same study, because tool schemas inflate the prompt.
- Student size. A 9B student is quoted at double the 4B rate per token. Picking one is covered in what size model do you need.
- Quantisation. Brief FP8 experiments in the benchmark gave about 15% more throughput and 44% less memory with no measurable accuracy loss. Distillation vs quantization explains why the two compose.
When does this stop being cheap?
When traffic is low, or when the task isn’t narrow. Both failure modes are real and neither is a pricing problem.
Below roughly 30% utilisation a dedicated GPU is worse value than paying per token, because you’re renting idle silicon. And the benchmark’s own per-dataset breakdown shows fine-tuned students ranking 5th and 6th on docstring generation and Banking77, tasks needing broad world knowledge or fine-grained label distinctions. A cheap model that’s wrong isn’t a saving.
Split the workload: fine-tune the structured, high-volume tasks and route open-ended work to an API. When does self-hosting beat an API works through the threshold, and is fine-tuning worth it covers the prior question of whether to train at all.