GPU Requirements and Cost for SLM Inference
Less GPU than you think. A fine-tuned 4B student measured by distil labs occupied 7.6 GiB while sustaining 222 requests per second on a single H100, a card with 80 GB. The memory was never the constraint; throughput on prompt-heavy work was what justified the hardware.
What GPU do you need to serve a fine-tuned SLM?
Whatever holds the weights plus headroom, which for most students in the catalog is modest. The supported models catalog tops out around 9B parameters, and the smallest entries are 135M.
The rule of thumb is two bytes per parameter at BF16, plus room for KV cache, activations and runtime overhead. A 1B student needs roughly 2 GB of weights; a 9B student needs roughly 18 GB. Which SLM fits in 4GB of VRAM does the full table at each precision.
What that means in practice: sub-1B students run on laptops and edge devices, mid-range students fit comfortably on a single mainstream accelerator, and only the 8–9B end needs a serious data-centre card at full precision.
How much memory does each size class need?
Weight storage is parameter count times bytes per parameter, nothing else. These are computed from the catalog, not measured.
| Student size | Example from catalog | BF16 weights | 4-bit weights |
|---|---|---|---|
| 135M–350M | SmolLM2 135M, LFM2.5 350M | 0.27–0.70 GB | 0.07–0.18 GB |
| 0.6B–1.2B | Qwen3 0.6B, LFM2.5 1.2B | 1.2–2.4 GB | 0.30–0.60 GB |
| 1.7B–3B | Qwen3 1.7B, Llama 3.2 3B | 3.4–6.0 GB | 0.85–1.5 GB |
| 4B | Qwen3 4B, Gemma 3 4B | 8.0 GB | 2.0 GB |
| 8B–9B | Qwen3 8B, Qwen3.5 9B | 16–18 GB | 4.0–4.5 GB |
Weights are a floor, not a total. KV cache grows with sequence length and batch size, which is why the 4B student that computes to 8.0 GB of weights measured 7.6 GiB in a real serving snapshot. Those are different units at a different moment, and the gap between the two closes as context grows.
What throughput does one GPU actually deliver?
Enough that volume is rarely the reason to add hardware. From the inference tax benchmark, serving a 4B Text2SQL student through vLLM on one H100 node:
| Metric | Measured |
|---|---|
| Max sustained RPS | 222 |
| p50 / p95 / p99 latency | 390ms / 640ms / 870ms |
| GPU memory | 7.6 GiB |
| Requests per day at ceiling | over 19 million |
One card, over 19 million requests a day. The benchmark notes these tasks are prefill-heavy and decode-light, which is exactly the profile where the H100’s compute advantage pays off even though 80 GB is far more memory than the model needs.
Quantisation moves both columns: brief FP8 experiments in the same study gave about 15% more throughput with 44% less memory and no measurable accuracy loss.
How does a GPU bill become a per-request cost?
Divide the hourly rate by the requests you actually serve in that hour. That’s the entire calculation, and it’s why self-hosted inference has no fixed price.
The benchmark used a single H100 node at about $2.40 per hour. At the 222 RPS ceiling that’s roughly $3 per million requests. At 10% of that ceiling, the same hardware bill spread over a tenth of the traffic gives roughly 10x the per-request cost. Hence the pricing page guidance that a dedicated GPU makes sense above roughly 30% utilisation.
Hourly rates vary widely by provider, region and commitment. Check Google Cloud or AWS for current on-demand rates rather than reusing the figure above, which is one benchmark’s assumption from early 2026.
When do you need more than one GPU?
For availability and burst headroom, usually before you need it for capacity.
- Redundancy. One card is a single point of failure regardless of how much spare throughput it has.
- Traffic spikes. Knowunity’s production traffic runs above 130 RPS with bursts, which is why we serve hosted models on autoscaling serverless GPU infrastructure that scales to zero rather than on fixed instances.
- Multiple models. Several specialists usually beat one generalist, and each needs somewhere to live.
Adding hardware for raw capacity comes last, because a smaller student is often the cheaper answer. Dropping from 4B to 0.6B on a task that supports it changes the memory and throughput picture entirely. What size model do you need covers how to tell, and hidden costs of running your own models covers what the GPU line item leaves out.