How We Benchmarked 12 Small Language Models
We fine-tuned 12 small language models on 8 tasks with identical hyperparameters and a single teacher, then ranked them per benchmark and averaged the ranks. Qwen3-4B-Instruct-2507 came first after fine-tuning at an average rank of 2.25, and matched or exceeded its 120B teacher on 7 of 8 benchmarks.
What was the headline result?
That fine-tuning matters more than base model choice. The differences between families are real before training and narrow sharply after it.
Three findings came out of the full benchmark, and they answer different questions:
| Question | Winner | Average rank |
|---|---|---|
| Best after fine-tuning | Qwen3-4B-Instruct-2507 | 2.25 (±1.03) |
| Most tunable (largest base-to-tuned gain) | Llama-3.2-1B-Instruct | 3.44 (±1.31) |
| Best before fine-tuning | Qwen3-8B | 1.75 (±0.72) |
Note that no model wins twice. The model with the best starting point isn’t the model with the best finish, and neither is the model that gains the most. Which of these three questions you care about depends entirely on whether you’re going to fine-tune.
Which model ranked best after fine-tuning?
The Qwen3 family, taking 4 of the top 6 places. Qwen3-4B-Instruct-2507 outranked the larger Qwen3-8B, which is the counterintuitive part.
| Model | Average rank | 95% CI |
|---|---|---|
| Qwen3-4B-Instruct-2507 | 2.25 | ±1.03 |
| Qwen3-8B | 2.75 | ±1.37 |
| Llama-3.1-8B-Instruct | 4.00 | ±1.42 |
| Qwen3-1.7B | 4.44 | ±1.60 |
| Llama-3.2-3B-Instruct | 4.56 | ±1.73 |
| Qwen3-0.6B | 5.11 | ±1.86 |
The confidence intervals overlap heavily. Qwen3-4B and Qwen3-8B are separated by 0.5 ranks with intervals of ±1.03 and ±1.37, so the ordering between those two is a lean rather than a verdict, and the top group is best read as a group.
Which model gained the most from fine-tuning?
The smallest ones. Tunability was measured as fine-tuned score minus base score, and the ranking inverts the size hierarchy.
| Model | Average rank | 95% CI |
|---|---|---|
| Llama-3.2-1B-Instruct | 3.44 | ±1.31 |
| Llama-3.2-3B-Instruct | 4.67 | ±1.93 |
| Qwen3-0.6B | 4.78 | ±1.78 |
| SmolLM2-1.7B-Instruct | 5.00 | ±1.46 |
| gemma-3-270m-it | 5.00 | ±2.77 |
The 8B models rank near the bottom for tunability, not because they’re worse but because they start strong and have less headroom. The practical reading, if you’re constrained to under 2B parameters: Llama-3.2-1B and Qwen3-0.6B close most of the gap. Which SLM fits in 4GB of VRAM and what size model do you need take that further.
Can the student match its teacher?
On 7 of 8 benchmarks, yes. This part of the study is reported with its actual teacher, GPT-OSS-120B, because changing that would falsify the comparison.
| Benchmark | Teacher | Qwen3-4B fine-tuned | Qwen3-4B base | Δ vs teacher |
|---|---|---|---|---|
| TREC | 0.89 | 0.93 | 0.51 | +0.03 |
| Banking77 | 0.92 | 0.89 | 0.87 | −0.03 |
| Docs | 0.82 | 0.84 | 0.64 | +0.02 |
| Ecommerce | 0.88 | 0.90 | 0.75 | +0.03 |
| HotpotQA | 0.93 | 0.93 | 0.88 | +0.00 |
| Mental Health | 0.81 | 0.82 | 0.78 | +0.01 |
| Roman Empire QA | 0.75 | 0.80 | 0.65 | +0.05 |
| SQuAD 2.0 | 0.52 | 0.71 | 0.26 | +0.19 |
That’s six wins, one tie, and one shortfall inside the margin of error. The SQuAD 2.0 result is the outlier worth understanding: closed-book QA asks the model to answer without context, and fine-tuning embeds domain knowledge in the weights more effectively than prompting can. Can a small model beat its teacher covers the mechanism.
How was it measured?
Twelve models across four families: Qwen3 (8B, 4B-Instruct-2507, 1.7B, 0.6B, with thinking disabled for parity), Llama (3.1-8B, 3.2-3B, 3.2-1B), SmolLM2 (1.7B, 135M), Gemma 3 (1b-it, 270m-it) and granite-3.3-8b-instruct.
Eight benchmarks: TREC, Banking77, Ecommerce and Mental Health for classification, a document-understanding set, and HotpotQA, Roman Empire QA and SQuAD 2.0 for question answering.
Each model was measured twice: few-shot base performance with prompting alone, and performance after training on roughly 10,000 synthetic examples per benchmark generated by the GPT-OSS-120B teacher. All models used identical hyperparameters: 4 epochs, learning rate 5e-5, linear scheduler, LoRA rank 64. Evaluation ran on held-out test sets never seen during training or synthetic data generation.
Results were aggregated by ranking models on each benchmark individually and averaging those ranks, with 95% confidence intervals as error bars. Ranking rather than averaging raw scores avoids one high-variance benchmark dominating the aggregate, a reasonable choice that discards information about how large each gap was.
What would change these numbers?
Four things, and the write-up is candid about the first two.
- More runs. Results were averaged over a limited number of iterations, and the stated plan is to expand that to tighten the intervals. The overlapping intervals above are the direct consequence.
- More models. The SLM landscape moves fast; the study names Qwen3.5, Phi-4 and Mistral variants as pending additions.
- More task types. Eight benchmarks covering classification, document understanding and QA. No summarisation, no code generation, no multi-turn dialogue, so nothing here speaks to multi-turn tool calling.
- A different teacher. Everything above used GPT-OSS-120B. A later distil labs study measured five teachers on a shared held-out set and ranked GLM-5 first at 0.835 against GPT-OSS-120B’s 0.765; which teacher model should you pick has the full table. That would likely lift the fine-tuned scores rather than change the model ordering, but it hasn’t been re-run.
For related evidence on whether a strong base model predicts a strong fine-tuned one, see does base model accuracy predict fine-tuned performance and best small language model for fine-tuning.