Benchmarks
Data-driven performance comparisons for small language models fine-tuned with distil labs.
When a small model needs to reason, and how long it should take
You can now train small models that reason on distil labs. Fine-tuned to answer directly, Qwen3.5-4B stops at 84 on invoice checks and 45 on flight connection checks; trained on the same data to reason first, it scores 98 and 95, around teachers 186x and 700x its size, in about 200 to 270 reasoning tokens.
The best mid-size model to fine-tune was the third-best model to prompt
We added NVIDIA's Nemotron 3.5 Lightning to our mid-size MoE fine-tuning benchmark and re-ran every model two to four times. It finishes first after fine-tuning, clearly beaten on only one of six tasks, despite ranking third before fine-tuning and scoring no better than a coin flip on binary classification untuned.
Does base-model accuracy predict fine-tuned performance: a mid-size MoE benchmark
Across four comparable mid-size MoE models on seven tasks, base-model rank does not predict fine-tuned rank: the lowest-ranking base (NVIDIA Nemotron-3-Nano-30B) gains the most and ties for the best, and once tuned it matches its 550B teacher on most tasks.
Why training on production traces fails (and what to do instead)
Training directly on production traces doesn't work as well as you'd expect. We tested across five scenarios and synthetic data from traces scores up to 26 percentage points higher in accuracy.
Fine-Tuning Liquid's LFM2.5: Accurate Tool Calling at 350M Parameters
Liquid AI's LFM2.5-350M reaches 96-98% tool call equivalence after fine-tuning with distil labs across three benchmarks, matching or exceeding a 120B teacher model while staying at 350M parameters.
What Small Language Model Is Best for Fine-Tuning
We benchmarked 15 small language models across 9 tasks to find the best base model for fine-tuning. Qwen3-8B ranks #1 overall. Liquid AI's LFM2 family is the most tunable. Fine-tuned Qwen3-4B matches a 120B+ teacher on 8 of 9 benchmarks.
The 10x Inference Tax You Don't Have to Pay
Benchmarking fine-tuned small language models (0.6B-8B) against 10 frontier LLMs across 8 datasets shows that task-specific SLMs match or beat frontier models at 10-100x lower inference cost.
When Does Reinforcement Learning Help Small Language Models?
A controlled experiment across 12 datasets reveals that adding RLVR after SFT consistently improves text generation tasks (+2.0pp) but provides no reliable benefit for structured tasks like classification and function calling.
We Benchmarked 12 Small Language Models Across 8 Tasks to Find the Best Base Model for Fine-Tuning
A systematic benchmark of 12 small language models across 8 tasks reveals Qwen3-4B as the best for fine-tuning, with fine-tuned models matching or exceeding a 120B+ teacher. Smaller models like Llama-3.2-1B show the highest tunability.