← All learn articles

Few-Shot Fine-Tuning: Train a Model with 10 Examples

Few-Shot Fine-Tuning: Train a Model with 10 Examples

Large language models are impressive few-shot learners. You give them a handful of examples in the prompt and they figure out the pattern. At scale that breaks down: prompts get expensive, latency climbs, and you’re still sending data to a third-party API.

Few-shot fine-tuning takes the opposite approach. Instead of stuffing examples into every prompt, you bake them directly into a small language model’s weights. The result is a compact, task-specific model that runs locally, responds in milliseconds, and doesn’t need examples at inference time.

What is few-shot fine-tuning?

Few-shot fine-tuning is the process of adapting a pre-trained language model using a very small labeled dataset, often between 5 and 50 examples. Unlike traditional fine-tuning that assumes hundreds or thousands of training samples, few-shot fine-tuning uses the knowledge already embedded in the base model and nudges it toward your specific task with minimal data.

This approach works especially well with modern small language models (SLMs) like Llama 3.2 1B, Qwen3 0.6B, and Gemma 3 1B, which have been pre-trained on broad, diverse corpora. A worked run at the low end of that example count is fine-tuning a model with 20 examples.

Few-shot fine-tuning vs in-context learning

In-Context Learning Few-Shot Fine-Tuning
Examples needed at inference Yes (in every prompt) No
Latency Higher (longer prompts) Lower (no examples in prompt)
Cost per request Higher (more tokens) Lower (smaller model, shorter prompts)
Consistency Variable High
Privacy Data sent to API Runs locally
Setup effort Minimal Requires training step

In-context learning is great for prototyping, but few-shot fine-tuning is better for production. Once you’ve validated that a task is solvable with a few examples, fine-tuning locks in that performance permanently. That’s the same trade as prompt engineering vs fine-tuning: a prompt has to be re-obeyed on every request, while trained behaviour is simply how the model responds.

How few-shot fine-tuning works with synthetic data

Synthetic data generation is what makes few-shot fine-tuning work reliably:

  1. Start with seed examples. Provide as few as 10 labeled examples that represent your task.
  2. Generate synthetic training data. A teacher model (like GLM-5) uses your seed examples to generate hundreds or thousands of diverse, high-quality training samples.
  3. Fine-tune the student model. A small language model is fine-tuned on the synthetic dataset using LoRA or full fine-tuning.
  4. Evaluate. The fine-tuned student is benchmarked against the teacher to verify quality. Clearing the teacher is one of the four comparisons that decide whether a fine-tuned model is good enough to ship.

This is the core workflow behind distil labs: you bring a prompt and a few examples, and the platform handles synthetic data generation and fine-tuning automatically.

When does few-shot fine-tuning outperform prompting?

Few-shot fine-tuning tends to win in these scenarios:

  • Structured output tasks. Classification, NER, information extraction, and tool calling benefit from the consistency that fine-tuning provides.
  • High-volume inference. When you’re making thousands of predictions per hour, the cost savings from a smaller model compound quickly.
  • Latency-sensitive applications. A 1B parameter model responds 10–50x faster than a 70B model behind an API. Measured end to end, the latency you can expect from an SLM runs from tens of milliseconds to a few hundred.
  • Privacy-constrained environments. Healthcare, finance, and government use cases where data can’t leave your infrastructure.
  • Edge deployment. Running models on devices, on-prem servers, or in air-gapped environments.

Getting started

The fastest way to try few-shot fine-tuning is with the distil labs CLI:

curl -fsSL https://cli-assets.distillabs.ai/install.sh | sh
distil signup                                          # or distil auth

distil seed-dataset create --data ./my-data-dir
distil training-dataset create-from-seed-dataset <seed-dataset-id>
distil slm create-from-training-dataset <training-dataset-id>

You provide your labeled examples in JSONL format, and the platform handles teacher evaluation, synthetic data generation, and student fine-tuning. The CLI reference covers every command.

Where did this idea come from?

Bootstrapping a large training set from a handful of seeds was popularised by Self-Instruct, which showed a model could generate its own instruction-following data from a small seed pool, and by Stanford Alpaca, which turned 175 seed tasks into a 52,000-example dataset. Few-shot fine-tuning applies the same mechanic to a narrow production task rather than to general instruction following.

Further reading

Sources

Related

All Fine-tuning articles →