Alternatives to Training a Small Language Model

When to fine-tune, when to use retrieval, when to just write a better prompt, and when a task-specific small model is the wrong tool entirely.

Training a task-specific model isn’t always the right move, and a page arguing otherwise wouldn’t be worth reading. This cluster covers the honest comparisons, including the cases where you should do something else.

The three-way choice

Most teams are really choosing between prompting a frontier model, retrieval, and training a small one. They solve different problems, and the confusion comes from the fact that all three can produce a working demo.

Prompting changes the instructions. It is the fastest to iterate on and requires no infrastructure, which is why almost everything starts here. It stops being the right answer when per-request cost becomes material, when latency matters, when outputs need to be consistent rather than prompt-sensitive, or when the data cannot leave your network.

Retrieval changes what the model sees, by fetching relevant context at query time. It is the right answer when the knowledge changes frequently, or when you need to show which source justified an answer. A fine-tuned model cannot point at a document; retrieval can.

Training changes the weights. It is the right answer for a well-defined, high-volume task where you want a small, fast, self-contained model with predictable behaviour.

These are not mutually exclusive. Fine-tuning a small model to handle retrieved context well is a common and effective combination.

When not to train

Worth stating directly:

  • The task is not well defined. If you cannot describe what a correct output looks like, you cannot generate training data for it or measure whether it worked.
  • Requirements change weekly. Retraining has a cycle time. If the specification moves faster than that, prompting is more honest.
  • Volume is low. The economics only work at sustained throughput. A few thousand requests a month does not justify the operational overhead.
  • You need broad general capability. A task-specific model is task-specific. If users can ask anything, you want a general model.
  • No teacher can solve it. If the frontier models fail at your task, distillation has nothing to transfer.

Routing, agents, and the “smaller pieces” question

Two adjacent patterns come up often. One is routing, sending easy requests to a small model and hard ones to a large one. The other is agent frameworks that orchestrate a general model across many tools.

Both are reasonable, and both are sometimes solving a problem that a single well-trained model removes. A small model can be trained to know when to defer, which collapses the router into the model itself. A model trained on your specific tool schema can be more reliable at calling them than a general model reading tool descriptions at runtime.

Where this cluster goes

The articles below cover fine-tuning against RAG and prompting directly, combining retrieval with training, alternatives to hosted fine-tuning APIs, open versus proprietary models, build versus buy, and when a small language model is the wrong choice.

Agent Frameworks vs a Fine-Tuned Tool-Calling Model

A framework wires tools to a model and manages state; it doesn't make the model pick the right tool. Which layer your failures actually live in, and why most systems need both.

Alternatives to the OpenAI Fine-Tuning API

The four categories of alternative, compared on where the model runs and what you own afterwards: self-hosted training libraries, managed open-weight platforms, cloud provider tuning services, and distillation platforms.

Distillation vs Pruning

Pruning deletes parts of a trained network and keeps what survives; distillation trains a fresh small model on a large one's behaviour. Why published pruning pipelines end with a distillation step anyway.

Model Routing vs a Single Specialised Model

A router is a separate component that decides which model handles a request. The alternative is training the decision into the small model itself. Here's what each costs to own.

Open Source vs Proprietary Models for Production

The decision is about control, not quality: who can serve the weights, what the licence permits, where your data travels, and who decides when the model changes underneath you.

Prompt Engineering vs Fine-Tuning

Prompting steers a model's existing behaviour at inference time; fine-tuning changes the weights. What that mechanical difference means for consistency, cost per request, and the ceiling each one hits.

RAG vs Fine-Tuning: Which One Fixes Your Problem?

RAG changes what the model can see; fine-tuning changes what the model does. Side-by-side on mechanism, update cost, and latency, with the measured evidence for each.

Build vs Buy: Training Your Own SLM

Building means owning synthetic data generation, hyperparameter search and evaluation yourself. A stage-by-stage breakdown of what each path costs and the cases where building is clearly right.

Do You Actually Need a Bigger Model?

Sometimes yes: broad reasoning and free-form generation still favour scale. But on bounded tasks the published rankings invert after fine-tuning, and this guide separates the two cases.

Fine-Tune, RAG, or Prompt: Which Should You Use?

Three techniques that fix three different problems: missing facts, wrong behaviour, and an under-specified task. A criteria table for telling them apart before you commit engineering time.

When Not to Use a Small Language Model

Five conditions that should make you walk away: an undefined task, weekly-changing requirements, low volume, a need for broad capability, and a teacher that can't solve it either.

Can You Combine RAG and Fine-Tuning?

Yes. You fine-tune the generator to answer strictly from retrieved passages, including deliberately irrelevant ones. What the training data looks like and where the combination breaks.