Fine-Tune, RAG, or Prompt: Which Should You Use?
Pick by failure mode, not by preference. If the model lacks facts, use retrieval. If it knows the facts but behaves wrong (wrong format, wrong tool, wrong length), fine-tune. If you can’t yet describe what correct looks like, stay on prompts until you can.
Which failure are you actually trying to fix?
Each technique changes something different about the system. Match the row to your symptom.
| Symptom | What is missing | Technique | What it changes |
|---|---|---|---|
| Model invents facts about your documents | Knowledge at inference time | RAG | The prompt gains retrieved passages |
| Model knows the facts but formats badly | Learned behaviour | Fine-tuning | The model weights |
| Model calls the wrong tool | Learned behaviour | Fine-tuning | The model weights |
| Output is nearly right, needs a nudge | Instruction clarity | Prompting | The instruction text |
| You can’t articulate the rule | A task definition | Prompting first | Your own understanding |
| Facts change hourly | Fresh knowledge | RAG | The retrieval index |
The distinction between the first two rows is the one that trips teams up most. Ovadia and colleagues tested it directly and found that unsupervised fine-tuning is a poor way to inject new facts. RAG outperformed it both on knowledge seen during training and on entirely new knowledge. Fine-tuning teaches behaviour reliably; it teaches arbitrary facts badly. RAG vs fine-tuning takes that split apart.
When is prompting enough?
Prompting is enough while the task is still moving. It costs minutes, it’s reversible, and it forces you to write down what you want, which is the artefact every later step depends on.
Stay on prompting when:
- The task definition is still changing week to week
- Volume is low enough that per-request cost doesn’t appear on a bill
- You need one general assistant, not a specialist
- You have no held-out examples to measure against yet
There’s a ceiling though: prompts get longer as edge cases accumulate, and long prompts are both slower and more fragile. Prompting Science Report 1 found the effect of a given prompting technique on output quality is hard to anticipate, with politeness helping in some cases and hurting in others. Is fine-tuning worth it covers where that ceiling usually lands.
When is RAG the right answer?
RAG is right when the answer exists in a document you control and that document changes. Retrieval keeps the knowledge outside the weights, so updating a manual updates the system with no retraining.
Use RAG when your corpus is large, versioned, or refreshed often: product documentation, ticket histories, policy libraries. The original RAG paper frames this well, in that the parametric memory of the model is fixed at training time and a non-parametric index is what lets you swap facts in and out.
RAG has its own failure mode, and it isn’t retrieval quality. A small model handed three retrieved chunks (only one of which is relevant) often answers from the wrong one. That’s a behaviour problem, which is why the next section isn’t an alternative to this one.
When is fine-tuning the right answer?
Fine-tune when the behaviour is wrong and the behaviour is stable. Classification, tool calling, structured extraction, and grounded answering from retrieved text are all learned behaviours, and they’re exactly where published distil labs benchmarks show fine-tuned small models ranking first against frontier APIs on 4 of 8 datasets.
The prerequisites are short, but they’re firm:
- A task you can describe in a paragraph
- Roughly 20–100 labelled examples, per the data preparation docs
- A held-out test set you trust
- A teacher model that can already solve the task
That last one is a gate, not a formality. The platform runs teacher evaluation before training for this reason: if a large model can’t solve your task from a prompt, a small one won’t learn it from that model’s outputs.
Can you use more than one?
Yes, and the RAG-plus-fine-tuning combination is the common production shape rather than an exotic one. Retrieval supplies the facts; fine-tuning teaches the model to answer strictly from them and ignore distractors. The question-answering-open-book task exists for precisely this, and the RAG tutorial walks the full path. Can you combine RAG and fine-tuning covers the mechanics.
Prompting also doesn’t disappear after fine-tuning. The job description you write for a training run is a prompt; it just gets compiled into weights instead of resent on every request.
What do teams get wrong here?
The most expensive mistake is fine-tuning to fix a knowledge problem. Teams fine-tune on a corpus hoping the model absorbs it, get a model that’s confidently wrong in a new way, and conclude fine-tuning doesn’t work. Fine-tuning worked as designed; it just taught the wrong thing.
Three more, in rough order of how often they cost time:
- Skipping the eval set. Without held-out examples you can’t tell whether any of the three techniques helped. Build it before you build anything else, following a test set that catches real failures.
- Fine-tuning a task nobody has defined. If two people on the team label the same example differently, no amount of training fixes that. When does distillation fail lists the symptoms.
- Reaching for retrieval when the answer is in the prompt already. Adding a vector store to a task with one page of context adds latency and a new failure surface for nothing.
Start with the prompt, since it’s the cheapest way to find out whether the task is well-formed. Add retrieval when the facts live outside the model, and fine-tune when the behaviour rather than the knowledge is what’s wrong. Whichever of the three you land on, build the held-out test set first.