Prompt Engineering vs Fine-Tuning
Prompt engineering steers behaviour the model already has, at inference time, through the text you send. Fine-tuning changes the weights so the behaviour is built in. The consequence: a prompt has to be resent and re-obeyed on every request, while a fine-tuned behaviour is simply how the model responds.
What is the mechanical difference?
Prompting is an inference-time technique. It can’t introduce a capability the model doesn’t already have, only make the model more likely to select one it does.
| Prompt engineering | Fine-tuning | |
|---|---|---|
| What it modifies | The input text | The model weights |
| When it applies | Every request | Once, then permanently |
| Introduces new behaviour | No, surfaces existing behaviour | Yes |
| Cost per request | Rises with prompt length | Falls with prompt length |
| Consistency across runs | Depends on sampling and phrasing | Higher, and temperature-controllable |
| Time to first result | Minutes | Hours |
| What you can hand a colleague | A string | A model |
The distil labs post on vibe-tuning frames the boundary usefully: prompt engineering guides the model’s existing knowledge through phrasing and structure, which means it can’t adjust weights to introduce knowledge the model was never trained on.
What can prompting not fix?
Three things, and recognising them saves weeks.
Behaviour that has to be identical every time. Strict JSON, an exact tool schema, a fixed label set. Prompt instructions are advisory; weights are not. This is why classification and tool calling are the archetypal fine-tuning tasks in task selection.
Unpredictable sensitivity to phrasing. Meincke and colleagues found the effect of a given prompting technique is hard to anticipate in advance, with politeness improving some responses and degrading others. You can measure your way past this, but you’re measuring a moving target.
The compounding cost of edge cases. Every handled exception adds tokens. A 2,000-token system prompt is 2,000 tokens on every request, forever, and it’s fragile in a way that’s hard to regression-test.
What prompting does fix, and fine-tuning doesn’t, is a task you haven’t defined yet. If you can’t write the instruction, you can’t label the data.
What does fine-tuning cost you in exchange?
It costs specialisation. A fine-tuned small model is good at its task and worse at everything else, which is the trade rather than a defect. Related risks are covered in what is catastrophic forgetting.
It also costs a data step. On a distillation platform that step is small, just a task description and roughly 20 to 100 examples, per data preparation. It isn’t zero though, and it needs a held-out test set you actually trust.
And it costs iteration latency. Changing a prompt is instant; changing a model is a training run. If your requirements move weekly, that latency dominates everything else. When not to use a small language model covers that case.
What does the measured evidence show?
Two independent lines point the same way for narrow, well-defined tasks.
Bucher and Martini found fine-tuned smaller LLMs significantly outperform zero-shot generative models on text classification, a result about the task category rather than about any one vendor.
The distil labs platform benchmarks make the same point with the base model held constant. Across the reported datasets, the same student model prompted versus fine-tuned differs sharply:
| Task | Base student (prompted) | Trained student (fine-tuned) |
|---|---|---|
| PII redaction | 0.54 | 0.87 |
| HotpotQA | 0.80 | 0.95 |
| Git tool calling | 0.03 | 0.95 |
| SQuAD 2.0 | 0.43 | 0.64 |
Figures from the platform benchmarking post, where the teacher was Llama 3 70B and the student Llama 3 3B. The tool-calling row is the clearest case: prompting a 3B model into schema-compliant function calls barely works at all, and the gap isn’t one you close with better wording.
Do you have to choose?
No, and the sequence matters more than the choice. Prompt first, since it’s how you discover whether the task is well-formed and what “correct” means. Collect the input-output pairs you liked. Fine-tune when prompt fragility, cost, or latency starts costing more than a training run would.
Fine-tuning also doesn’t remove prompting. The job description you write for a training run is a prompt; it’s just compiled into weights rather than resent every time. And a fine-tuned model still takes a system message at inference.
For the version of this decision that includes retrieval as a third option, see fine-tune, RAG, or prompt. For the shortest path from a prompt to a trained model, few-shot fine-tuning covers how far ten examples go.