LoRA vs Full Fine-Tuning: When to Use What
Fine-tuning a language model means updating its weights so it performs better on your specific task. But you don’t always need to update every weight. That’s the core idea behind LoRA (Low-Rank Adaptation). Understanding when it’s enough, and when it isn’t, can save you significant time and compute.
What is full fine-tuning?
Full fine-tuning updates all of the model’s parameters during training. For a 1B-parameter model, that means all one billion weights are adjusted to fit your dataset.
Pros:
- Maximum flexibility, since the model can change its behaviour significantly
- Often delivers the best possible accuracy on a given task
- Well-understood, standard approach
Cons:
- High GPU memory requirements, since you need to store gradients and optimizer states for every parameter
- Slower training iterations
- Risk of catastrophic forgetting on general capabilities
- Produces a full-size model checkpoint for every experiment
What is LoRA?
LoRA freezes the original model weights and injects small trainable matrices into each transformer layer. Instead of updating a large weight matrix W directly, LoRA learns two small matrices A and B such that the update is ΔW = A × B. The rank of these matrices (the “r” in LoRA) controls how expressive the adaptation is.
The original LoRA paper showed this matches full fine-tuning quality on many tasks while training a tiny fraction of the parameters. QLoRA took it further by quantizing the frozen base model to 4 bits, cutting memory again without a meaningful accuracy cost. LoRA vs QLoRA covers when that extra step is worth the slower training.
Pros:
- Lower memory usage, often 5–10x less than full fine-tuning
- Faster training iterations
- Tiny adapter files (often < 100 MB) instead of full model checkpoints
- Easy to swap adapters for different tasks on the same base model
Cons:
- Slightly lower ceiling on task accuracy for complex tasks
- Choosing the right rank and alpha requires some experimentation
- Not every architecture benefits equally
When should you choose LoRA?
LoRA is the right default for most practical scenarios:
- Limited GPU budget. You can fine-tune a 3B model on a single consumer GPU with LoRA.
- Multiple tasks on one base model. Swap lightweight adapters instead of managing multiple full checkpoints.
- Rapid experimentation. Shorter training cycles mean faster iteration on data and hyperparameters.
- The task is well-scoped. Classification, NER, and structured extraction tasks rarely need full fine-tuning to reach production quality.
When should you choose full fine-tuning?
Full fine-tuning still makes sense in specific situations:
- The task requires deep behavioural change. For example, teaching a model a new output format or reasoning style that differs substantially from its pre-training.
- You have ample compute and data. If cost isn’t a constraint and you want to squeeze out every last point of accuracy.
- Very small models. For models under 500M parameters, the memory savings of LoRA are less meaningful, and full fine-tuning is straightforward.
How do they compare side by side?
| Factor | LoRA | Full Fine-Tuning |
|---|---|---|
| GPU memory (1B model) | ~6–8 GB | ~24+ GB |
| Training speed | Faster per step | Slower per step |
| Adapter size | 10–100 MB | Full model (2–16 GB) |
| Accuracy ceiling | Very close to full FT | Highest possible |
| Best for | Scoped tasks, fast iteration | Deep adaptation, small models |
Those memory figures are for training. Serving the result is a separate budget: how much VRAM a 1B, 3B or 8B model needs covers what the weights and KV cache cost at inference.
For most teams fine-tuning small language models for production tasks like classification, question answering, or tool calling, LoRA is the recommended starting point. You can always fall back to full fine-tuning if you hit an accuracy wall, and on well-scoped tasks that is rarely necessary.
If you’re earlier than this decision and still weighing whether to fine-tune at all, see is fine-tuning worth it. For the surrounding workflow, how to fine-tune a small language model walks through the full path.
How does distil labs handle this?
When you fine-tune a model with distil labs, LoRA is enabled by default. The platform automatically selects a sensible rank and alpha based on the model size and task type, so you don’t need to tune these hyperparameters yourself. If you need full fine-tuning, you can disable LoRA in your configuration, but we recommend trying LoRA first. The available knobs are documented in the config reference.