← All learn articles

Can You Combine RAG and Fine-Tuning?

Can You Combine RAG and Fine-Tuning?

Yes. The retrieval layer stays exactly as it is; you fine-tune the model that consumes its output, teaching it to answer strictly from the retrieved passage and to ignore passages that were retrieved but don’t help. The task type for this is question answering, with the retrieved passages inside the user message.

How does the combination work?

The two layers stay separate and only meet at the prompt. Retrieval is unchanged: same embeddings, same vector store, same top-k. What changes is the generator, which has been trained on examples of exactly the shape it will see at inference.

That training signal is what a base model doesn’t have. Lewis and colleagues established the pattern of pairing a fixed parametric memory with an editable non-parametric index; what the original formulation left open is how well the generator handles an imperfect index. RAFT closed that gap by training the model with distractor documents in context, so it learns to cite the right sequence from the relevant document instead of averaging over everything it was handed.

On the distil labs platform it’s the question-answering task, and the distractors have to be added to your training data (previously, we had a built-in setting):

base:
  task: question-answering
  student_model_name: Qwen3-1.7B
  teacher_model_name: zai.glm-5.3-low-thinking
evaluation:
  llm_as_a_judge_model_name: zai.glm-5.3-low-thinking
synthgen:
  teacher_temperature: 0.6

The temperature value isn’t arbitrary. Reasoning teachers such as zai.glm-5.3-low-thinking require 0.5 to 0.7, per the supported models catalog.

What does the training data look like?

Each training example is a question, the passages the model is handed, and the answer, not a question-answer pair alone. The passages go inside the user turn, exactly the way your retriever will present them at inference: the right chunk plus the irrelevant ones that came back with it.

Part of the row Contents Why it’s there
user turn The retrieved passages, then the question Teaches grounding, not recall; the distractor chunks teach the model to ignore what doesn’t help
assistant turn The answer, taken from the right passage The behaviour to learn
traces.jsonl (optional) Real conversations from your current RAG system Steers synthetic generation toward your domain and your retriever’s output shape

The question answering data preparation guide has the exact JSONL shapes. In practice the passages are your existing RAG chunks. If you’ve already chunked a corpus for retrieval, building rows is a re-export of what your retriever returns, not new work.

When is the combination worth the training run?

It’s worth it when retrieval is already finding the right passage and the answer is still wrong. To check, run your retriever, read the top chunk yourself, and see whether a careful reader could answer from it. If they could, the generator is your bottleneck.

Two situations make it clearly worth it:

  • The model has to be small. In a distil labs test on a Siemens PLC manual, a Llama 1B base model scored 45.1% across 144 questions and 61.1% after fine-tuning, reaching parity with a Llama 3B base model at 60.4%. The write-up covers the setup. At 1B, fine-tuning is the difference between viable and not.
  • The corpus has hostile formatting. Signal paths, parameter codes, nested tables. Base models below a few billion parameters misattribute values across table rows and invent plausible-looking codes.

It isn’t worth it when a large hosted model already answers correctly from your chunks and volume is low, since the training run would buy you nothing you don’t already have.

What breaks when you combine them?

The most common breakage is a mismatch between training-time and inference-time context. If you train on clean, hand-picked passages and serve on noisy top-3 retrieval, the model has never seen the distribution it now faces, which is the exact problem distractor context in the training rows exist to prevent.

Three others worth knowing:

  1. Chunk size drift. Retrain if you change your chunking strategy. A model trained on 1000-character chunks handles 4000-character ones worse.
  2. Corpus churn outrunning the model. The facts live in the index and update freely, but if the shape of your documents changes (new manual format, new ticket schema), the learned behaviour goes stale even though the facts are fresh.
  3. Evaluating grounding with the wrong metric. Exact match punishes correct paraphrase. The metrics guide covers why LLM-as-a-judge is usually the right primary score for answering over retrieved passages.

Sources

Related

All Alternatives articles →