Chapter 2.10 — RAG vs fine-tuning vs prompt engineering¶
🎯 Objective¶
End the false dichotomy of "fine-tune or RAG" and offer an operational criterion for choosing between prompt engineering, RAG, fine-tuning, structured outputs, policy-as-code, distillation and model routing.
🧠 Guiding principle¶
The right question is rarely "fine-tune or RAG?". The right question is:
What is missing from the model so that it does what I need reliably?
The answer to that question points to the mechanism, and the mechanism points to the technique.
| What is missing | The right mechanism |
|---|---|
| Mutable knowledge (FAQ, docs, internal data) | RAG |
| Rigid output format (JSON/Pydantic) | Structured outputs + schema |
| Critical business rules | Policy-as-code + deterministic validation |
| Consistent style / voice / format | Light fine-tuning (LoRA) or stable few-shot |
| Reducing the cost of a large model for a specific task | Distillation |
| Choosing a model by query complexity | Model routing |
| Repetitive task with a labeled dataset and tight latency | Traditional ML or fine-tuning |
| Reordering results | Reranker |
| Ensuring an action only happens with permission | Authorization (IAM/OAuth) + policy |
🚨 Frequent anti-patterns¶
- Fine-tuning to insert facts. Mutable factual knowledge goes to RAG or tools. Fine-tuning for facts ages in weeks and requires retraining.
- Prompt as a security policy. A prompt is a probabilistic instruction. A critical policy goes in code.
- RAG without retrieval evaluation. A new index is promoted based on "intuition"; recall drops 20%; nobody notices until the incident.
- Swapping a large model for a "different" large model without first examining routing, RAG, prompt and structured outputs. There is almost always a cheaper gain before the upgrade.
🧪 Decision heuristic¶
- First of all, structured outputs + RAG + a good prompt. This solves most cases.
- Routing between small and large models when load variability justifies it. Reduces cost without affecting average quality.
- Fine-tuning only if there is eval showing that prompt + RAG do not reach the result and that there is a dataset stable enough to train and maintain.
- Distillation when the large model is the dominant cost, there is a well-defined specific task, and there is a budget for continuous eval of the student against the teacher.
- Multi-agent / complex agent only when workflow is already exhausted.