Skip to content

Chapter 7.6 — Fine-tuning, distillation and alignment in production

🎯 Objective

Treat fine-tuning as an engineering discipline, with a criterion for when to use it, when not to, and what the minimum operational obligations are. This chapter speaks directly with Ch. 2.2 (quantization + LoRA/QLoRA) and with Ch. 2.10 (RAG vs FT vs prompt).

🧠 What fine-tuning is really for

Fine-tuning is good for changing:

  • Format (a recurring output structure that prompt + structured output do not stabilize sufficiently yet).
  • Style / voice (company tone, specific terminology, domain technical language).
  • Tool calling behavior with well-defined schemas and a stable dataset.
  • A specialized domain with vocabulary far outside the training distribution.

Fine-tuning is bad for:

  • Mutable factual knowledge -> use RAG.
  • Business rules -> use policy-as-code.
  • "Guaranteeing" safety behavior -> use authorization + policy.

🧠 When to use each technique

Technique What for Cost Main risk
Full fine-tuning Broad change; reorganize behavior High (compute, data, eval) Regression on unseen tasks
LoRA Low-rank adapters over selected layers Medium Quality depends on hyperparameters and dataset coverage
QLoRA LoRA over a 4-bit base (NF4) Medium-low (single GPU) Numerical variation; test carefully
Distillation Reduce inference cost while keeping behavior High upfront, pays off in serving The student diverges from the teacher over time
Instruction tuning / SFT Teach following structured instructions Medium Overfit to style
Preference tuning (DPO/KTO/ORPO) Align to preferences High + delicate Regression on uncovered capabilities; out of scope for this book

🧠 The difference between behavior, format, style and knowledge

In a typical planning conversation:

  • If the frustration is "the model does not respond in format X" -> use structured outputs, prompt and function calling. Fine-tuning only when the problem persists at scale.
  • If the frustration is "the model does not talk like us" -> use light fine-tuning (LoRA) for style, keeping knowledge in RAG.
  • If the frustration is "the model does not know X" -> use RAG or tools. Not fine-tuning.
  • If the frustration is "the model sometimes ignores the policy" -> use policy-as-code and HITL. Not fine-tuning.

🛡️ Minimum operational obligations

  • Pre- and post-FT eval on golden + adversarial + regression. Compare against the baseline on all relevant dimensions, not just the one that motivated the FT.
  • Immutable version + moving alias. A fine-tuned model is a versioned artifact; it lives in the registry.
  • A defined rollback. Which model comes back? In how long?
  • Regression metrics monitored in production. FT can fix A and silently break B.
  • Dataset documentation (datasheet/dataset card): origin, license, known bias, legal basis.

🚨 Failure modes

  • FT on a dataset with untreated PII -> leakage via memorization.
  • FT with a small, noisy dataset -> catastrophic overfitting.
  • FT for facts -> the knowledge "ages"; it goes back to being wrong within weeks.
  • Mixing QLoRA adapters trained on different versions of the 4-bit base.
  • Promoting an FT model without a comparative eval.

⚠️ On RLHF / DPO within this book's scope

RLHF, DPO, KTO and ORPO are preference-alignment techniques. They are relevant for teams that train the whole base model. For the vast majority of enterprise teams that consume a ready-made model and do light FT, these techniques are out of operational scope — mentioned so the vocabulary is not absent, but without going deep into specific techniques.

📚 References