Chapter 1.9 — Monitoring, drift and retraining

🎯 Objective

Distinguish types of drift and define a retraining strategy.

🧠 Types of drift

Type	What changes	How to detect
Data drift	Feature distribution	KS test, PSI, Jensen-Shannon
Concept drift	Feature -> label relationship	Performance drift, A/B vs baseline
Label drift	Label distribution	Per-class frequency over time
Performance drift	Quality metric drops	Continuous metric monitoring
Feedback drift	User behavior changes	Business metric

🛡️ Retraining strategies

• Reactive — retrain when drift is detected.
• Periodic — fixed window (weekly, monthly).
• Online incremental — small continuous updates (rare in enterprise).
• Business-metric trigger — retrain when the KPI drops.

🚨 Failure modes

• Retraining without a defined rollback.
• Retraining with data contaminated by the model itself (feedback loop).
• Retraining without comparison against a baseline.

🧰 Related practical example (planned)

• EX-ML-05 — a drift monitor with Evidently.

📌 Checklist

• [ ] Is there automated drift detection?
• [ ] Does retraining have regression tests?
• [ ] Is there a defined rollback?

📚 References

• Evidently AI — Concept drift and Data drift docs.

---