Chapter 1.6 — Training, validation and leakage

🎯 Objective

Establish splits, cross-validation and leakage prevention as mandatory practices.

🧠 Concepts

• Train/validation/test split — three distinct, non-overlapping partitions.
• Cross-validation — k-fold for small datasets; use with caution on temporal data.
• Temporal validation — for data with temporal order, train on the past and validate on the future.
• Stratified split — preserves the class distribution.
• Group split — preserves the entity (user, tenant) between training and test.

🚨 Types of leakage

• Target leakage — a feature contains label information.
• Label leakage — the label appears as a feature in a transformation.
• Train-test contamination — overlap between splits.
• Feature contamination — a feature uses a statistic computed over the whole dataset.
• Temporal leakage — future information enters past training.

🛡️ Mitigations

• Validate splits programmatically.
• Compute normalization statistics on the training set only.
• Apply a group split when there are repeated entities.
• Audit suspicious correlation between feature and label.

📌 Checklist

• [ ] Is the split reproducible with a seed?
• [ ] Is temporal validation used when applicable?
• [ ] Are there tests that detect target leakage?

---