Chapter 1.3 — Practical statistics for ML
🎯 Objective
Cover the operational statistical minimum: bias, variance, overfitting/underfitting, calibration, ECE, confidence intervals and significance.
🧠 Core concepts
- Bias — systematic error from excessive simplification. Leads to underfitting.
- Variance — excessive sensitivity to the dataset. Leads to overfitting.
- Underfitting — high error on training and test. The model does not capture the pattern.
- Overfitting — low training error, high test error. The model memorizes noise.
- Calibration — correspondence between predicted probability and observed real frequency.
- ECE (Expected Calibration Error) — weighted average difference between confidence and accuracy across bins.
- Confidence interval — uncertainty range for a metric.
- Statistical significance — t-test, bootstrap; they let you say whether a difference is real or noise.
📏 Classification metrics
| Metric |
Formula |
When to use |
| Accuracy |
(TP+TN)/total |
Balanced classes |
| Precision |
TP/(TP+FP) |
High false-positive cost |
| Recall |
TP/(TP+FN) |
High false-negative cost |
| F1 |
2·P·R/(P+R) |
Imbalanced classes |
| ROC-AUC |
Area under ROC |
Threshold-independent comparison |
| PR-AUC |
Area under PR |
Heavily imbalanced classes |
| Brier score |
Squared probability error |
Calibration |
| Log loss |
-log(p truth) |
Calibration and ranking |
| ECE |
Confidence vs accuracy difference |
Calibration |
📐 Regression metrics
| Metric |
When to use |
| MAE |
Robust to outliers |
| RMSE |
Penalizes outliers |
| MAPE |
Percentage error |
| R² |
Explained variance |
| Quantile loss |
Quantile prediction |
📊 Ranking and recommendation metrics
| Metric |
When to use |
| NDCG@k |
Ranking with ordinal relevance |
| MAP@k |
Multiple relevant items |
| MRR |
Position of the first relevant item |
| Hit rate@k |
Basic coverage |
| Coverage |
Diversity of recommended items |
| Diversity / serendipity |
Non-trivial recommendation |
🚨 Failure modes
- High accuracy with a rare class ignored.
- The default threshold (0.5) treated as sacred.
- A miscalibrated model used in a financial decision.
- A statistical difference not tested before declaring "a better model".
🛡️ Mitigations
- Evaluate with multiple metrics.
- Report confidence intervals via bootstrap.
- Apply isotonic calibration or Platt scaling when needed.
- Use paired tests (bootstrap, McNemar) before declaring victory.
📚 References