Skip to content

Chapter 1.4 — Loss functions and objective alignment

🎯 Objective

Show that loss, offline metric and business metric rarely coincide — and that this misalignment is a recurring source of models that "work" but generate no value.

🧠 Concepts

  • Loss defines what the model optimizes.
  • Offline metric measures what we measure before deploying.
  • Business metric measures what matters to the user or the company.

Examples of misalignment:

  • Loss cross-entropy -> metric F1 -> business metric satisfaction.
  • Loss MSE -> metric RMSE -> business metric savings.
  • Loss pointwise -> metric NDCG -> business metric engagement.

🛡️ Best practices

  • Explicitly define the chain loss -> offline metric -> business metric.
  • Validate the correlation between metrics before deploying.
  • Accept that the business metric is only measurable in production (via A/B test or longitudinal monitoring).

🚨 Failure modes

  • Optimizing CTR and degrading trust or diversity.
  • Optimizing accuracy and ignoring a critical segment.
  • Optimizing short-term engagement and losing long-term retention.

📌 Checklist

  • [ ] Is the chain loss -> offline metric -> business metric documented?
  • [ ] Is the business metric monitored after deployment?
  • [ ] Is there per-segment analysis, not just a global average?