Skip to content

Chapter 1.2 — Real data, quality and failure modes

🎯 Objective

Treat data as the main operational risk of an ML system.

🧠 Data sources

Source Characteristic Main risk
Logs and events Semi-structured, high volume Fragile parsing, implicit schema
Transactional systems Structured, historical Schema change, snapshot vs point-in-time
Data lake / warehouse Volume, latency Freshness and governance
External APIs Heterogeneous, dependency Latency, rate limit, unstable contract
Manual data Entered by users Errors, bias, inconsistency
Synthetic data Artificially generated False representativeness

🚨 Data failure modes

  • Label leakage — a variable close to the label enters as a feature.
  • Target leakage — a feature contains future information.
  • Distribution shift — production differs from training.
  • Class imbalance — rare classes dominate the error.
  • Selection bias — biased data collection.
  • Late labels — labels arrive with delay, creating distorted feedback.
  • Cross-tenant leakage — one tenant's data appears in another's model.
  • Stale data — expired data treated as current.
  • Silent schema drift — upstream change without notice.

🛡️ Controls

  • Versioned data contracts (YAML/JSON), with quality rules.
  • Automated validation in the pipeline (Great Expectations, Pandera, dbt tests).
  • Lineage recorded via a data orchestrator (Dagster, Airflow).
  • Per-tenant isolation from ingestion onward.
  • Drift detection (Evidently, custom KS/PSI).

📝 Data contract example

name: transactions_v1
description: Transactions dataset for fraud detection.
owner: fraud-team
version: 1.2.0
schema:
  - name: transaction_id
    type: string
    required: true
    unique: true
  - name: timestamp
    type: datetime
    required: true
  - name: amount
    type: float
    required: true
  - name: user_id
    type: string
    required: true
  - name: tenant_id
    type: string
    required: true
quality:
  freshness_minutes: 60
  completeness_pct: 99.9
contracts:
  - rule: "timestamp <= now()"
    description: "Future events are invalid"
  - rule: "amount >= 0"
    description: "Negative values rejected"
  - rule: "tenant_id IN allowlist"
    description: "Known tenants only"
  • EX-ML-01 — scikit-learn + pandera pipeline to validate features.

📌 Checklist

  • [ ] Does the dataset have a versioned data contract?
  • [ ] Do validations fail the pipeline (rather than passing silently)?
  • [ ] Is lineage recorded?

📚 References