Skip to content

Chapter 7.7 — Traditional ML at scale: distributed training

🎯 Objective

Cover, at a minimal and useful level, the distributed training options and the situations where this really matters for a production team. This chapter does not intend to be a distributed-training manual; it closes the gap between "training a traditional model on a laptop" and "training/fine-tuning on a cluster".

🧠 When it matters

  • Large models (open-weight LLMs in fine-tuning, some vision and speech models) that do not fit in a single GPU.
  • Large datasets where training time on one machine is unfeasible.
  • Training throughput matters for fast iteration.

When it does not matter for most teams:

  • Classic models (gradient boosting, linear, small networks).
  • Light fine-tuning with LoRA/QLoRA on one GPU.
  • Data volume in hundreds of MB to a few GB.

🧠 Core concepts

  • Data parallelism (DP). Each GPU has a full copy of the model; data is partitioned. The gradient is synchronized across GPUs (AllReduce). Simpler; hits a memory limit.
  • Model parallelism. The model is partitioned across GPUs. Necessary when the model does not fit in a single GPU.
  • Tensor parallelism (TP). Splits each tensor into pieces across GPUs (by row/column). Good intra-node efficiency.
  • Pipeline parallelism (PP). Splits layers across sequential GPUs. Useful for very deep models; requires care with pipeline bubbles.
  • ZeRO (Zero Redundancy Optimizer). Partitions optimizer states, gradients and parameters across GPUs, reducing per-GPU memory. Implemented in DeepSpeed (stages 1, 2, 3).
  • FSDP (Fully Sharded Data Parallel, PyTorch). Conceptually equivalent to ZeRO-3 integrated in native PyTorch: parameters, gradients and optimizer states are sharded.
  • Activation checkpointing. Recomputes activations in the backward pass in exchange for less memory; trades compute for memory.
  • Mixed precision (FP16/BF16 + FP32 master). Reduces memory and speeds up with Tensor Cores; standard in modern training.

🧠 The difference between training large models and operating classic models

Operating a classic classifier in production is a topic of traditional MLOps: ingestion, features, monitoring, periodic retraining. The cycle is monthly/quarterly, with modest hardware.

Training/fine-tuning a large model is a topic of training infrastructure: GPU cluster, schedulers, checkpoints, recovery, GPU-hour cost. The cycle is rare (a few per year) and requires a specialized team.

Enterprise teams that consume ready-made models via API usually do not need this infrastructure. When they do, it is an explicit strategic decision.

⚠️ Honesty about scope

This book does not intend to replace official documentation and materials specialized in distributed training. The goal here is to close the vocabulary gap so that architects do not confuse "training an LLM" with "fine-tuning with LoRA" nor underestimate the cost of cluster infrastructure.

📚 References