Chapter 1.8 — Model serving¶
🎯 Objective¶
Cover the four serving modes and their trade-offs.
🧠 Modes¶
| Mode | When to use | Trade-off |
|---|---|---|
| Batch | Daily recommendations, mass scoring | High latency, low cost |
| Online (request/response) | Interactive applications | Low latency, higher cost |
| Streaming | Continuous events (fraud, alerts) | Low latency, high complexity |
| Edge / on-device | Mobile, IoT, privacy constraints | Small model, no network dependency |
🏗️ Patterns¶
- Containerization with Docker and orchestration via Kubernetes/ECS.
- Specialized servers — TF Serving, TorchServe, Triton, ONNX Runtime, Ray Serve.
- API gateway with authentication, rate limiting and metrics.
- Auto-scaling based on SLO (p95) and cost.
- Warm pools to reduce cold start.
🚨 Failure modes¶
- Acceptable latency on average, unacceptable at p95/p99.
- Cold start breaking the SLO at peak times.
- A large model causing OOM in production.
- Inconsistent versioning between weights, code and schema.
🛡️ Mitigations¶
- Define the SLO at p95/p99, not at the average.
- Load tests with realistic profiles.
- Quantization and pruning to reduce cost.
- Canary deploy.
🧰 Related practical example (planned)¶
EX-ML-04— a scikit-learn classifier served via FastAPI with Docker and health checks.
📚 References¶
- BentoML — Prefill–decode disaggregation: https://bentoml.com/llm/inference-optimization/prefill-decode-disaggregation
- NVIDIA Triton Inference Server — docs: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/
- Ray Serve — docs: https://docs.ray.io/en/latest/serve/index.html
- TensorFlow Serving — docs: https://www.tensorflow.org/tfx/guide/serving
- TorchServe — docs: https://pytorch.org/serve/