Chapter 7.3 — Rollout strategies¶
🎯 Objective¶
Map rollout strategies with a criterion for choosing. In AI, shadow and canary gain special importance because much of the behavior only appears under real traffic.
📊 Strategies¶
| Strategy | When to use | Trade-off | AI-specific |
|---|---|---|---|
| Canary | Medium-risk change | Validation time | The quality metric can take time to manifest; needs a window |
| Blue/green | Fast switching | Temporary doubled cost | Useful when rollback must be instant |
| Shadow | Compare versions without affecting the user | No UX feedback; only valid for components without side effects | Excellent for LLMs (no effect) and retrieval; bad for tools that mutate state |
| Feature flag | Enable by segment | Configuration complexity | Allows activating a new agent for one tenant before the rest |
| Progressive rollout | Tenant by tenant | Longer | The recommended pattern for changes with regression risk |
| A/B test | Business metric | Sample size | Necessary to validate real gain |
🧠 Declared reversal (rollback) criteria¶
Before the rollout, record objectively what triggers reversal:
- A > X% drop in task success in a window of Y hours.
- A > Z% increase in cost per success.
- An increase in policy violations or severe incidents.
- An increase in p95 outside the SLO.
- Real complaints above a threshold.
Automatic reversal when possible; manual reversal with a published criterion when not.
📌 Rollout checklist¶
- [ ] Are the success and reversal criteria written before the rollout?
- [ ] Is there a pre-promotion eval?
- [ ] Is there a rollout-specific dashboard, separating control and candidate traffic?
- [ ] Is reversal tested (drill) periodically?
- [ ] Is there communication to stakeholders about the window and impact?