Skip to content

Chapter 6.2 — Latency, throughput and percentiles

🎯 Objective

Establish the minimum basis for discussing latency and throughput of AI systems without falling into the classic anti-pattern of "the average latency is fine". In production, what hurts is the tail.

🧠 Metrics

  • p50 shows the average experience of whoever is using the system right now.
  • p95 shows how the system responds on a bad day — it is the metric that usually becomes the SLO.
  • p99 shows potential incidents and small clients being silently harmed by contention, batching or retries.
  • Throughput measures aggregate capacity and sizes the infrastructure.
  • Rate limits and quotas come from the provider; monitor both sides (what you consume and what the provider imposes).

In AI, latency also appears on two different axes — TTFT (prefill) and TPOT (decode) — discussed in detail in Ch. 6.3. Optimizing the wrong axis is a recurring source of "we swapped the model and the user's perception did not change".

🚨 Anti-pattern

Defining the SLO only on the average. An AI SLO should use percentiles, and the choice between p95 and p99 should be explicit — not a default.