Chapter 5.5 — Dashboards, alerts and SLOs

🎯 Objective

Define the minimum dashboards and alerts so that AI operations are seen early, before becoming an incident that shows up in a support ticket.

📊 Minimum dashboards

1. Quality. Task success by intent/tenant/language; groundedness; regressions detected in eval; human agreement on a sample.
2. Cost. Cost per task, per success, per tenant, per agent; tokens per step; retry cost; eval cost.
3. Latency. p50/p95/p99 per model, per tool, per retriever; TTFT/TPOT in self-host; end-to-end latency per intent.
4. RAG. top-k, precision, recall, citation coverage; indexing lag; median and p95 freshness; no-answer rate; truncation rate.
5. Tools. Calls per tool, error rate, timeouts, unauthorized attempts, use of deprecated tools.
6. Security. Prompt injection detections; policy violations; DLP triggers; secret scanning hits; cross-tenant attempts.
7. Operation. Fallback rate, escalation rate, open incidents, error budget, SLO burn.

🚨 Minimum alerts

• A sudden increase in cost per task (>X% in a window).
• p95/p99 above the SLO for a defined window.
• An increase in tool errors.
• An increase in policy violations.
• A drop in groundedness or task success per intent.
• An increase in no-answer rate or fallback.
• A high retry rate.
• Use of a deprecated tool.
• A cross-tenant access attempt.
• Prompt injection detected by a classifier.
• Schema validation errors rising.
• RAG indexing lag > SLA.

🧠 SLOs in AI

Traditional SLOs (availability, latency) still hold, but in AI quality SLOs appear:

• p95 task success > X.
• Median groundedness > Y per critical intent.
• Cost per success ≤ Z for task A.
• p95 citation coverage ≥ W in answers with RAG.

Since quality is hard to measure in real time, quality SLOs live in a moving window over continuous eval, not in an instantaneous production metric.

---