Skip to content

Chapter 5.2 — Observability for ML, LLMs and agents

🎯 Objective

Map the observability layers in AI: model, LLM, agent and business — and propose a defensible open-source stack.

🧠 Layers

Layer Focus Main metrics
ML Isolated model p95 latency, accuracy/F1, calibration, drift
LLM Model call Tokens, cost, latency (TTFT/TPOT), quality
Agent End-to-end execution Task success, tool accuracy, fallback rate, steps
Product User journey Time to resolution, satisfaction, deflection
Business Outcome Aggregated KPI, cost per success, retention

Each layer matters for distinct reasons. Optimizing only the "model" layer is the modern equivalent of optimizing gettimeofday() in a system whose slowness comes from the database.

🧠 Operational difference between observing model, product and system

  • Observing the model: is it responding? At what latency? With what token distribution? With what error rate?
  • Observing the product: is the user getting what they want? Does the journey meet the promised SLA?
  • Observing the system: is tenant X consuming a disproportionate budget? Is the RAG index lagging? Is the policy rejecting more than usual?

The three answers are in different layers. Mixing them results in dashboards nobody opens after the first incident.

🧰 A defensible open-source stack

  • OpenTelemetry — unified tracing, metrics, logs; adopt GenAI semantic conventions from the start.
  • Jaeger or Grafana Tempo — tracing backend.
  • Prometheus + Grafana — metrics and dashboards.
  • Loki — structured logs.
  • OpenSearch / Elasticsearch — when textual search in traces is needed.

For AI-specific needs, common complements: Langfuse, Arize Phoenix, TruLens. They usually emit/consume OTel or export to the backends above.

📚 References