Skip to content

Chapter 2.12 — LLM observability

🎯 Objective

Define the minimum necessary to make an LLM call operable. This chapter covers the model layer in isolation; observability of agents and of the full system is in Part 5.

🧠 Why LLM tracing is different

In a traditional API, "what happened" is summarized by: status, latency, payload. In an LLM, the same call must also answer:

  • Which model responded? In which version?
  • Which prompt was sent? In which version?
  • How many tokens went in, out, came from the cache?
  • What was the cost?
  • Which documents did the retriever deliver?
  • What decision did the policy make?
  • Which tool was called, with which arguments (redacted)?
  • Did any guardrail trigger? Any fallback?

Without these fields, an LLM incident is a black box. Generic textual logging is not enough.

🧠 Minimum tracing fields per call

  • request_id, tenant_id (hash), user_id (hash).
  • model.name, model.version, prompt.version, prompt.template_id.
  • input.tokens, output.tokens, cached.tokens.
  • cost.usd, latency.ms, ttft.ms, tpot.ms (when available).
  • sampling.temperature, sampling.top_p.
  • retriever.index, retriever.dimensions, retriever.top_k, retriever.scores.
  • tool.name, tool.version, tool.args (redacted), tool.result_size.
  • policy.bundle, policy.decision, policy.reason.
  • guardrail.triggered, guardrail.action.
  • error.code, error.kind (if applicable).
  • safety.classification (if there is risk classification).

🧠 Standardization: OpenTelemetry GenAI

OpenTelemetry maintains GenAI semantic conventions with canonical names for LLM, embeddings, retrieval and agent span attributes. Adopting this convention from the start avoids rewriting dashboards and queries after the tracing backend changes.

🛡️ Principles

  • Redaction before logging. PII, secrets and sensitive data never go raw into tracing.
  • Conscious sampling. 100% may be expensive; 1% may hide the hard case. Use a sampling tier (higher on errors, on new cases, on critical tenants).
  • Always version. Model, prompt, schema, tool, policy. Without a version, it is not debuggable.

📚 References