Chapter 5.1 — Why logs are not enough¶
🎯 Objective¶
Establish the operational difference between a textual log, an aggregated metric and semantic tracing — and show why AI requires all three, with disproportionate emphasis on the third.
🧠 Concept¶
In a traditional application, log + metric usually suffices for 80% of operations. In AI, that percentage inverts: what distinguishes a useful diagnosis from an educated guess is the semantic trace, because most behavior depends on context, versions and intermediate decisions that do not fit in a log line nor in an aggregated metric.
Textual logs answer "what happened". Semantic tracing answers "how" and "why". In AI, the "why" involves:
- the context sent (model, prompt, versions, retrieval, tools);
- intermediate decisions (policy decision, fallback, retry, routing);
- states (memory used, tokens consumed);
- human approvals;
- cost and latency per step.
Without semantic tracing, an AI incident ends in "the model answered wrong" — with no visibility to diagnose it.
🧠 Practical difference¶
| Layer | What it answers | Limit |
|---|---|---|
| Metric | "How many times? In what latency range?" | Has no context |
| Log | "What did the service say?" | Textual, fragile, fragmented |
| Trace | "How did this specific case traverse the system?" | More expensive to produce; irreplaceable in an incident |
🚨 Frequent anti-patterns¶
- Logging whole prompts without redaction. Becomes a privacy incident.
- A "quality" metric as a global average without segmentation by tenant, language, intent.
- Tracing only on errors. When the hard case happens without an explicit error, nobody can investigate.