Chapter 5.3 — Semantic tracing¶
🎯 Objective¶
Define, at an operational level, the minimum content of a trace in AI systems. Without standardization, traces become illegible and incident investigation turns into prospecting.
🧠 Span types¶
| Type | Key attributes |
|---|---|
| LLM call | model.name, model.version, prompt.version, input/output tokens, cost, latency, ttft |
| Embedding | model.name, dimensions, batch_size, latency |
| Tool call | tool.name, tool.version, args (redacted), result_size, error |
| Retrieval | index, dimensions, top_k, filters, scores |
| Agent execution | agent.name, agent.version, total tokens, steps, decision_path |
| Handoff (multi-agent / A2A) | from, to, task_id, payload size |
| Human approval | requester, approver, decision, latency |
| Policy decision | policy.bundle, decision, reason |
| ML inference | model.name, model.version, features hashing, latency, output |
📊 Suggested minimum fields¶
trace.id, span.id, parent.id,
agent.name, agent.version,
model.name, model.version, prompt.version,
tenant.id (hash), user.id (hash),
input.tokens, output.tokens, cached.tokens,
cost.usd, latency.ms, ttft.ms,
retriever.index, retriever.dimensions, retriever.scores,
tool.name, tool.version,
policy.bundle, policy.decision, policy.reason,
guardrail.triggered, guardrail.action,
error.code, error.kind (if any)
🧠 Conscious sampling¶
Tracing 100% at scale is expensive. Poorly done sampling hides exactly the interesting case. Useful strategies:
- Tiered sampling. High on errors, high on new cases, high on critical tenants; lower on routine traffic.
- Head sampling (decided at the start) is cheaper; tail sampling (decided later) captures tails with more fidelity.
- Force 100% during active incidents with an operational toggle.
🧠 Trace replay¶
Saving structured traces enables replay: re-running the same input against a candidate version and measuring the difference. Replay is the most honest form of "test in production" without affecting a real user.
For replay to work:
- Inputs must be captured in a reproducible form.
- PII must be redacted from capture onward.
- Tools with side effects must have a mock/replay mode.
🚨 Observability cost¶
Semantic tracing is not free. Typical costs:
- Span volume per task: dozens in complex agents.
- Retention: regulation may require 6–24 months for audit.
- Backend: storage + indexing + queries.
Cost-reduction decisions involve: sampling, payload compaction, tiered retention (hot/warm/cold) and separating operational traces (short-term) from audit traces (long-term).