Chapter 6.1 — Total cost of AI systems
Total cost =
input_tokens + output_tokens
+ embeddings + retrieval + reranking
+ tool calls + storage
+ observability + evals
+ retries + retraining
+ human review
+ serving_infra
+ incidents
📊 Cost units
| Unit |
Why it matters |
| Cost per call |
Basic estimate |
| Cost per step |
Bottlenecks in the agent |
| Cost per task |
The main product metric |
| Cost per journey |
Long workflows |
| Cost per tenant |
Chargeback / showback |
| Cost per error |
Retries and failures can dominate spend |
| Cost per eval |
Continuous tests consume budget |
| Cost per success |
Real efficiency (cost/success, not cost/call) |
| Cost per human review |
Latency and human operation enter |