Skip to content

Part 6 — Summary

Part 6 treated cost as an architectural dimension, not as a surprise at the end of the month. What was established:

  • Total cost is more than tokens: it includes infra, observability, evals, retries, retraining and human review.
  • Latency has two different axes: TTFT (prefill) and TPOT (decode); the wrong optimization goes to the wrong place.
  • AI server sizing depends on the KV cache, batching and workload type, not just on "the weights' VRAM".
  • Self-hosting vs API provider is a trade-off with a break-even that depends on real utilization, not on rhetoric.
  • Caching (prompt and semantic), model routing and batching are levers, each with its own risk.
  • FinOps in AI requires per-tenant/agent budgets, anomaly alerts and cost per success, not just per call.

Part 7 closes the cycle: how these systems are versioned, deployed, evolved and deprecated.