Skip to content

Chapter 6.5 — FinOps for AI

🎯 Objective

Treat FinOps in AI as a continuous discipline, not a quarterly audit. The variable nature of LLM and agent cost — tokens per call, retries, continuous eval, observability, multi-tenant spikes — makes it impractical to manage cost with a monthly report alone.

🧠 Concept

FinOps for AI combines three things that, in traditional systems, usually live in separate teams: visibility (who spends what), allocation (whose spend it is) and optimization (how to reduce without losing quality). The right unit of measurement is almost never "cost per call"; it is cost per success — how much, on average, it costs to actually resolve a task.

Without that unit, inefficient agents look cheap (few tokens per call, many useless calls) and expensive but effective agents look wasteful. The difference only appears when the calculation includes retries, fallbacks, eval and human review.

🛡️ Practices

  • Budget per product, tenant and agent.
  • Consumption anomaly alerts.
  • Cost per success (not just cost per call).
  • Cost per error.
  • Eval costs.
  • Tracing/storage costs.
  • Retention policies.
  • Model routing.
  • Review of long prompts.
  • Formal chargeback / showback.
  • EX-COST-01 — a simple cost budget enforcer for LLM calls (planned).