Skip to content

Chapter 6.3 — How to estimate the capacity and cost of an AI server

🎯 Objective

Give the reader the tools to estimate — before buying a GPU, signing a contract with a provider, or promising an SLA — how much it costs, in latency and money, to serve a model. The focus is reasoning, not a recipe: the formulas are simple and real metrics vary by hardware, kernel, batch and workload type.

🧠 Core concept

LLM inference has two phases with very different economics:

  • Prefill — processes the whole prompt in parallel. Cost dominated by compute (parallel FLOPs). Latency here dominates the TTFT (Time To First Token).
  • Decode — generates tokens one by one. Cost dominated by memory bandwidth (it must read all the weights for each token). Latency here dominates the TPOT (Time Per Output Token).

Therefore, cost and capacity depend heavily on prompt length vs response length, batch and KV cache usage.

🧮 Essential metrics

Metric What it measures Why it matters How to optimize
tokens/s (decode) Generation throughput Real server capacity Batching, paged attention, speculative decoding
TTFT (Time To First Token) Prefill latency (ms to the 1st token) Perceived UX in streaming Prefill chunking, prompt caching, smaller models
TPOT (Time Per Output Token) Time between tokens (ms/token) Long-response latency More memory bandwidth, aggregated batching, optimized KV cache
Requests/s (RPS) Request rate Throughput capacity Continuous batching, paged attention
Supported concurrency How many simultaneous requests Sizing KV cache and batching optimizations
p50/p95/p99 latency Latency distribution Real SLA Backpressure, queue policy, autoscaling
GPU utilization % SM/Tensor core usage Efficiency Batching, kernel tuning
Memory bandwidth utilization % HBM usage Efficiency in decode Quantization, paged attention
VRAM used GPU memory Sizing Quantization, KV offload, shorter sequences
Cost per task / per success Cost per unit of value Real FinOps Routing, caching, batching, smaller models
Cost per tenant Cost per customer Chargeback Rate limits, budgets, allocation
Cost per error / retry Wasted cost Identifies waste Eval, retries with backoff, fallback

🧮 Conceptual formulas

Memory for model weights:

VRAM_weights ≈ params × bits_per_param / 8

Quick examples:

  • 7B in FP16: 7e9 × 2 = ~14 GB
  • 7B in INT8: ~7 GB
  • 7B in INT4: ~3.5 GB
  • 70B in FP16: ~140 GB (typical of tensor parallelism across several GPUs)

KV cache per sequence:

KV_per_token ≈ 2 × num_layers × num_heads × head_dim × bytes_per_value
KV_per_sequence ≈ KV_per_token × seq_len
KV_total ≈ KV_per_sequence × concurrency

For a 7B FP16 model with 32 layers, 32 heads, head_dim 128, 4 KB/token is a reasonable order of magnitude. At a 4096-token window and 32 simultaneous sequences, the KV cache can easily exceed tens of GB, which explains why batching and paged attention are so critical.

Total tokens per task:

task_tokens = prompt_tokens + response_tokens + tools_overhead + retries

In RAG with tools, overhead can dominate: each tool call adds prompt + result to the context of the next call.

Cost per task (API provider):

task_cost ≈ prompt_tokens × price_in + response_tokens × price_out
          + other_calls (embedding, reranker)
          + retry_tokens × p(failure)

Cost per task (self-hosting):

task_cost ≈ (GPU_hour_cost / 3600)
          × (prompt_tokens / prefill_throughput
             + response_tokens / decode_throughput)

Supported concurrency (approximate):

concurrency ≈ total_decode_throughput / accepted_response_tokens_per_second

If the server delivers 2000 tokens/s in decode and each user generates ~50 perceptible tokens/s, it serves ~40 simultaneous sessions in decode.

Warning. These formulas are pedagogical. The real values depend on hardware, framework (vLLM, TensorRT-LLM, TGI, llama.cpp), format (FP16/FP8/INT4), prompt size, request arrival pattern and scheduling heuristics. Use them for orders of magnitude, then measure in an environment equivalent to production.

🧠 Why reducing VRAM does not always increase throughput

Dropping from FP16 to INT4 cuts weights by ~4x, but:

  • decode is limited by memory bandwidth — if the INT4 kernel does not unpack efficiently, the gain vanishes;
  • prefill is limited by compute — INT4 may even worsen it if the kernel needs frequent upcasting;
  • batching tends to be more efficient in FP16 with dedicated Tensor Cores;
  • speculative decoding may yield more gain than aggressive quantization, in suitable scenarios.

Conclusion: measure, do not assume. VRAM always drops; throughput depends.

🚀 Key modern serving techniques

Technique What it does Where it appears
Continuous batching Adds new sequences to the batch at each step, instead of waiting for the batch to close vLLM, TGI, TensorRT-LLM in-flight batching
PagedAttention KV cache fragmented into paged blocks, avoids fragmentation and enables large batches vLLM
Paged KV cache Same idea, exposed in other engines TensorRT-LLM, SGLang
Speculative decoding A fast draft model proposes tokens; the large model verifies in parallel vLLM, TensorRT-LLM, Medusa, Lookahead
Prompt caching Reuses the KV cache of stable prefixes OpenAI, Anthropic, vLLM (prefix caching)
Semantic caching Reuses responses for similar queries Gateways (Portkey, Kong AI, LiteLLM); use with care
Streaming Delivers tokens as they decode Reduces perceived TTFT; does not change throughput
Chunked prefill Breaks prefill into pieces so it does not block decode vLLM and similar
Tensor parallelism Splits the model across several GPUs by tensor For models that do not fit in one GPU
Pipeline parallelism Splits layers across sequential GPUs Very large models
CPU offload KV cache or weights in RAM Allows running larger on smaller hardware; loses latency
Disaggregated prefill/decode Separates into specialized servers Reduces contention; complicates orchestration

🏗️ Self-hosting vs API provider

Axis API provider (OpenAI, Anthropic, Bedrock, Vertex) Self-hosting (vLLM, TGI, TensorRT-LLM)
Initial cost Low (pay per use) High (GPUs, team)
Cost at scale Grows linearly Can be significantly lower at high utilization
Latency Depends on provider and region Controllable, more predictable
Capacity Limited by rate limits and quotas Limited by hardware
Available models The provider's catalog Any compatible open-weight model
Compliance / residency Depends on the contract Full control
Maintenance Zero Continuous cluster operation
Sensitivity to provider failure High Under your control
Optimization Limited (the provider does it for you) Full (kernel, batch, prompt caching)

Rule of thumb: small, variable traffic -> API provider. High, predictable traffic -> self-hosting tends to pay off; the break-even depends on real utilization.

🚨 Failure modes

  • Defining an SLA based on the average, not on p95/p99.
  • Sizing based on the weights' VRAM, ignoring the KV cache.
  • Forgetting the overhead of tool calls in agents (multiplies tokens).
  • Underestimating the cost of retries during an error spike.
  • Forgetting the costs of embedding, reranker, gateway and observability.
  • Not isolating concurrency per tenant.

🛡️ Controls and mitigations

  • Benchmarks with realistic profiles (a mix of prompt and response sizes, spikes, multi-tenant).
  • Metrics on two axes: latency (TTFT/TPOT/p95/p99) and cost (per task, per success, per tenant).
  • Autoscaling based on queue + memory bandwidth, not just CPU.
  • Explicit backpressure: reject before degrading all clients.
  • Per-tenant budgets + alerts + circuit breakers.

📌 Checklist for AI server sizing

  • [ ] Are there measured (not estimated) numbers for TTFT, TPOT, RPS on the target hardware?
  • [ ] Was the KV cache considered in the VRAM sizing?
  • [ ] Were realistic load profiles (prompt + tools + retries) tested?
  • [ ] Is the SLA on p95/p99 and on cost per success?
  • [ ] Was the self-host vs API comparison done with realistic utilization?
  • [ ] Are the degradation modes under peak clear (timeout, queue, reject)?
  • EX-LLM-11 — a synthetic load test with vLLM measuring TTFT/TPOT/p95.
  • EX-COST-03 — a cost-per-success calculator (provider vs self-host).

📚 References