| Prompt caching |
Reduces cost and latency of prefixes |
Requires a stable prefix |
| Semantic cache |
Avoids recomputing similar responses |
Risk of a stale response |
| Model routing |
Uses a smaller model when possible |
Routing can err |
| Selective reranking |
Improves quality where it matters |
Additional cost |
| Streaming |
Improves latency perception |
Does not reduce total cost |
| Batch |
Reduces cost in async |
Increases latency |
| Context compaction |
Reduces tokens |
May lose evidence |
| Tool result compression |
Reduces context |
May hide details |
| Early exit |
Stops when confidence is high |
May stop too early |