Where the money actually goes
In most production AI systems the bill is dominated by input tokens sent repeatedly — the same system prompt, the same instructions, the same retrieved context, on every single call. That is also the cheapest thing to fix.
These are the techniques that reduce spend meaningfully, ordered by return per hour of engineering, with the trade-off each one carries stated honestly.
Measure before optimising
Optimisation without attribution wastes effort. Instrument first:
- Cost per call, broken into input and output tokens.
- Attribution — which feature, which user segment, which task type.
- Cost per completed outcome, not per API call. Three cheap calls that fail cost more than one that works.
- Distribution, not just average. A small number of very expensive calls often dominates, and averages hide them.
The most common surprise: a single feature nobody considered important generating the majority of spend — usually because it runs on every page load, or retrieves far more context than it needs. You cannot find that without attribution, and no amount of prompt tuning elsewhere will offset it.
The techniques, by return
1. Prompt caching — usually the largest single win
If your calls share a long common prefix — system prompt, instructions, few-shot examples, static context — caching that prefix means you stop paying full price for it on every request.
- Trade-off: requires structuring prompts so the stable part comes first and stays byte-identical.
- Best for: any system with a substantial fixed preamble, which is most of them.
2. Trim retrieved context
RAG systems routinely retrieve ten chunks when three would answer the question. Every unnecessary chunk is input tokens on every call.
- Retrieve more, then rerank and keep only the top few.
- Set a relevance floor — do not include weak matches to fill a quota.
- Shorten chunks where possible without breaking meaning.
- Trade-off: trimming too aggressively causes missed answers. Measure against your evaluation set.
3. Route by task complexity
Not every request needs your most capable model. Classification, routing, extraction and simple formatting are frequently handled well by a smaller, cheaper model.
| Task | Model tier | Rationale |
|---|---|---|
| Intent classification | Small | Narrow, well-defined |
| Data extraction from structured text | Small | Pattern matching |
| Summarisation | Mid | Quality matters, reasoning does not |
| Customer-facing answers | Mid to large | Quality is visible externally |
| Multi-step reasoning | Large | Where capability genuinely differs |
Trade-off: routing logic is itself code to maintain, and misrouting degrades quality invisibly. Measure per-route accuracy.
4. Cache complete responses
Identical or near-identical questions recur far more than teams expect — particularly in support. Caching the full response for repeated queries eliminates the call entirely.
Trade-off: stale answers if underlying data changes. Cache with a short TTL, and invalidate on content updates.
5. Cap output length deliberately
Output tokens cost more per token. Models are verbose by default. Instructing concision and setting max tokens reduces cost and usually improves the user experience.
6. Batch non-urgent work
Where a task does not need an immediate response — overnight classification, bulk enrichment, backfill — batch processing is materially cheaper on providers that offer it.
Trade-off: latency. Only for genuinely asynchronous work.
Do these in order and measure after each. Teams frequently jump to changing models — the most disruptive option — before doing caching and context trimming, which are cheaper to implement, lower risk, and often deliver more. Model changes should be the last lever, not the first.
What not to do
- Do not degrade quality silently. A cheaper system producing worse answers costs you more in support and trust than it saves in tokens.
- Do not optimise before instrumenting. You will improve something that was not the problem.
- Do not remove guardrails to save calls. Verification steps cost tokens and prevent expensive mistakes.
- Do not over-cache dynamic data. Confidently stale answers are worse than slower correct ones.
Governance that prevents surprises
- Spend alerts at a threshold you choose, before the invoice arrives.
- Per-user and per-session rate limits — an agent in a loop is the classic runaway.
- Hard caps on autonomous workflows, so a failure mode cannot spend unbounded.
- Monthly cost review alongside quality metrics, so trade-offs are visible together.
Track cost per successful outcome, not cost per call. A system that halves its call cost while doubling its failure rate has become more expensive, and the token metric will not show it.
Running production AI with a bill that is growing faster than usage? Tell us your architecture and volumes. See our AI agent service, RAG architecture, and AI observability.