Cutting AI Agent Costs Without Making It Worse
When a token bill surprises someone, the instinct is to switch to a cheaper model everywhere and accept the quality loss. That is rarely necessary. In the systems we have audited, the majority of spend came from context that did not need to be there and from steps that did not need the expensive model.
The method below is boring and effective: measure first, then apply four changes in order of return, then decide whether you still have a problem. Most teams stop after the second change.
Measure per run before changing anything#
Aggregate monthly spend tells you nothing actionable. Log, for every run: input tokens, output tokens, number of model calls, model used per call, and the task type. Then look at cost per completed task, split by task type. Almost always one or two task types dominate, and within them one step dominates. Optimising anything else is effort spent where the money is not.
Include failed and abandoned runs in the denominator. A retry loop that burns three attempts before succeeding is a cost problem disguised as a quality problem.
The four changes, in order of return#
| Change | Typical saving | Risk |
|---|---|---|
| Trim context: discard used documents, compress history | 20–40% | Low, if the goal stays pinned |
| Route cheap steps to a smaller model | 20–40% | Low, with per-step evaluation |
| Cache the stable prompt prefix | 10–30% on repeat traffic | Low |
| Reduce steps: better tools, fewer retries | 10–25% | Medium — requires tool work |
Context is the bill#
Every turn resends the accumulated context, so a run of eight steps can pay for the same document eight times. Three habits fix most of it: discard retrieved passages once the step that needed them is finished; compress old turns into short factual notes instead of carrying transcripts; and trim tool results to the fields the agent uses rather than passing whole API responses. None of these reduce capability — they remove text the model was not using anyway, and quality often improves because attention is less diluted.
Route by step, not by taste#
Extraction, classification and formatting rarely need your strongest model; planning and final user-facing prose often do. Move the first group down one tier, run your evaluation set, and keep the change only if the numbers hold. Doing this per step rather than globally is what lets you cut a third of the bill without anyone noticing a difference in the output they read.
- Start with the highest-volume, lowest-judgement step.
- Change one step at a time and re-run the evaluation set after each.
- Log which model produced each decision so a later regression can be attributed.
- Set a per-run spend cap so a pathological case cannot be unbounded.
What not to do#
Do not cut the retrieval that grounds your answers — hallucinated output is far more expensive than tokens once someone has to correct it. Do not remove the critic pass on irreversible actions to save a call. And do not chase micro-optimisations in prompt wording; the savings are noise next to context trimming, and you will spend engineering hours worth more than the difference.
Frequently asked questions
Is caching worth setting up?
If your runs share a long stable prefix — system instructions, tool definitions, policy text — yes, and it is one of the cheapest wins available. Structure the prompt so the stable part comes first and the variable part last, or caching cannot help you.
Should I fine-tune to save money?
Only for a high-volume, narrow, stable step where a small model with tuning matches a larger one. Fine-tuning adds a maintenance obligation and a re-training cycle; at low volumes routing and context trimming beat it comfortably.
How do I stop one pathological run from costing a fortune?
Cap steps and spend per run, detect repeated identical tool calls, and stop with a partial result rather than continuing. Then alert on runs that hit the cap — they are usually a bug worth fixing, not merely an expense.
ai agent costllm cost optimizationtoken cost reductionprompt cachingmodel routing
