Choosing a Model for Your Agent: Capability, Latency and Cost
The question teams ask is which model is best for agents. The question that produces a good system is which model is best for this step, on our data, at our latency budget — and the answer is usually more than one model.
An agent run is not homogeneous. Deciding the next action needs reasoning. Extracting three fields from a returned document does not. Summarising a result for a user does not. Treating all of these as one procurement decision is how teams end up paying frontier prices to reformat JSON.
Split the run before you choose#
| Step | What it needs | Sensible model tier |
|---|---|---|
| Plan or choose next action | Reasoning, instruction adherence | Strongest you can afford |
| Call a tool with arguments | Reliable structured output | Mid tier with strict schemas |
| Extract fields from a result | Accuracy on short text | Small and fast |
| Classify or route | Consistency | Small, or a fine-tuned classifier |
| Write the user-facing reply | Tone and clarity | Mid tier |
Benchmarks are a shortlist, not a decision#
Public benchmarks tell you which models are plausible. They cannot tell you which one handles your tool schemas, your document formats and your awkward customers, because none of those are in the benchmark. Build a set of thirty real cases from your own logs — including the five that embarrass you — and run the shortlist against it. The gap between benchmark order and your-own-data order is routinely large enough to change the decision.
Include cases where the correct behaviour is to refuse or to ask a question. Models differ far more on knowing when to stop than on knowing what to say.
The three constraints that actually bind#
- Latency floor: each model call has one, and an agent makes several. Measure the whole run, not one call.
- Structured output reliability: a model that returns valid arguments 97% of the time fails one run in ten with three tool calls.
- Context handling: long contexts cost more and degrade attention; measure accuracy at the context length you will really use, not at the marketing maximum.
Routing without a research project#
Model routing sounds sophisticated and is usually a config file. Assign a default model per step type, allow an override per tool, and log which model produced each decision so you can attribute failures. Start by moving only the extraction and classification steps down a tier; that alone often removes a third to a half of the token bill without touching the quality of the part users notice.
Plan for deprecation, because it is coming#
Model versions are retired on the provider’s schedule, not yours. Two habits make that a non-event: pin an explicit version rather than a floating alias so nothing changes under you silently, and keep your evaluation set runnable with a single command so re-qualifying a replacement is an afternoon rather than a project. Teams without those habits discover the deprecation email and the incident on the same morning.
Frequently asked questions
Should I use the biggest model for everything?
Only if you have not measured. The decision step usually benefits from the strongest model; extraction, classification and formatting rarely do. Splitting by step is the single easiest cost reduction available and it does not touch quality where users see it.
Do open-weight models work for agents?
For narrow, well-defined steps with strict schemas, frequently yes, and the economics at volume can be compelling. For open-ended planning across many tools they still tend to need more scaffolding. Test on your own thirty cases rather than on a leaderboard.
How often should I re-evaluate my model choice?
Whenever a provider ships a version you might adopt, and on a fixed cadence of about two quarters otherwise. That is only sustainable if re-running your evaluation set is a single command, which is the real reason to invest in it.
choosing an llmmodel selection for agentsllm routingagent latencystructured output reliability
