The framework, not the answer
Model rankings change constantly, so an article naming "the best model" is wrong within months. What lasts is the selection method — and the discipline of choosing on your own evaluation set rather than a public leaderboard.
Written to be useful regardless of which models are ahead when you read it.
The five dimensions that actually decide it
| Dimension | Question to answer | How to test |
|---|---|---|
| Task capability | Does it do your task well? | Your evaluation set |
| Cost | What does it cost at your volume? | Model your real traffic |
| Latency | Fast enough for your interaction? | Measure at your prompt size |
| Context window | Fits your retrieved content? | Check against realistic prompts |
| Data policy | Acceptable to your legal team? | Read the terms, do not assume |
The data policy row is the one that stops projects late. Retention, training use, and processing location vary by provider and by tier. Enterprise tiers commonly offer zero retention, but confirm in writing rather than assuming. Discovering a policy problem after building is expensive.
Capability is task-specific
Models differ in ways that only matter for some tasks:
- Long-context reasoning — synthesising across many retrieved documents.
- Instruction adherence — reliably following format and constraint rules.
- Structured output — producing valid JSON consistently.
- Tool selection — choosing correctly among many available tools.
- Refusal calibration — declining appropriately without being over-cautious.
- Multilingual quality — varies considerably by language.
A model excelling at one may be mediocre at another. Which is why the only reliable comparison is against your task.
Model tiers within a provider
The choice is rarely just between providers — every provider offers a range, and using the largest model everywhere is a common and expensive mistake.
| Task | Tier | Reasoning |
|---|---|---|
| Intent classification | Small | Narrow and well-defined |
| Field extraction | Small | Pattern recognition |
| Routing decisions | Small | Speed matters more than depth |
| Summarisation | Mid | Quality visible, reasoning shallow |
| Customer-facing answers | Mid to large | Errors are externally visible |
| Multi-step reasoning | Large | Where capability genuinely differs |
Routing by task tier typically reduces cost substantially with no perceptible quality loss — because most calls in a production system are not the hard ones.
A selection process that survives model churn
- Build an evaluation set — 50–200 real cases with expected outcomes. This is the durable asset.
- Shortlist by rough capability — benchmarks are adequate for narrowing to three or four candidates.
- Run each against your set, scoring accuracy, and recording latency and token usage.
- Model the cost at your projected volume, not per call.
- Check the data policy with whoever needs to approve it.
- Choose, and record why — so the decision can be revisited rationally later.
The evaluation set is the thing that keeps paying. Every time a new model appears, re-running it takes an hour and tells you definitively whether switching helps. Without it, every model release becomes an argument based on impressions.
Architecture that keeps you portable
- Abstract model calls behind your own interface — never scatter provider SDK calls through the codebase.
- Keep prompts in configuration, not hard-coded, so they can be tuned per model.
- Do not depend on provider-specific features unless the value clearly justifies the lock-in.
- Version your prompts alongside the model they were tuned for.
- Maintain the evaluation set as a first-class artefact.
Done this way, a provider change is a configuration update plus a test run — rather than a project.
When to consider switching
- Your evaluation set shows a measurable improvement, not a plausible one.
- Cost at your volume has become a genuine constraint.
- Latency is affecting user experience.
- A data policy change makes your current provider unacceptable.
- You need a capability your current model lacks — larger context, better structured output.
Not on that list: a new model topping a benchmark. That is a reason to test, not a reason to switch.
Choosing a model for a production application? Tell us the task and your constraints. See our AI agent service, building evaluation sets, and cost optimisation.