The realistic default
Use APIs. Self-hosting wins on three specific grounds — data control, sustained high volume, and latency requirements APIs cannot meet. If none of those apply to you, running your own inference is buying operational burden you do not need.
This decision is frequently made on principle rather than arithmetic. Here is the arithmetic, and the cases where principle is genuinely the deciding factor.
What each actually costs you
| API | Self-hosted | |
|---|---|---|
| Cost shape | Per token, scales with use | Fixed infrastructure, regardless of use |
| At low volume | Very cheap | Expensive — idle GPUs still cost |
| At sustained high volume | Can become significant | Amortises well |
| Operational burden | None | Substantial and ongoing |
| Model quality ceiling | Frontier models | Best open weights |
| Data leaves your environment | Yes, under vendor terms | No |
| Latency floor | Network round trip | Local, potentially lower |
| Upgrade path | Automatic | You manage it |
The operational burden row is the one that is consistently underestimated. Self-hosting means GPU provisioning, inference server tuning, batching configuration, model version management, capacity planning, monitoring and on-call. That is a real ongoing role, not a one-time setup — and its cost frequently exceeds the API spend it was meant to replace.
The three legitimate reasons to self-host
1. Data cannot leave your environment
Regulatory constraints, contractual commitments to customers, or data residency requirements that no vendor tier satisfies. This is the strongest reason and the one that overrides cost arithmetic entirely.
2. Sustained high throughput
Continuous, predictable, high-volume inference — not bursty. GPUs you have paid for are only economical when they are busy. Variable load is exactly where APIs win, because you pay nothing when idle.
3. Latency requirements APIs cannot meet
Real-time applications where the network round trip is itself the problem. Genuinely rare in business applications, common in embedded and edge contexts.
Notice that "APIs feel expensive" is not on the list. That is a reason to optimise your usage — caching, context trimming, model routing — before it is a reason to buy GPUs.
Reasons that do not survive scrutiny
- "We want control." Control over what, specifically? If you cannot name it, you are buying operational burden for a feeling.
- "Open weights are free." The weights are free; the inference is not. GPU hours cost money whether the model was free or not.
- "We do not want vendor lock-in." A good abstraction layer addresses this. Self-hosting swaps vendor dependency for hardware and ops dependency.
- "It will be cheaper." Model it properly at your real volume, including engineering time, before asserting this.
Modelling it honestly
Include everything, not just the obvious line items:
| API side | Self-hosted side |
|---|---|
| Token spend at projected volume | GPU instances or hardware |
| Growth in volume over the period | Capacity headroom for peaks |
| — | Engineering time for setup and tuning |
| — | Ongoing operations and on-call |
| — | Model evaluation and upgrade work |
| — | Monitoring infrastructure |
Before comparing, optimise the API side. Prompt caching, context trimming and routing simple tasks to smaller models frequently reduce API spend substantially. Comparing self-hosting against an unoptimised API bill overstates the case for switching — sometimes by enough to reverse the decision.
The hybrid that often makes sense
- Self-host the narrow, high-volume task — classification or extraction running constantly, where a smaller open model performs well.
- Use APIs for complex reasoning and anything customer-facing where quality is most visible.
- Self-host anything with hard data constraints, regardless of volume.
- Abstract the routing, so which model handles which task is configuration rather than architecture.
If you do self-host
- Benchmark open models on your task against your evaluation set before committing to infrastructure.
- Size capacity for peak, not average — a queue during your busiest hour is a user-facing failure.
- Use a proper inference server with batching rather than naive per-request loading.
- Plan model updates — new open weights appear regularly and evaluating them is ongoing work.
- Keep an API fallback for capacity overflow and outages.
- Monitor GPU utilisation. Consistently low utilisation means the economics are not working.
Weighing this for a production workload? Tell us your volume, latency needs and data constraints — that usually settles it quickly. See our AI agent service, cost optimisation, and data privacy.