The short answer
Start with RAG. It handles the overwhelming majority of business use cases, keeps your knowledge current, and lets answers cite their sources. Fine-tune only when you have measured a specific behavioural gap that retrieval cannot close.
This decision gets made backwards surprisingly often — teams fine-tune first because it sounds more sophisticated, then discover their model still cannot answer questions about last week's pricing.
What each actually does
| RAG (retrieval) | Fine-tuning | |
|---|---|---|
| Changes | What the model can see | How the model behaves |
| Good for | Facts, documents, current data | Format, tone, task patterns |
| Updating knowledge | Re-index — minutes | Retrain — hours to days |
| Citations | Native — points at the source | Not possible |
| Upfront cost | Moderate engineering | Data preparation plus training |
| Ongoing cost | Retrieval infra, larger prompts | Retraining as behaviour drifts |
| Model portability | Works with any model | Locked to that model family |
The row that decides most business cases is "updating knowledge". Your prices change, your policies change, your product catalogue changes. With retrieval that is a re-index. With fine-tuning it is a retraining cycle — and until you run it, your model is confidently telling customers last quarter's information.
Symptom-based diagnosis
Rather than choosing by architecture, choose by what is actually wrong:
| Symptom | Likely fix |
|---|---|
| Doesn't know our products or policies | RAG |
| Gives outdated information | RAG |
| Cannot cite sources | RAG |
| Right facts, wrong tone | Prompting first, then fine-tuning |
| Won't follow our output format | Prompting first, then fine-tuning |
| Doesn't understand our domain jargon | RAG with a glossary; fine-tuning if persistent |
| Too verbose or too terse | Prompting |
| Inconsistent across similar inputs | Fine-tuning, once prompting is exhausted |
Notice how many rows resolve to prompting. A large share of "we need to fine-tune" conclusions are reached before anyone seriously iterated on the prompt — which costs nothing and takes an afternoon.
Try these in order
- Prompt engineering — free, immediate, and resolves more than expected. Exhaust it first.
- Few-shot examples — showing the model three good outputs often fixes format and tone entirely.
- RAG — for anything factual, current, or requiring citations.
- Prompt caching and context tuning — if cost or latency is the issue rather than quality.
- Fine-tuning — only for a measured, persistent behavioural gap the above cannot close.
The test before fine-tuning: can you write down the specific behaviour you want, and demonstrate with an evaluation set that prompting and few-shot examples do not achieve it? If not, you do not yet have evidence that fine-tuning is the answer — and it is the most expensive option to get wrong.
Where fine-tuning genuinely earns its cost
- Strict output format at scale — when every response must conform exactly and prompting still produces variation.
- A specialised task pattern — classification or extraction in a narrow domain where consistency matters more than flexibility.
- Latency or cost pressure — a fine-tuned smaller model can outperform a larger prompted one on a narrow task, at lower cost per call.
- Deeply domain-specific language — where terminology is genuinely unlike general usage and retrieval alone leaves the model confused.
- Tone that resists prompting — a house style long prompts approximate but never quite hold.
The costs people underestimate
RAG costs
- Chunking and indexing engineering
- Vector store or search infrastructure
- Larger prompts, so higher per-call token cost
- Content hygiene — bad docs, bad answers
- Retrieval quality tuning is real work
Fine-tuning costs
- Training data curation — the largest hidden cost
- Training compute
- Retraining whenever behaviour must change
- Lock-in to one model family
- Evaluation to prove it improved anything
Training data curation deserves emphasis. Producing several hundred consistent, high-quality examples is significant expert time — and inconsistent examples actively make the model worse.
A practical recommendation
For nearly every business application — support agents, internal knowledge assistants, document processing, sales enablement — start with retrieval and disciplined prompting. Measure the result against a real evaluation set.
Revisit fine-tuning only when you can point at a specific, measured behavioural gap that survived prompting. At that point it is a targeted fix rather than an expensive guess.
Deciding between these for a real use case? Describe the symptom you are trying to fix — that usually settles it quickly. See our AI agent service, production RAG architecture, and why prompting is not enough.