The proportion nobody expects
Prompts are roughly 10–15% of a production AI system. The rest is retrieval, evaluation, integration, observability, cost control and failure handling — the engineering that determines whether the thing can be deployed at all.
The industry spent two years talking about prompting, which gave the impression it was the job. For anything reaching production, it is a small and shrinking part of it.
What a production system actually consists of
| Component | Share of effort | What breaks without it |
|---|---|---|
| Data ingestion and preparation | 15–20% | Garbage retrieval, garbage answers |
| Retrieval pipeline | 15–20% | Right answer never reaches the model |
| Prompting | 10–15% | Inconsistent format and tone |
| Integration with your systems | 15–20% | Cannot access real data |
| Evaluation | 10–15% | No basis for a go-live decision |
| Guardrails and safety | 10% | Cannot pass security review |
| Observability and cost control | 8–12% | Silent degradation; runaway bills |
Data preparation being the largest item surprises people. Extracting text cleanly from real documents, preserving structure, deduplicating, and keeping content current is unglamorous work that caps everything downstream. No prompt recovers from a corpus where half the PDFs were extracted badly.
What prompting genuinely fixes
- Output format and structure.
- Tone and register.
- Instruction adherence — what to do and what to avoid.
- Reasoning approach for a given task.
- How to behave when information is insufficient.
What prompting cannot fix
- Missing information. If retrieval did not find it, no instruction conjures it.
- Bad source content. Outdated documentation produces outdated answers, confidently.
- Absent system access. The model cannot look up an order without a tool that does.
- Unmeasured quality. Prompting without evaluation is changing things and hoping.
- Cost problems. A prompt cannot reduce a bill driven by oversized retrieved context.
- Security exposure. "Ignore malicious instructions" is not a security control.
Teams stuck iterating endlessly on prompts are almost always fighting a retrieval or data problem. The symptom presents at the prompt; the cause is upstream.
Diagnosing where the real problem is
| Symptom | Usually blamed on | Usually caused by |
|---|---|---|
| Wrong or vague answers | The prompt | Retrieval missing the content |
| Outdated information | The model | Stale source documents |
| Inconsistent quality | The model | No evaluation, so drift is invisible |
| High cost | The model's pricing | Retrieving far more context than needed |
| Slow responses | The model | Sequential calls; oversized prompts |
| Wrong tool called | The model | Ambiguous tool descriptions |
The diagnostic that resolves most arguments: take a failing case and manually paste the correct information into the prompt. If the model then answers well, your problem is retrieval — and no amount of prompt tuning will fix it. If it still fails, the prompt or the model is genuinely the issue.
What to build alongside the prompt
- An evaluation set — real cases with expected outcomes. Build it first; it makes every later decision measurable.
- Retrieval scored separately from generation, so failures are attributable.
- Tracing — what was retrieved, what was sent, what came back, what it cost.
- Refusal and handoff behaviour, tested deliberately.
- Cost attribution per feature.
- Regression gating — no prompt change ships without the evaluation passing.
What to look for when hiring
- Software engineers who understand LLM behaviour — not prompt specialists without engineering depth.
- Experience with retrieval systems and search relevance.
- Comfort with evaluation and measurement as a discipline.
- Systems integration capability, since most of the work is connecting things.
- Observability instincts — knowing what to instrument before it breaks.
Iterating on prompts without the results improving? Describe the failure pattern — the cause is usually upstream. See our AI agent service, RAG architecture, and evaluation frameworks.