Why this matters more than usual
An LLM application can be 100% available and quietly wrong. Uptime dashboards will show green while answer quality degrades after a model update, a content change, or a retrieval regression. Quality has to be instrumented deliberately.
Observability is the difference between finding out from your dashboard and finding out from a customer. Here is what to instrument.
Four categories of signal
| Category | Examples | Answers |
|---|---|---|
| Operational | Latency, error rate, throughput | Is it working? |
| Cost | Tokens, spend per outcome | What is it costing? |
| Quality | Eval scores, refusal rate, confidence | Is it still good? |
| Behavioural | Escalations, retries, user feedback | Are users being served? |
Most teams instrument only the first two. Those are the familiar ones and they catch outages and bill shocks. They do not catch the failure mode specific to AI — a system that responds quickly, cheaply, and with worse answers than it gave last month.
What to log per request
- Request ID correlating every step of a multi-stage interaction.
- The user query and any preprocessing applied.
- Retrieved chunks with their relevance scores and source identifiers.
- The assembled prompt, or a hash plus the variable parts.
- Model, version and parameters.
- The response, plus token counts in and out.
- Tool calls — which, with what arguments, and what returned.
- Latency broken down by stage — retrieval versus generation.
- Cost for the whole interaction.
- Outcome — answered, refused, escalated, errored.
Log the retrieved chunks. When someone reports a bad answer weeks later, that single field usually tells you immediately whether retrieval or generation failed — and without it you are guessing.
Quality signals worth alerting on
| Signal | A change usually means |
|---|---|
| Refusal rate up | Retrieval degraded, or content removed |
| Refusal rate down sharply | Guardrails weakened — answering things it should not |
| Retrieval scores falling | Index drift, or query patterns shifting |
| Escalation rate up | Users not getting what they need |
| Repeat questions in a session | First answer was unsatisfactory |
| Response length shifting | Model behaviour changed |
| Cost per outcome rising | Inefficiency, looping, or context bloat |
Refusal rate is the most underrated single metric. It moves before complaints arrive, it is cheap to compute, and it moves for reasons worth knowing in both directions — up means retrieval is failing, down sharply means your guardrails have loosened.
Continuous evaluation in production
Pre-deployment evaluation proves the system was good at launch. Production evaluation proves it still is:
- Sample live traffic — a percentage of real interactions, scored against your rubric.
- Run the full evaluation set on a schedule, and always after any change.
- Re-run after every model version change. Provider updates can shift behaviour without notice.
- Track scores over time, so gradual drift is visible rather than only sudden breaks.
- Alert on regression beyond a threshold you set.
Tracing multi-step agents
For anything beyond a single call, you need to see the whole chain:
- Every step, in order, with input and output.
- Which tools were called and what they returned.
- Where time and cost accumulated.
- Where the chain diverged from what you expected.
- Retry attempts and why they were triggered.
Without this, diagnosing a failure in a four-step agent means guessing which step went wrong.
Privacy considerations
Your logs now contain user queries and retrieved business content. That is personal and commercially sensitive data with a retention policy attached. Decide deliberately: what is logged in full, what is redacted, how long it is kept, and who can read it. Teams frequently build excellent observability and accidentally create a data-protection problem.
- Redact obvious personal identifiers where you can do so without losing diagnostic value.
- Set a retention period and enforce it.
- Restrict log access as you would any sensitive store.
- Include logs in your deletion process when a user exercises that right.
Starting simply
- Structured logging of the fields above, into whatever you already use.
- A weekly evaluation run, scored and recorded.
- Alerts on refusal rate, cost per outcome, and error rate.
- A dashboard showing quality alongside cost, so trade-offs are visible together.
- Specialist tracing tools only when multi-step complexity demands them.
Running AI in production without visibility into quality? Tell us what you are running. See our AI agent service, evaluation frameworks, and cost optimisation.