What makes AI risk different
Not that agents make mistakes — people do too. It is speed and scale. A person makes one error and notices. An unbounded agent can make the same error a thousand times in a minute. Containment matters more than prevention.
This is a practical framework for deploying autonomous agents in a way your risk function can accept, written around the assumption that the system will eventually behave unexpectedly.
Classify by consequence first
Before any technical control, map what the agent could do:
| Class | Example | Default posture |
|---|---|---|
| Reversible, internal | Update a note, create a draft | Autonomous, logged |
| Reversible, visible | Change a record customers see | Autonomous with notification |
| Irreversible, low value | Send an internal message | Confirm or batch-approve |
| Irreversible, external | Email a customer | Always confirm |
| Financial | Refund, payment, credit | Always confirm, with limits |
| Destructive | Delete records | Confirm, or do not expose at all |
The bottom row deserves a hard look. Ask whether the agent needs deletion capability at all. Soft-delete or archive achieves the business purpose while removing the worst outcome from the possible set. The most reliable control is not exposing the capability.
The containment controls
1. Bounded tool scope
The primary defence. If the agent's tools cannot delete, cannot send unrestricted outbound messages, and cannot spend beyond a limit, then even a fully manipulated agent has a small blast radius.
2. Rate and volume limits
- Actions per minute, per hour, per day.
- Total actions per session.
- Distinct-recipient limits on anything outbound.
- Hard stop and alert on breach, not a soft warning.
3. Spend caps
Both model API spend and any business spend the agent can authorise. A loop that costs money is the most common runaway.
4. Approval gates on irreversible actions
Per-action, not per-session. Session-level approval means one click authorising unknown future actions, which is not meaningful consent.
5. Sandboxed execution
If the agent runs code or accesses external resources, it should do so in an isolated environment with no access to production credentials or internal networks.
6. A tested kill switch
Practise the shutdown before you need it. "We would revoke the key" is a plan, not a capability. Run a drill: disable the agent, confirm it stops within seconds, and confirm in-flight actions are handled safely. The first time you do this should not be during an incident.
Granting autonomy incrementally
Autonomy should be earned against evidence, not granted on optimism:
- Stage 1 — shadow mode. The agent proposes; humans decide. Nothing it produces takes effect. Collect agreement data.
- Stage 2 — approval on everything. Actions execute after a human clicks. Measure override rate by action category.
- Stage 3 — selective autonomy. Auto-approve categories where the record shows near-total agreement. Keep gates elsewhere.
- Stage 4 — steady state. Routine actions autonomous, irreversible actions permanently gated, ongoing sample audit.
The organisations that get this wrong skip to stage three on the strength of a good demo. The ones that succeed treat every relaxation as a decision requiring evidence.
Monitoring for the failure you did not predict
- Volume anomalies — a sudden spike in actions is the earliest signal of a loop.
- Repeated identical actions — a strong indicator something is stuck.
- Rising error or retry rates — the system struggling before it fails visibly.
- Confidence distribution shifts — often precedes a quality drop.
- Cost per outcome climbing — inefficiency or looping.
- Escalation rate changes — in either direction, worth investigating.
Incident response, prepared in advance
- Detect — automated alerting on the signals above.
- Contain — kill switch, tested and documented.
- Assess — the audit log tells you exactly what was done and to whom.
- Remediate — reverse what can be reversed; notify where it cannot.
- Diagnose — traces show which step failed and why.
- Adjust — thresholds, tool scope, or gates. Then re-enable deliberately.
Having this written before deployment is what turns an incident into an inconvenience rather than a crisis.
What to document for governance
- The full list of actions the agent can take, classified by consequence.
- Which are gated and which are autonomous, with the rationale.
- Rate, volume and spend limits.
- Who can change those settings.
- The kill switch procedure and who can invoke it.
- Audit log schema and retention.
- Incident response steps and escalation contacts.
Deploying autonomous agents and needing a defensible risk position? Tell us what actions are involved. See our agentic AI service, approval gate design, and guardrails to demand.