Scope note before anything else: healthcare software regulation is complex, jurisdiction-specific, and consequential. Whether a particular use constitutes a regulated medical device, what your obligations are, and what agreements you need are determinations for your regulatory and legal advisers. This article covers engineering architecture that supports compliance — it is not compliance advice.
Healthcare has real AI opportunities and a narrow band where they are appropriate. Understanding that boundary before building saves expensive redirection.
The line that matters most
| Use | Regulatory weight | Practical viability |
|---|---|---|
| Appointment scheduling and reminders | Light | High |
| Insurance and billing workflows | Light | High |
| Clinical documentation assistance | Moderate | High, with review |
| Records summarisation for clinicians | Moderate | Good, with citations |
| Patient-facing triage | Heavy | Requires regulatory pathway |
| Diagnostic support | Heavy — likely device | Formal approval route |
| Treatment recommendation | Heavy — likely device | Formal approval route |
Almost all near-term healthcare AI value is in the top four rows — administrative burden that consumes clinician time without touching clinical judgement.
Where the value actually is
Documentation assistance
Clinicians spend a substantial share of their time on documentation. AI that drafts notes for clinician review — with the clinician editing and signing — reduces that burden without making clinical decisions.
Design requirement: the clinician must review and approve before anything enters the record. Never auto-file.
Records summarisation
Surfacing relevant history from a long record, with every claim linked to its source document. The clinician verifies rather than trusting.
Administrative workflows
Prior authorisation paperwork, insurance queries, appointment coordination, referral letters. High volume, rule-describable, and clinically peripheral.
Internal knowledge access
Staff questions about protocols, policies and procedures — the same internal knowledge base pattern that works in any organisation.
Architecture requirements
- Comprehensive audit logging — who accessed what patient data, when, and why. Non-negotiable and must be reviewable.
- Access control mapped to clinical roles, enforced at the data layer.
- Encryption in transit and at rest.
- Minimum necessary data — send only what the task requires to any external service.
- Human review before anything reaches the record.
- Citations on every factual claim, linked to the source.
- Retention and deletion consistent with your obligations.
Design the audit trail before the feature. In healthcare, being able to demonstrate who saw what and when is frequently more important than the feature itself. Retrofitting audit logging into a working system is difficult; building around it from the start is straightforward.
The vendor questions your compliance team will ask
- Which model provider, on which tier, and what are the data terms?
- Is a business associate agreement available and signed?
- Is content retained by the provider? Used for training?
- Where is processing performed geographically?
- What is logged on the provider side?
- What is the breach notification process and timeline?
Get these answered in contract terms before building, not during a security review afterwards.
Where AI should not be used
Be conservative here, deliberately. Diagnosis, treatment recommendation, triage decisions and medication guidance carry patient-safety consequences and regulatory obligations that make casual deployment inappropriate. If a proposed use could influence a clinical decision, that is a regulatory conversation before it is an engineering one.
- Anything a patient might act on as medical advice.
- Decisions affecting care without clinician review.
- Medication dosing or interaction guidance.
- Triage that could delay urgent care.
- Anything where being confidently wrong could harm someone.
A sensible deployment sequence
- Start administrative — billing, scheduling, internal knowledge. Real value, light regulatory weight.
- Establish audit and access patterns that your compliance function accepts.
- Move to documentation assistance, with mandatory clinician review.
- Add summarisation with citations, for clinician use rather than patient-facing.
- Anything clinical only via a regulatory pathway, led by your regulatory team.
Building healthcare software with AI components? Tell us the workflow — and involve your compliance function early, because it shapes the architecture. See our AI agent service and data privacy architecture.