HIPAA wasn’t written for this
HIPAA was written in 1996. Even with later updates (HITECH, the Omnibus Rule), the framework assumes a world of discrete electronic records held in defined systems by defined custodians. LLM-backed products break most of those assumptions: the data flows through model providers, the “system” is a chain of calls, and the custodian relationship is murky.
That doesn’t mean HIPAA doesn’t apply. It does. It just means the standard “we’re SOC 2 + HIPAA” checkbox approach that worked for traditional health-tech SaaS won’t hold up for an AI-backed product. The controls have to be shaped differently.
This post is the architecture that has worked across multiple healthcare AI builds. It is not legal advice. It is the technical posture that gives a healthcare counsel team something to defend.
Principle 1: PHI never reaches a non-BAA model provider
The first decision is binary and non-negotiable: if a model provider has not signed a Business Associate Agreement covering the specific service you’re using, no PHI may flow to that service.
In practice, this means:
- Default routing assumption: BAA-covered only. OpenAI, Anthropic, Google, AWS Bedrock, and Azure OpenAI all offer BAAs for specific tiers. Your default router uses one of those.
- No “occasional” non-BAA usage. Some teams allow non-BAA providers “for testing.” This always leaks. Treat it as a one-way street.
- The BAA covers the specific endpoint, not the company. OpenAI’s BAA covers their enterprise tier; their consumer ChatGPT doesn’t have BAA coverage. Anthropic’s BAA covers their API; Claude.ai consumer doesn’t. The matrix is specific.
The architectural fix is mechanical: the model client layer hard-codes the allowed endpoints. Adding a new endpoint requires a code change and review. No one fat-fingers a non-BAA endpoint into production.
Principle 2: De-identification, where possible, before the call
For many workflows, the model doesn’t need to see PHI to do its job. A clinical summary task often needs the clinical content, not the patient identifier. A scheduling assistant needs the appointment shape, not the diagnosis.
The pattern: a de-identification layer between your application and the model client. It tokenizes identifiers (replace patient name, MRN, DOB with internal tokens), strips fields the model doesn’t need, and re-identifies on the response side if necessary.
Two notes on this:
- De-identification is hard. True HIPAA Safe Harbor de-identification removes 18 categories of identifiers, and even then quasi-identifiers can re-identify in combination. Be modest about claims.
- Expert determination is sometimes worth it. For datasets that are processed in bulk, paying a qualified statistician for an expert-determination assessment of your de-id pipeline pays for itself.
De-identification is a defense-in-depth measure even when you’re using BAA-covered endpoints. It reduces the blast radius if anything goes wrong upstream.
Principle 3: Logging that supports the access accounting requirement
HIPAA’s access accounting requirement (45 CFR 164.528) gives individuals the right to know who accessed their PHI. For traditional EHRs this is well-understood. For AI-backed systems it’s been an afterthought.
The right pattern: every AI call that touches PHI is logged at the model-call level with:
- Patient identifier (or token mapping).
- User who initiated the action.
- Purpose (treatment, payment, operations, etc.).
- Specific data categories accessed.
- Model used, timestamp, outcome.
The log is immutable, retained per the organization’s HIPAA retention policy, and queryable by patient identifier. If a patient requests an access accounting, you can produce a complete record including the AI-mediated touches.
Most healthcare AI products we audit don’t have this. They have generic application logs that show “the chatbot ran” but not “this chatbot accessed Patient X’s records.” The first time a patient requests an accounting, this becomes a problem.
Principle 4: Minimum necessary, enforced in the data retrieval layer
HIPAA’s minimum necessary standard requires that PHI access be limited to what’s needed for the task. For AI systems, this translates to a retrieval discipline: the model receives only the PHI necessary for the specific task it’s doing.
Two patterns work:
- Scoped retrieval at the tool layer. Tools that fetch patient data accept a “purpose” parameter and return only the fields relevant to that purpose. A scheduling tool doesn’t return diagnosis. A clinical summary tool doesn’t return billing.
- Prompt-level redaction. Even within scoped data, irrelevant fields can be removed before they reach the model.
This is more engineering work than it sounds, but it’s the difference between an architecture that defends well to an OCR auditor and one that doesn’t.
Principle 5: Patient-facing AI gets human review on clinical actions
If the AI surface is patient-facing and it can take or recommend clinical actions, the default is human review. We have not yet seen a defensible HIPAA architecture that fully automates clinical recommendations to patients without licensed clinician review.
The architectural shape:
- Patient-facing AI drafts responses and recommendations.
- Clinician reviews and approves before the response is delivered.
- The review is logged (who, when, what changes).
- The patient sees the clinician-approved response, not the raw model output.
This is unsatisfying to product teams that want fully autonomous experiences. It is also the only posture that survives an OCR investigation. Until the regulatory framework evolves, plan for the clinician-in-the-loop pattern.
Principle 6: Breach notification readiness
HIPAA breach notification requires specific actions within specific timelines. AI-backed systems introduce novel breach scenarios that the standard playbook doesn’t cover.
Specifically:
- Prompt-leak breach. A user’s PHI ends up in another user’s response because of a context-mixing bug. Treat as breach. Notification timeline applies.
- Model provider breach. Your BAA-covered provider has a security incident that exposes PHI. The provider must notify you per the BAA; you must notify affected individuals per HIPAA.
- Inferred PHI. Particularly thorny — when the model infers PHI from non-PHI inputs and outputs it inappropriately. Less well-settled legally. Plan for it as if it counts.
The runbook for each of these should be written before the incident, not during. The first time you experience one of these is the wrong time to debate whether it triggers notification.
What this changes about product roadmap
Designing this way affects product velocity. Features that would ship in two weeks on a non-HIPAA product ship in five or six on a HIPAA-aware one. The friction is the cost of doing business in healthcare.
The trap to avoid: treating HIPAA as a final-quarter compliance pass. Teams that defer compliance to the end ship products that have to be rebuilt. Teams that bake HIPAA-aware patterns into the architecture from day one ship slower per feature but build durable products.
If you’re considering whether to enter healthcare with an AI product, this is the calculus to run. Healthcare is one of the largest markets for AI but it is also one of the most demanding. Underestimate the architectural cost and you’ll be the company whose first audit cycle is also their last.
The take
HIPAA-aware AI design is not a layer you add. It’s a shape you build to. The six principles above are the minimum architecture that holds up across the products we’ve shipped. The work is real, the velocity hit is real, and so is the resulting market access. Healthcare AI is a long game; the companies that treat it that way win the decade.
Healthcare AI architecture is one of the most demanding builds we do. Custom AI Builds for healthcare always start with the architecture above. Schedule a call if you’re evaluating a healthcare AI build and want a pre-engagement architecture review.
