Building RAG over PHI is a different engagement from building RAG over public documents. Here's the architecture we used for a healthcare client - what we changed from the default stack and why.
The client was a regional healthcare provider. The ask was an internal assistant for clinicians and care coordinators to query patient records, treatment guidelines and internal protocols.
The technical problem was familiar - retrieval-augmented generation over a mixed corpus of structured and unstructured documents. The compliance problem was the entire reason this project needed a senior team. Protected Health Information (PHI) handling under HIPAA isn't a layer you bolt on; it shapes the architecture from the data pipeline upward.
Default RAG stacks make three assumptions that don't hold under HIPAA.
Embeddings are not safe to send to third-party APIs. Even though an embedding vector isn't human-readable, it can be inverted under the right conditions and is considered a derivative of PHI by most healthcare compliance teams. Sending PHI-derived embeddings to a third-party embedding API is, in practice, sending PHI.
Logs are part of the system. Every API gateway, observability tool and error tracker by default captures request and response bodies. Each of those becomes a PHI store the moment it logs a query that contains a patient name.
Access boundaries cross documents. A clinician on Ward A is authorised to view records for patients on Ward A. The vector index doesn't know about wards. If retrieval returns a chunk from a patient outside the clinician's authorisation, the query has produced an unauthorised disclosure even if the answer is technically correct.
The client's existing data governance gave us a clear PHI inventory and an access matrix. What it didn't give us was guidance on how those access controls interact with vector retrieval, model inference, or response generation. We had to design those interactions and document them for the compliance team.
The architecture had to satisfy three distinct review processes - HIPAA Security Rule compliance, the client's internal data governance board and a third-party penetration test. We designed for all three from the start.
Five elements that diverge from the default RAG playbook.
1. Self-hosted embedding and inference. Both the embedding model and the answer-generation model run on infrastructure the client owns, inside a Business Associate-covered environment. No PHI, no PHI-derived vectors and no PHI-containing prompts cross a network boundary into a third-party API. The performance hit relative to hosted APIs was real but manageable for the volume.
2. Pre-retrieval ACL filtering. Every document chunk in the vector index is tagged with the access scope of its source - patient ID, department, sensitivity level. At query time, the user's effective scope is computed first and retrieval is filtered to chunks the user is authorised to see. Filtering happens before reranking, not after, so unauthorised chunks never enter the candidate set.
def retrieve(query, user_context):
scopes = compute_authorised_scopes(user_context)
candidates = vector_index.search(
embed(query),
filter={"scope": {"$in": scopes}},
k=20,
)
return reranker.rank(query, candidates, top_k=6)
3. PHI redaction in audit logs, not in production data. Application logs, error trackers and analytics tools see a redacted form of every prompt and response - patient identifiers replaced with stable hashes. Production data stores the original. This lets the compliance team review system behaviour without needing PHI clearance for every engineer who reads a log.
4. An access trail keyed by record, not by request. Every PHI access - retrieval, display, copy-out - logs to a tamper-evident audit store with (user, patient_id, record_id, access_type, timestamp). When the compliance team asks "who saw this patient's records last quarter," the answer is one query away. This was non-negotiable for the security review.
5. Hard limits on response generation. The generation step is constrained to citing only retrieved chunks. If the model is about to emit content not grounded in a retrieved source, we fall back to a "no sufficient information" response. Hallucination in a clinical context is a safety issue, not just a quality issue.
We would invite the compliance team into design reviews from week one, not present finished architecture for sign-off. The reviews we held early surfaced concerns we could absorb cheaply. Later reviews surfaced concerns that required rework.
We would also have invested earlier in synthetic PHI test data. Real PHI couldn't be used outside the compliant environment, so our development environment used a synthetic generator. Getting that generator to produce realistic-enough data took longer than expected and our early test results were less representative than they should have been.
A HIPAA-compliant RAG stack isn't a normal RAG stack with extra logging. It's a different architecture - self-hosted inference, pre-retrieval access control, redacted observability and hard guardrails on generation. Design those in from the start. Retrofitting them is more expensive than building them in and the compliance review will find every shortcut you took.