Encryption at rest and in transit are well-understood. Encryption in use is where AI inference still leaks. Here's how we used confidential computing to close that gap for a regulated workload.
A financial services client wanted to deploy a customer-facing AI assistant that handled account queries. The catch: their compliance posture treated personally identifiable information (PII) in plaintext memory as a finding, regardless of where the memory was.
That's not unreasonable. Inference workloads load model weights into GPU memory and process plaintext PII alongside. The host operating system can read that memory. The cloud provider's hypervisor can read that memory. A privileged insider, in theory, can read that memory. "Encrypted at rest, encrypted in transit, plaintext during inference" is the default - and increasingly, regulated industries don't accept it.
Confidential computing is the answer that's matured to the point of being usable. It's also one of the harder pieces of infrastructure to get right.
Confidential computing isolates data and code inside a hardware-backed Trusted Execution Environment (TEE) - a portion of CPU and, increasingly, GPU memory that even the host operating system cannot read. The TEE provides three guarantees the regular execution environment does not.
Memory confidentiality. Data inside the TEE is encrypted with keys held in hardware. Even with full host access, the contents are not readable.
Memory integrity. The hardware detects tampering with TEE memory and halts execution rather than serving a corrupted state.
Remote attestation. A client can verify, cryptographically, that it is talking to genuine TEE-protected code running an expected version. The model server can prove to the caller that the request will be handled inside the TEE.
The major cloud providers offer this through hardware-backed VMs and, more recently, confidential GPUs. Each has a different SDK and a different attestation flow. None of them are entirely turnkey.
Three things made the inference case harder than the typical confidential-computing demo.
Model weights are large. Loading multi-gigabyte model weights into a TEE-protected GPU is slow on cold start. Cold start of a confidential GPU instance was measurably worse than a standard one and warm pools needed to be sized accordingly.
Attestation has to happen before the first PII byte arrives. The client application must verify the server's attestation evidence before sending the request body. We had to insert an attestation handshake into the API contract - extra latency on first call, cached for subsequent calls.
Observability becomes a leakage risk. Standard observability captures request/response payloads. If the goal is "PII never appears in plaintext outside the TEE," that includes logs, traces and metrics. We had to redesign observability to capture only non-PII metadata from inside the TEE and export aggregates, not raw payloads.
Confidential GPU instances behind an attestation-gated gateway. The inference servers run inside confidential VMs with confidential GPU support. An attestation gateway sits in front of them; clients verify the gateway's certificate chain, which is bound to the underlying hardware attestation. PII only flows after attestation succeeds.
# Simplified client-side attestation check
attestation = gateway.fetch_attestation()
if not verifier.verify(
attestation,
expected_image_hash=KNOWN_INFERENCE_IMAGE_HASH,
expected_policy=PRODUCTION_POLICY,
):
raise AttestationFailure()
# Only now is it safe to send PII-bearing requests
response = gateway.infer(payload, attestation_token=attestation.token)
Key release gated by attestation. The encryption keys used to decrypt user data inside the TEE are released by a separate key management service that itself requires valid attestation evidence. If the inference environment isn't a legitimate TEE running the approved image, the keys aren't released and the data can't be decrypted.
Internal-only telemetry from inside the TEE. Inference servers emit only structured, PII-free metrics - request count, latency, model version, error class. Anything more detailed lives inside the TEE and is purged on instance recycle.
Pinned, signed inference images. The image hash that attestation checks against is part of the deployment manifest. A new inference build means a new attestation policy. This makes supply chain changes visible and reviewable.
We would over-invest in warm-pool sizing from the start. Cold starts on confidential GPU instances were the single biggest performance issue we hit in production and we resized the warm pool three times before settling on the right floor. A capacity model based on observed traffic would have got us there faster.
We would also document the attestation policy as a first-class artefact in the repo, alongside Terraform and application code. Compliance reviews kept asking "what exactly is being attested?" and the answer lived across three different places. A single attestation-policy file, versioned and signed, made every subsequent review faster.
Confidential computing closes the "in use" gap that traditional encryption leaves open. It is not free - cold starts, attestation handshakes and observability redesigns all have real cost. For workloads where PII in plaintext memory is a compliance finding, the cost is paid willingly. For workloads where it isn't, the simpler stack is still the right choice.