← All guides

Azure AI

Engineer Key Vault rotation and recovery for AI workloads

Separate secret replacement from cryptographic key rotation, verify consumer adoption, and recover vault dependencies without losing access to protected data.

Key Vaultsecretsrotationrecoverycustomer-managed keys
Create a credential version, verify consumer adoption and retire the old version with a recovery plan.
Rotation is complete only after the consuming workload uses the intended credential and recovery has been tested.View full-size diagram

Creating a new secret version does not prove that the application adopted it. Creating a new encryption-key version does not prove that old data can survive removal of the previous version. Treat rotation as a distributed change with observable states, not a scheduled vault operation. This design applies to AI connectors, retrieval pipelines, and services using customer-managed encryption keys; it has not been exercised in a tenant.

Classify what is being rotated

Maintain an inventory with object type, owning system, consumer identities, versioned or versionless references, cache behavior, expiry, recovery owner, and last verified use. Separate three cases: a credential accepted by an external service, a cryptographic key used for encryption or wrapping, and a certificate with its own issuance/renewal process. Do not copy one lifecycle policy across them.

Prefer managed identity where the target supports it. For remaining secrets, distinguish changing the value in Key Vault from changing the authoritative credential at the destination. Microsoft’s single-credential tutorial updates both systems and explicitly describes a temporary inconsistency window. That sample is a pattern to assess, not a zero-downtime guarantee for every connector. Secret rotation tutorial.

Our rotation record includes an operation ID and states such as prepared, target-updated, vault-updated, consumer-verified, and old-version-retired. Persist the state outside process memory so a retry or operator can determine what happened after a crash. Never put secret values in the record.

Separate rotation authority from consumption

Use one identity to manage the credential lifecycle and a different runtime identity to consume the minimum required object. Restrict the rotator’s destination permissions as well as its vault permissions. A function that can replace every application credential is a powerful control-plane service even if it runs only monthly.

Prefer the RBAC permission model and inspect existing assignments before migrating. In the legacy access-policy model, vault-write authority can permit self-granting of data access. A migration must preserve intended runtime access and remove that escalation path without creating a production outage. Authorization-model boundaries.

Prove adoption before retirement

For a secret with overlapping credentials, our recommended sequence is to create the replacement at the authoritative target, publish the new vault version, refresh consumers, verify successful use, and then revoke the previous credential. Record the maximum refresh interval of every consumer. If the target supports only one credential, explicitly plan the inconsistent interval, bounded retries, and maintenance fallback.

Use a synthetic transaction that exercises the real target operation with harmless data. A successful GetSecret proves only retrieval. A health endpoint that uses an old cached credential can hide a failed rollout. Track the credential identifier or version used where the target exposes safe metadata, without recording the credential itself.

Cryptographic key rotation creates new key material under a new version. Consumers that support automatic refresh typically need a versionless key reference, while decryption must retain the correct historical version relationship. Service adoption timing varies; consult the consuming service before disabling the old key. Scheduled key rotation also has its own operation cost. Key rotation and integration behavior.

For customer-managed encryption, prove both new writes and reads of older data. Retain the earlier key versions for as long as any active data, backup, snapshot, or recovery path needs them. “All new requests succeed” is not sufficient evidence that an old key is no longer necessary.

Recover the surrounding system

Soft-delete preserves recoverable vaults and objects for a configured retention period. Purge protection prevents early permanent removal during that period. The retention setting is chosen at vault creation; recovering a deleted vault does not automatically recreate associated RBAC assignments or Event Grid subscriptions. Deletion and recovery semantics.

Keep reproducible configuration for those dependencies, network access, diagnostic settings, and rotation subscriptions. Restore the vault, restore the necessary integrations, verify the intended permission model, and run consumer checks before declaring recovery complete. Test with disposable resources; never demonstrate recovery by deleting the only production encryption key.

Acceptance tests and operational signals

Failure injected Expected behavior
Duplicate expiry event Existing operation is resumed without uncontrolled repeated rotation
Target succeeds, vault update fails State identifies the split and the documented reconciliation path works
Consumer holds old cached value Adoption monitoring detects it before retirement
New key active, old encrypted object read Historical object remains decryptable
Runtime identity attempts rotation Denied independently of normal read access
Vault recovery in a test environment Role assignments and event subscriptions are restored explicitly
Rotation notification stops Independent expiry monitoring detects loss of the automation path

Alert on failed rotation, approaching expiry without verified adoption, unexpected credential-management activity, and absent scheduled checks. Keep the evidence needed to explain who changed what and which consumers were proven healthy. Continue with workload identity design and AI incident response for authentication and containment decisions.

Go to the source

References & further reading

  1. Microsoft Learn — Configure cryptographic key rotation
  2. Microsoft Learn — Rotate a secret for a single-credential resource
  3. Microsoft Learn — Key Vault soft-delete and recovery
  4. Microsoft Learn — RBAC versus access policies