← All guides

Azure AI

Design AI gateway policies that preserve security boundaries

Separate caller authorization, backend identity, semantic-cache isolation, token budgets and private connectivity in Azure API Management.

API ManagementAI gatewaymanaged identitycachingnetworking
A workload reaches an AI resource through a private path while dependencies remain distinct access paths.
Locate the gateway in the actual request path. Gateway policies, backend authorization and downstream connectivity are separate controls.View full-size diagram

An AI gateway concentrates valuable authority: it can authenticate callers, select models, hold backend credentials and return cached responses. That concentration helps enforce consistent controls, but also creates a place where one policy mistake can cross application or tenant boundaries. Review it as a security component with its own deployment and incident procedures.

Product scope and threat model

This guide covers Azure API Management managed gateways and their documented LLM policies. Check the selected tier, gateway and model API against each policy’s support table. For example, llm-token-limit does not list the Consumption tier, and support for particular provider schemas differs. This is not a configuration guide for the separately documented AI Gateway public preview, whose resource type is Microsoft.ApiManagement/aigateways. Do not assume its interface, release status or policy behavior is interchangeable with an APIM service resource. APIM capabilities, preview resource scope.

Threats include an unauthorized application using the gateway’s broader identity, one tenant retrieving another tenant’s cached answer, a caller rotating an untrusted quota key, and a client bypassing the gateway to reach the model directly. Also include a policy editor: configuration access can confer practical use of attached identities even without access to application code.

Before implementation, record the API shape, model deployments, region, gateway tier, caller identity types and backend role assignments. Identify DNS and private-endpoint owners. If caching is proposed, add an external-cache owner, retention policy and approved data classes. Define whether the gateway governs model access only or also makes user-specific authorization decisions; the latter requires an explicit entitlement source.

Separate caller identity from backend identity

Validate incoming Entra tokens before deriving routing, quota or cache attributes. Configure the intended tenant, audience and permitted client/required claims for the API. A token that is correctly signed but intended for another service should not grant access here. Distinguish delegated user scopes from application roles and test both token shapes deliberately. Entra validation policy.

After validation, derive a stable consumer identity from trusted claims or a controlled application mapping. Ignore caller-provided “tenant,” “department” or “premium” headers unless an authenticated upstream has established and protected them. Authorize the requested model/deployment and operation before selecting the backend. An allowed caller should not be able to choose an arbitrary backend URL or obtain a higher-privilege model alias through a path parameter.

For a supported Azure backend, authentication-managed-identity obtains a token for the configured resource and caches it until expiry. Managed gateways can use system-assigned or explicitly selected user-assigned identities. Give that identity only the required backend role and scope; it does not automatically preserve the original user’s permissions. Microsoft warns that policy editors can use attached identities and may expose their tokens. Restrict policy-writing access and keep tokens out of traces. Managed-identity behavior and security warning.

A practical review artifact is a policy-order table: validate caller, establish permitted target, enforce request controls, perform any approved cache lookup, obtain backend identity, forward, then apply response/storage controls. Inspect inherited policies at each scope. A locally correct operation policy can still be undermined by an inherited backend selection or an omitted control.

Design quotas around an accountable consumer

Use a counter key whose meaning survives ordinary retries and cannot be cheaply changed by the caller. Include the tenant/application and intentional budget scope. An IP-only key can combine unrelated users behind a proxy; an unvalidated header allows quota evasion. Decide explicitly whether separate APIs share a budget.

The LLM policy can enforce token rate and period quotas: excess rate produces 429, excess quota 403. Counters with the same key are shared across configured scopes, but token usage is tracked independently per gateway, including regional gateways. Concurrent requests can overshoot before responses arrive. Streaming uses estimated prompt and completion counts; without prompt estimation, a request can reach the backend before exhaustion is recognized. Consequently, a token quota is neither an exact billing meter nor a global spending cap. Token accounting and gateway limits.

Allocate headroom for retries, multiple regions and different model prices. Combine request-volume controls with token controls: a stream of cheap requests can still exhaust connection capacity. Define a client retry policy that respects the returned delay and stops on persistent authorization or quota failures. Monitor gateway and provider usage separately so accounting disagreements are visible.

Treat semantic caching as a data-release decision

Semantic caching returns a previous answer based on vector proximity, not exact equivalence. The lookup policy supports vary-by partitions and warns about incorrect, outdated or unsafe matches. Authenticate before lookup; partition by the actual access boundary, and do not assume one APIM subscription equals one end user. Review whether excluding system messages could merge prompts governed by different instructions. Lookup behavior and partitioning.

For a user-specific retrieval application, prefer disabling response caching unless you can prove the cached answer remains authorized for the current caller. A per-user partition still needs a strategy for permission revocation, changed source data, model versions and policy changes. Consider a controlled policy/data-generation identifier in the partition and invalidate old generations during a release. This is an application design pattern, not automatic entitlement-aware caching supplied by APIM.

Store only approved response classes with a deliberate TTL. The store policy pairs with lookup and supports a storage condition; it does not establish whether an answer is appropriate to share. Keep request rate protection around cache misses/outages so losing the cache does not overload the backend. Cache storage and outage considerations.

Close the network bypass

An inbound private endpoint protects access to the APIM gateway; it does not create outbound access to a private model endpoint. Standard v2 and Premium v2 support outbound VNet integration, while other injection options have distinct tier and endpoint scope. Outbound integration alone leaves inbound endpoints public. Select the documented combination and verify public-network settings separately. Networking models.

Test resolution and connectivity from the actual gateway path, including model, cache and embedding dependencies. Confirm that an external test client cannot use a leftover backend key or public endpoint to bypass the gateway. A private backend plus excessive RBAC still permits unauthorized operations from a compromised allowed caller; network placement supplements authorization.

Acceptance matrix and troubleshooting

Use synthetic prompts and isolated resources. Record policy revision, identity, selected backend, cache decision, status and request identifiers without logging secrets or confidential prompts.

Test Expected boundary Evidence
Wrong audience or tenant Rejected before backend or cache access Validation decision and absent backend request
Allowed caller selects forbidden deployment Denied despite gateway’s backend role Target-authorization result
Similar prompts from different access groups No cross-boundary cached answer Partition and controlled response markers
Permission revoked after cache population Old protected content is unavailable Revocation/invalidation test
Concurrent and streamed requests near limit Observed behavior matches documented estimates Client results and provider usage
Cache unavailable Controlled backend load and predictable failures Rate-limit and backend metrics
Backend public path attempted directly Required gateway path cannot be bypassed External connectivity/authentication result
Backend times out during retry No uncontrolled retry amplification Attempt counts and timeout decisions

For unexpected 403 responses, distinguish caller validation, quota exhaustion, backend RBAC and network restrictions before changing permissions. For unexpected cache hits, inspect identity derivation and policy generation before lowering similarity thresholds. For private-path failures, verify DNS and outbound connectivity before enabling public access. Keep rollback able to disable caching or a backend route without weakening authentication.

This is a documentation-reviewed design, not a deployed or load-tested gateway. Release only after retaining negative-test results, measured concurrency behavior and an owner for emergency policy changes.

Go to the source

References & further reading

  1. Microsoft Learn — AI gateway capabilities in API Management
  2. Microsoft Learn — AI Gateway preview concepts
  3. Microsoft Learn — Validate Microsoft Entra token policy
  4. Microsoft Learn — Authenticate with managed identity policy
  5. Microsoft Learn — LLM token limit policy
  6. Microsoft Learn — LLM semantic cache lookup policy
  7. Microsoft Learn — LLM semantic cache store policy
  8. Microsoft Learn — API Management virtual networking models