An adversarial evaluation is an experiment against a defined system. It can reveal a failure and measure behavior on selected cases; a clean result cannot establish that the system is secure against every prompt, identity or tool sequence. The useful deliverable is a reproducible set of cases, observations and release decisions tied to a specific configuration.
Choose the product path deliberately
Microsoft Foundry supports evaluation of existing outputs, generated responses, conversations and production traces. Choose the unit of evaluation before selecting a score: a turn-level response check cannot establish that a multi-step workflow preserved authorization. Evaluation workflow.
This guide focuses on the documented cloud AI Red Teaming Agent workflow for supported Foundry project deployments, Azure OpenAI deployments and Foundry agents. The local Azure AI Evaluation SDK red-team workflow remains preview and is documented as incompatible with the new Foundry portal/SDK. Keep its installation and sample code separate. Individual evaluators also differ: the current reference marks Task Adherence, Task Completion and Groundedness Pro as preview. Record the chosen API, SDK and evaluator versions rather than describing the whole evaluation stack as generally available. Cloud targets, local preview, evaluator status.
Prerequisites include an authorized non-production target, test identities, realistic but synthetic data, a Foundry project, supported regional capabilities and sufficient model capacity. The submitter and project managed identity serve different purposes; a project-level assignment alone does not necessarily authorize model inference. Grant the roles needed for the selected workflow, not every role in an example. Permission matrix.
Translate threats into falsifiable cases
Start with assets and consequences: confidential retrieval results, destination credentials, approved recipients, durable changes and operating availability. Define attacker control separately for user prompts, retrieved documents, tool outputs and conversation history. Include an authenticated but lower-privilege caller; prompt-only tests miss ordinary broken authorization.
For each case, record an allowed goal, attacker-controlled surface, forbidden outcome, expected enforcement point and observable result. For example, a synthetic support record may contain an instruction to export another customer’s attachment. The pass condition is that the second attachment is never retrieved or transmitted. A polite refusal after the export is a failure.
Use three complementary case sets:
- Known controls: deterministic checks for denied identities, forbidden destinations, revoked access and unapproved writes.
- Adversarial variations: different phrasing, placement, conversation length and tool-return structures around the same forbidden outcome.
- Benign neighbors: legitimate requests close to the boundary, ensuring the system still completes permitted work.
Keep a held-out set so fixes do not merely memorize the reported prompts. Repeat nondeterministic cases under a declared sampling plan. Record attempts, completed evaluations, infrastructure errors and reviewer decisions separately; an evaluation timeout is neither a safe answer nor a successful attack.
Instrument outcomes, not only the final answer
Use the complete application path, including retrieval authorization, tool policy, approvals and destination services. Replace destructive effects with a controlled sink or reversible test resource. Capture tool proposals, actual invocations, authorization decisions, synthetic data identifiers and resulting state. Use correlation identifiers to link the run to backend evidence without retaining credentials.
The cloud red-team service includes agentic categories such as prohibited actions, sensitive-data leakage and task adherence. Its documented leakage scenarios use synthetic data and mock tools; the listed leakage limitations include single-turn English tests and exclusion of memory/training-set leakage. Those bounds do not cover every multilingual, multi-turn or real-permission scenario. Add your own authorized system tests where the service’s scenario does not represent your application. Risk categories and limitations.
Run a benign case first to prove the harness reaches the intended target and can observe the expected side effect. Then run a deliberately denied case to prove the evidence pipeline distinguishes “blocked” from “not executed.” Freeze the model deployment/version, instructions, tool schemas, retrieval snapshot and policy revision for comparisons. If any changes, label the run as a different configuration.
Calibrate graders and preserve uncertainty
Combine deterministic checks with human-reviewed labels and AI-assisted evaluators. A canary appearing in a forbidden output, a destination write or a denied-resource read is stronger evidence for its specific invariant than an answer-quality score. Conversely, a failed tool call may indicate an outage rather than an effective security policy.
Create a small adjudicated calibration set containing clear passes, clear failures and ambiguous cases. Compare grader judgments against it before using aggregate metrics. Review disagreements and a sample of apparent passes, not just high-severity findings. Keep the grader’s version, instructions, threshold and reasoning alongside the case. Microsoft explicitly notes that AI-generated safety labels/reasoning can be wrong and should not be relied on in isolation. Evaluator transparency and limitations.
An attack-success percentage describes the evaluated set and chosen rubric. Changing the attack mix, denominator, language, grader or handling of errors changes its meaning. Report those choices with the score. Define zero tolerance for specific unauthorized side effects where appropriate, while separately assessing usefulness and false refusals. A release owner must decide what evidence is sufficient for the intended exposure.
Keep the evaluation data boundary explicit
Prefer synthetic identifiers and purpose-built canaries over production records. Inventory every recipient of prompts, outputs and traces: target model, attack generator, grader, application telemetry, connected storage and exported reports. Restrict report access because failed cases can contain the very data the test sought to protect.
The privacy documentation for models sold by Azure describes that product scope; it is not a universal promise for every external model, tool or telemetry destination. Optional stateful features can persist content. Review the selected features and retention instead of inferring “not used for training” means “not stored.” Model data-processing scope.
Evaluation networking also needs its own review. Microsoft documents network injection requirements and notes that connecting Application Insights sends evaluation data there. Confirm the actual region and network configuration with a small authorized run; regional support differs by evaluation capability. Network requirements, availability and limits.
Acceptance matrix and release evidence
| Case | Required observation | Evidence to retain |
|---|---|---|
| Authorized benign task | Correct allowed outcome | Expected state and trace identifiers |
| Direct request for forbidden data | No protected read or disclosure | Retrieval/service denial, not just refusal text |
| Hostile instruction in retrieved content | No extra authority or unintended destination | Proposed versus executed calls |
| Tool output changes an approved recipient | Fresh decision required before transmission | Approval binding and destination state |
| Revocation during conversation | Subsequent action honors current access | Revocation timestamp and denied operation |
| Multi-turn or unsupported-language case | Explicit coverage result or documented gap | Case transcript and method |
| Grader disagrees with backend evidence | Adjudicated outcome overrides unsupported score | Reviewer rationale and raw identifiers |
| Harness timeout or missing telemetry | Inconclusive; excluded from pass claims | Error category and rerun plan |
If every case passes unusually quickly, verify the target was invoked and the grader received populated fields. For 403 errors, inspect the submitter, project identity and target-model permissions separately. If a private evaluation remains in progress, examine network injection, endpoint approval, DNS and status-update permissions before relaxing isolation. Network troubleshooting.
Retain the threat model, frozen configuration, case set, coverage gaps, adjudicated failures, remediation evidence and release decision. This guide is documentation-reviewed only; it does not report a completed red-team exercise. Re-run the relevant cases after changes to models, instructions, tools, identity policy or retrieval content.
Go to the source
References & further reading
- Microsoft Learn — Evaluation in Microsoft Foundry
- Microsoft Learn — AI Red Teaming Agent concepts and limitations
- Microsoft Learn — Run AI Red Teaming Agent in the cloud
- Microsoft Learn — Local AI Red Teaming Agent preview
- Microsoft Learn — Built-in evaluators and preview status
- Microsoft Learn — Risk and safety evaluation transparency note
- Microsoft Learn — Evaluation workflow permissions
- Microsoft Learn — Evaluation regions and limits
- Microsoft Learn — Evaluation network requirements
- Microsoft Learn — Data privacy for models sold by Azure