-
Notifications
You must be signed in to change notification settings - Fork 1.3k
.NET: Add Foundry Evaluation samples (Safety + Quality) #3697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
bcb947c
Initial plan
Copilot 5948eff
Add Foundry evaluation samples for Red Teaming and Self-Reflection
Copilot 1dd32f2
Refactor evaluation samples with real implementations in local functions
Copilot bfea890
Uncomment function signatures and bodies, keep only invocations comme…
Copilot 14ffe13
Update Foundry evaluation samples with observability support
rogerbarreto af12ba1
Restructure evaluation samples to follow FoundryAgents naming convention
rogerbarreto 5b83ab0
Rewrite Step01 to use Azure.AI.Projects RedTeam API and address revie…
rogerbarreto 5fb5a8a
Add note about agent-targeted red teaming limitations in README
rogerbarreto 006e225
Add classic Foundry disclaimer to red teaming sample README
rogerbarreto 1d4c9a0
Address PR review comments on Step02 SelfReflection
rogerbarreto 8b154a1
Merge branch 'main' into copilot/support-foundry-observability
rogerbarreto File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 changes: 16 additions & 0 deletions
16
...ryAgents_Evaluations_Step01_RedTeaming/FoundryAgents_Evaluations_Step01_RedTeaming.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFrameworks>net10.0</TargetFrameworks> | ||
|
|
||
| <Nullable>enable</Nullable> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Azure.AI.Projects" /> | ||
| <PackageReference Include="Azure.Identity" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
100 changes: 100 additions & 0 deletions
100
...mples/GettingStarted/FoundryAgents/FoundryAgents_Evaluations_Step01_RedTeaming/Program.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| // This sample demonstrates how to use Azure AI Foundry's Red Teaming service to assess | ||
| // the safety and resilience of an AI model against adversarial attacks. | ||
| // | ||
| // It uses the RedTeam API from Azure.AI.Projects to run automated attack simulations | ||
| // with various attack strategies (encoding, obfuscation, jailbreaks) across multiple | ||
| // risk categories (Violence, HateUnfairness, Sexual, SelfHarm). | ||
| // | ||
| // For more details, see: | ||
| // https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent | ||
|
|
||
| using Azure.AI.Projects; | ||
| using Azure.Identity; | ||
|
|
||
| string endpoint = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_PROJECT_ENDPOINT") ?? throw new InvalidOperationException("AZURE_FOUNDRY_PROJECT_ENDPOINT is not set."); | ||
| string deploymentName = Environment.GetEnvironmentVariable("AZURE_FOUNDRY_PROJECT_DEPLOYMENT_NAME") ?? "gpt-4o-mini"; | ||
|
|
||
| Console.WriteLine("=" + new string('=', 79)); | ||
| Console.WriteLine("RED TEAMING EVALUATION SAMPLE"); | ||
| Console.WriteLine("=" + new string('=', 79)); | ||
| Console.WriteLine(); | ||
|
|
||
| // Initialize Azure credentials and clients | ||
| // WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production. | ||
| // In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid | ||
| // latency issues, unintended credential probing, and potential security risks from fallback mechanisms. | ||
| DefaultAzureCredential credential = new(); | ||
| AIProjectClient aiProjectClient = new(new Uri(endpoint), credential); | ||
|
|
||
| // Configure the target model for red teaming | ||
| AzureOpenAIModelConfiguration targetConfig = new(deploymentName); | ||
|
|
||
| // Create the red team run configuration | ||
| RedTeam redTeamConfig = new(targetConfig) | ||
| { | ||
| DisplayName = "FinancialAdvisor-RedTeam", | ||
| ApplicationScenario = "A financial advisor assistant that provides general financial advice and information.", | ||
| NumTurns = 3, | ||
| RiskCategories = | ||
| { | ||
| RiskCategory.Violence, | ||
| RiskCategory.HateUnfairness, | ||
| RiskCategory.Sexual, | ||
| RiskCategory.SelfHarm, | ||
| }, | ||
| AttackStrategies = | ||
| { | ||
| AttackStrategy.Easy, | ||
| AttackStrategy.Moderate, | ||
| AttackStrategy.Jailbreak, | ||
| }, | ||
| }; | ||
|
|
||
| Console.WriteLine($"Target model: {deploymentName}"); | ||
| Console.WriteLine("Risk categories: Violence, HateUnfairness, Sexual, SelfHarm"); | ||
| Console.WriteLine("Attack strategies: Easy, Moderate, Jailbreak"); | ||
| Console.WriteLine($"Simulation turns: {redTeamConfig.NumTurns}"); | ||
| Console.WriteLine(); | ||
|
|
||
| // Submit the red team run to the service | ||
| Console.WriteLine("Submitting red team run..."); | ||
| RedTeam redTeamRun = await aiProjectClient.RedTeams.CreateAsync(redTeamConfig); | ||
|
|
||
| Console.WriteLine($"Red team run created: {redTeamRun.Name}"); | ||
| Console.WriteLine($"Status: {redTeamRun.Status}"); | ||
| Console.WriteLine(); | ||
|
|
||
| // Poll for completion | ||
| Console.WriteLine("Waiting for red team run to complete (this may take several minutes)..."); | ||
| while (redTeamRun.Status != "Completed" && redTeamRun.Status != "Failed" && redTeamRun.Status != "Canceled") | ||
| { | ||
| await Task.Delay(TimeSpan.FromSeconds(15)); | ||
| redTeamRun = await aiProjectClient.RedTeams.GetAsync(redTeamRun.Name); | ||
| Console.WriteLine($" Status: {redTeamRun.Status}"); | ||
| } | ||
|
|
||
| Console.WriteLine(); | ||
|
|
||
| if (redTeamRun.Status == "Completed") | ||
| { | ||
| Console.WriteLine("Red team run completed successfully!"); | ||
| Console.WriteLine(); | ||
| Console.WriteLine("Results:"); | ||
| Console.WriteLine(new string('-', 80)); | ||
| Console.WriteLine($" Run name: {redTeamRun.Name}"); | ||
| Console.WriteLine($" Display name: {redTeamRun.DisplayName}"); | ||
| Console.WriteLine($" Status: {redTeamRun.Status}"); | ||
|
|
||
| Console.WriteLine(); | ||
| Console.WriteLine("Review the detailed results in the Azure AI Foundry portal:"); | ||
| Console.WriteLine($" {endpoint}"); | ||
| } | ||
| else | ||
| { | ||
| Console.WriteLine($"Red team run ended with status: {redTeamRun.Status}"); | ||
| } | ||
|
|
||
| Console.WriteLine(); | ||
| Console.WriteLine(new string('=', 80)); | ||
101 changes: 101 additions & 0 deletions
101
...tingStarted/FoundryAgents/FoundryAgents_Evaluations_Step01_RedTeaming/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # Red Teaming with Azure AI Foundry (Classic) | ||
|
|
||
| > [!IMPORTANT] | ||
| > This sample uses the **classic Azure AI Foundry** red teaming API (`/redTeams/runs`) via `Azure.AI.Projects`. Results are viewable in the classic Foundry portal experience. The **new Foundry** portal's red teaming feature uses a different evaluation-based API that is not yet available in the .NET SDK. | ||
|
|
||
| This sample demonstrates how to use Azure AI Foundry's Red Teaming service to assess the safety and resilience of an AI model against adversarial attacks. | ||
|
|
||
| ## What this sample demonstrates | ||
|
|
||
| - Configuring a red team run targeting an Azure OpenAI model deployment | ||
| - Using multiple `AttackStrategy` options (Easy, Moderate, Jailbreak) | ||
| - Evaluating across `RiskCategory` categories (Violence, HateUnfairness, Sexual, SelfHarm) | ||
| - Submitting a red team scan and polling for completion | ||
| - Reviewing results in the Azure AI Foundry portal | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| Before you begin, ensure you have the following prerequisites: | ||
|
|
||
| - .NET 10 SDK or later | ||
| - Azure AI Foundry project (hub and project created) | ||
| - Azure OpenAI deployment (e.g., gpt-4o or gpt-4o-mini) | ||
| - Azure CLI installed and authenticated (for Azure credential authentication) | ||
|
|
||
| ### Regional Requirements | ||
|
|
||
| Red teaming is only available in regions that support risk and safety evaluators: | ||
| - **East US 2**, **Sweden Central**, **US North Central**, **France Central**, **Switzerland West** | ||
|
|
||
| ### Environment Variables | ||
|
|
||
| Set the following environment variables: | ||
|
|
||
| ```powershell | ||
| $env:AZURE_FOUNDRY_PROJECT_ENDPOINT="https://your-project.services.ai.azure.com/api/projects/your-project" # Replace with your Azure Foundry project endpoint | ||
| $env:AZURE_FOUNDRY_PROJECT_DEPLOYMENT_NAME="gpt-4o-mini" # Optional, defaults to gpt-4o-mini | ||
| ``` | ||
|
|
||
| ## Run the sample | ||
|
|
||
| Navigate to the sample directory and run: | ||
|
|
||
| ```powershell | ||
| cd dotnet/samples/GettingStarted/FoundryAgents/FoundryAgents_Evaluations_Step01_RedTeaming | ||
| dotnet run | ||
| ``` | ||
|
|
||
| ## Expected behavior | ||
|
|
||
| The sample will: | ||
|
|
||
| 1. Configure a `RedTeam` run targeting the specified model deployment | ||
| 2. Define risk categories and attack strategies | ||
| 3. Submit the scan to Azure AI Foundry's Red Teaming service | ||
| 4. Poll for completion (this may take several minutes) | ||
| 5. Display the run status and direct you to the Azure AI Foundry portal for detailed results | ||
|
|
||
| ## Understanding Red Teaming | ||
|
|
||
| ### Attack Strategies | ||
|
|
||
| | Strategy | Description | | ||
| |----------|-------------| | ||
| | Easy | Simple encoding/obfuscation attacks (ROT13, Leetspeak, etc.) | | ||
| | Moderate | Moderate complexity attacks requiring an LLM for orchestration | | ||
| | Jailbreak | Crafted prompts designed to bypass AI safeguards (UPIA) | | ||
|
|
||
| ### Risk Categories | ||
|
|
||
| | Category | Description | | ||
| |----------|-------------| | ||
| | Violence | Content related to violence | | ||
| | HateUnfairness | Hate speech or unfair content | | ||
| | Sexual | Sexual content | | ||
| | SelfHarm | Self-harm related content | | ||
|
|
||
| ### Interpreting Results | ||
|
|
||
| - Results are available in the Azure AI Foundry portal (**classic view** — toggle at top-right) under the red teaming section | ||
| - Lower Attack Success Rate (ASR) is better — target ASR < 5% for production | ||
| - Review individual attack conversations to understand vulnerabilities | ||
|
|
||
| ### Current Limitations | ||
|
|
||
| > [!NOTE] | ||
| > - The .NET Red Teaming API (`Azure.AI.Projects`) currently supports targeting **model deployments only** via `AzureOpenAIModelConfiguration`. The `AzureAIAgentTarget` type exists in the SDK but is consumed by the **Evaluation Taxonomy** API (`/evaluationtaxonomies`), not by the Red Teaming API (`/redTeams/runs`). | ||
| > - Agent-targeted red teaming with agent-specific risk categories (Prohibited actions, Sensitive data leakage, Task adherence) is documented in the [concept docs](https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent) but is not yet available via the public REST API or .NET SDK. | ||
| > - Results from this API appear in the **classic** Azure AI Foundry portal view. The new Foundry portal uses a separate evaluation-based system with `eval_*` identifiers. | ||
|
|
||
| ## Related Resources | ||
|
|
||
| - [Azure AI Red Teaming Agent](https://learn.microsoft.com/azure/ai-foundry/concepts/ai-red-teaming-agent) | ||
| - [RedTeam .NET API Reference](https://learn.microsoft.com/dotnet/api/azure.ai.projects.redteam?view=azure-dotnet-preview) | ||
| - [Risk and Safety Evaluations](https://learn.microsoft.com/azure/ai-foundry/concepts/evaluation-metrics-built-in#risk-and-safety-evaluators) | ||
|
|
||
| ## Next Steps | ||
|
|
||
| After running red teaming: | ||
| 1. Review attack results and strengthen agent guardrails | ||
| 2. Explore the Self-Reflection sample (FoundryAgents_Evaluations_Step02_SelfReflection) for quality assessment | ||
| 3. Set up continuous red teaming in your CI/CD pipeline |
25 changes: 25 additions & 0 deletions
25
..._Evaluations_Step02_SelfReflection/FoundryAgents_Evaluations_Step02_SelfReflection.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFrameworks>net10.0</TargetFrameworks> | ||
|
|
||
| <Nullable>enable</Nullable> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Azure.AI.OpenAI" /> | ||
| <PackageReference Include="Azure.AI.Projects" /> | ||
| <PackageReference Include="Azure.Identity" /> | ||
| <PackageReference Include="Microsoft.Extensions.AI.Evaluation" /> | ||
| <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Quality" /> | ||
| <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Safety" /> | ||
| <PackageReference Include="Microsoft.Extensions.AI.OpenAI" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\..\..\src\Microsoft.Agents.AI.AzureAI\Microsoft.Agents.AI.AzureAI.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.