[Failing test]: Aspire.Hosting.Azure.Tests.AzureDeployerTests fail: `the given key 'AZURE_CONTAINER_REGISTRY_NAME' was not present in the dictionary.`

[radical]

Is there an existing issue for this?

• I have searched the existing issues

Build information

Build:

• Build URL: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1161145&view=ms.vss-test-web.build-test-results-tab&runId=32113528&resultId=101295&paneView=debug
• Console Log: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-aspire-refs-heads-copilot-fix-24cb4838bb2e5d7eb0a64615bc/Aspire.Hosting.Azure.Tests/1/console.b87ecd36.log
• CI System: Azure DevOps (Helix)
• OS: Windows 11 (10.0.26100-SP0)
• Target Framework: net8.0 | x64
• Aspire Version: 9.6.0-ci
• Build: 20250929.20
  Build error leg or test failing:
  Aspire.Hosting.Azure.Tests.AzureDeployerTests
  Pull Request: N/A

Fill in the error message template

Error Message

Multiple tests in Aspire.Hosting.Azure.Tests.AzureDeployerTests are failing exclusively on Azure DevOps (passing on GitHub Actions workflows) with KeyNotFoundException when trying to access Azure resource configuration keys.

Primary Failures (KeyNotFoundException)

All 7 tests fail with the same root cause - missing Azure resource configuration keys:

1. DeployAsync_WithMultipleComputeEnvironments_Works (line 370) - Missing: AZURE_CONTAINER_REGISTRY_NAME
2. DeployAsync_WithContainer_Works (line 188) - Missing: AZURE_CONTAINER_REGISTRY_NAME
3. DeployAsync_WithProjectResource_Works (line 291) - Missing: AZURE_CONTAINER_REGISTRY_NAME
4. DeployAsync_WithDockerfile_Works (line 235) - Missing: AZURE_CONTAINER_REGISTRY_NAME
5. DeployAsync_WithOnlyAzureResources_PrintsDashboardUrl (line 639) - Missing: AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN
6. DeployAsync_WithSingleRedisCache_CallsDeployingComputeResources (line 589) - Missing: AZURE_CONTAINER_REGISTRY_NAME
7. DeployAsync_WithAzureFunctionsProject_Works (line 891) - Missing: AZURE_CONTAINER_REGISTRY_NAME

Secondary Failures (Missing Project Files)

2 additional tests failed due to missing test project files:

• PublishAsAzureAppServiceWebsite_ThrowsIfNoEnvironment
• MultipleAzureAppServiceEnvironmentsSupported

Stack Trace

System.Collections.Generic.KeyNotFoundException : The given key 'AZURE_CONTAINER_REGISTRY_NAME' was not present in the dictionary.
    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
    at Aspire.Hosting.Azure.Tests.AzureDeployerTests.DeployAsync_WithMultipleComputeEnvironments_Works() in D:\a\_work\1\s\tests\Aspire.Hosting.Azure.Tests\AzureDeployerTests.cs:line 370
    --- End of stack trace from previous location ---

Similar stack traces occur for other failing tests at different line numbers (188, 235, 291, 589, 639, 891).

Notable Environment Differences (AzDO vs GitHub Actions)

• Docker Unavailability: Could not harvest all abandoned containers {"Error": "exec: \"docker\": executable file not found in %PATH%"}
• User Secrets Path Issues (warnings):

warn: Aspire.Hosting.Azure.Provisioning.Internal.DefaultUserSecretsManager[0]
      Failed to save user secrets.
      System.InvalidOperationException: User secrets path could not be determined.

Other info

Test Context and Environment

The tests are using mock Azure provisioning with test credentials:
info: Aspire.Hosting.Azure.Provisioning.Internal.PublishModeProvisioningContextProvider[0]
Default subscription: Test Subscription (/subscriptions/12345678-1234-1234-1234-123456789012)
info: Aspire.Hosting.Azure.Provisioning.Internal.PublishModeProvisioningContextProvider[0]
Tenant: 87654321-4321-4321-4321-210987654321
info: Aspire.Hosting.Azure.Provisioning.Internal.PublishModeProvisioningContextProvider[0]
Using existing resource group test-rg.

Analysis

Root Cause: The tests expect Azure resource configuration keys (primarily AZURE_CONTAINER_REGISTRY_NAME and AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN) to be present in a dictionary during the deployment provisioning phase, but these keys are not being populated in the AzDO/Helix environment.

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Last 10 failures:

Run date	Build Type	Test Name	OS

Summary

Last 24 hrs	Last 7 days	Last 30 days	Most Affected OS	Failure Rate
0	0	0	N/A	0.0%

-- Updated on 10/27/2025 5:54:46 AM UTC

[radical]

cc @mitchdenny @eerhardt

[captainsafia]

The fact that these are failing only in AzDO is sus. These tests work by mocking out the data response that would expect to get from ARM so that it can be explicitly set in output state (ref). The fact that all the tests are failing in AzDo indicates that something might be off with the way the mocks are setup.

We can probably debug this by adding additional logs to BicepProvisioner and verifying that the codepaths that take the ARM outputs and set them on the resource are actually getting invoked.

[mitchdenny]

Just hit this on a GitHub actions run. So not unique to AzDO.

[mitchdenny]

Disabled the test entirely in this PR: #11870

[radical]

Another instance Aspire.Hosting.Azure.Tests.AzureDeployerTests.DeployAsync_WithOnlyAzureResources_PrintsDashboardUrl :

        System.InvalidOperationException: Step 'provision-infra' failed: Collection was modified; enumeration operation may not execute.
         ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
           at System.Collections.Generic.List`1.Enumerator.MoveNext()
           at System.Text.Json.JsonPropertyDictionary`1.ContainsProperty(String propertyName)
           at System.Text.Json.JsonPropertyDictionary`1.TryAddValue(String propertyName, T value)
           at System.Text.Json.JsonPropertyDictionary`1.Add(String propertyName, T value)
           at System.Text.Json.Nodes.JsonObject.Add(String propertyName, JsonNode value)
           at System.Collections.Generic.CollectionExtensions.TryAdd[TKey,TValue](IDictionary`2 dictionary, TKey key, TValue value)
           at Aspire.Hosting.Azure.JsonExtensions.Prop(JsonNode obj, String key) in /_/src/Aspire.Hosting.Azure/Provisioning/JsonExtensions.cs:line 24
           at Aspire.Hosting.Azure.Provisioning.BicepProvisioner.GetOrCreateResourceAsync(AzureBicepResource resource, ProvisioningContext context, CancellationToken cancellationToken) in /_/src/Aspire.Hosting.Azure/Provisioning/Provisioners/BicepProvisioner.cs:line 225
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 242
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 265
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 267
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 270
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 271
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 330
           --- End of inner exception stack trace ---
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 335
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 125
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 137
           at Aspire.Hosting.Publishing.Publisher.PublishAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/Publisher.cs:line 163
           at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 41
  fail: Microsoft.Extensions.Hosting.Internal.Host[9]
        BackgroundService failed
        Aspire.Hosting.DistributedApplicationException: Publishing failed exception message: Step 'provision-infra' failed: Collection was modified; enumeration operation may not execute.
         ---> System.InvalidOperationException: Step 'provision-infra' failed: Collection was modified; enumeration operation may not execute.
         ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
           at System.Collections.Generic.List`1.Enumerator.MoveNext()
           at System.Text.Json.JsonPropertyDictionary`1.ContainsProperty(String propertyName)
           at System.Text.Json.JsonPropertyDictionary`1.TryAddValue(String propertyName, T value)
           at System.Text.Json.JsonPropertyDictionary`1.Add(String propertyName, T value)
           at System.Text.Json.Nodes.JsonObject.Add(String propertyName, JsonNode value)
           at System.Collections.Generic.CollectionExtensions.TryAdd[TKey,TValue](IDictionary`2 dictionary, TKey key, TValue value)
           at Aspire.Hosting.Azure.JsonExtensions.Prop(JsonNode obj, String key) in /_/src/Aspire.Hosting.Azure/Provisioning/JsonExtensions.cs:line 24
           at Aspire.Hosting.Azure.Provisioning.BicepProvisioner.GetOrCreateResourceAsync(AzureBicepResource resource, ProvisioningContext context, CancellationToken cancellationToken) in /_/src/Aspire.Hosting.Azure/Provisioning/Provisioners/BicepProvisioner.cs:line 225
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 242
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 265
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 267
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 270
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 271
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 330
           --- End of inner exception stack trace ---
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 335
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 125
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 137
           at Aspire.Hosting.Publishing.Publisher.PublishAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/Publisher.cs:line 163
           at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 41
           --- End of inner exception stack trace ---
           at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 68
           at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
  crit: Microsoft.Extensions.Hosting.Internal.Host[10]
        The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
        Aspire.Hosting.DistributedApplicationException: Publishing failed exception message: Step 'provision-infra' failed: Collection was modified; enumeration operation may not execute.
         ---> System.InvalidOperationException: Step 'provision-infra' failed: Collection was modified; enumeration operation may not execute.
         ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
           at System.Collections.Generic.List`1.Enumerator.MoveNext()
           at System.Text.Json.JsonPropertyDictionary`1.ContainsProperty(String propertyName)
           at System.Text.Json.JsonPropertyDictionary`1.TryAddValue(String propertyName, T value)
           at System.Text.Json.JsonPropertyDictionary`1.Add(String propertyName, T value)
           at System.Text.Json.Nodes.JsonObject.Add(String propertyName, JsonNode value)
           at System.Collections.Generic.CollectionExtensions.TryAdd[TKey,TValue](IDictionary`2 dictionary, TKey key, TValue value)
           at Aspire.Hosting.Azure.JsonExtensions.Prop(JsonNode obj, String key) in /_/src/Aspire.Hosting.Azure/Provisioning/JsonExtensions.cs:line 24
           at Aspire.Hosting.Azure.Provisioning.BicepProvisioner.GetOrCreateResourceAsync(AzureBicepResource resource, ProvisioningContext context, CancellationToken cancellationToken) in /_/src/Aspire.Hosting.Azure/Provisioning/Provisioners/BicepProvisioner.cs:line 225
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 242
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 265
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.<>c__DisplayClass16_1.<<ProvisionAzureBicepResourcesAsync>b__2>d.MoveNext() in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 267
        --- End of stack trace from previous location ---
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 270
           at Aspire.Hosting.Azure.AzureEnvironmentResource.ProvisionAzureBicepResourcesAsync(DeployingContext context, ProvisioningContext provisioningContext) in /_/src/Aspire.Hosting.Azure/AzureEnvironmentResource.cs:line 271
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 330
           --- End of inner exception stack trace ---
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteStepAsync(PipelineStep step, DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 335
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 125
           at Aspire.Hosting.Pipelines.DistributedApplicationPipeline.ExecuteAsync(DeployingContext context) in /_/src/Aspire.Hosting/Pipelines/DistributedApplicationPipeline.cs:line 137
           at Aspire.Hosting.Publishing.Publisher.PublishAsync(DistributedApplicationModel model, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Publishing/Publisher.cs:line 163
           at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 41
           --- End of inner exception stack trace ---
           at Aspire.Hosting.DistributedApplicationRunner.ExecuteAsync(CancellationToken stoppingToken) in /_/src/Aspire.Hosting/DistributedApplicationRunner.cs:line 68
           at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

  failed Aspire.Hosting.Azure.Tests.AzureDeployerTests.DeployAsync_WithOnlyAzureResources_PrintsDashboardUrl (9s 924ms)
    Xunit.Runner.InProc.SystemConsole.TestingPlatform.XunitException: System.Collections.Generic.KeyNotFoundException : The given key 'AZURE_CONTAINER_APPS_ENVIRONMENT_DEFAULT_DOMAIN' was not present in the dictionary.
      at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
      at Aspire.Hosting.Azure.Tests.AzureDeployerTests.DeployAsync_WithOnlyAzureResources_PrintsDashboardUrl() in /home/runner/work/aspire/aspire/tests/Aspire.Hosting.Azure.Tests/AzureDeployerTests.cs:619
      --- End of stack trace from previous location ---