Repository navigation
Conversation
There was a problem hiding this comment.
Pull request overview
This is a work-in-progress PR that migrates Azure Storage testing infrastructure from GitHub Actions services to Testcontainers, specifically focusing on Azurite. The change is part of a broader initiative (issue #9708) to simplify the testing infrastructure and make tests easier to run locally.
Changes:
- Introduces
AzuriteContainerManagerto manage a singleton Azurite container using Testcontainers - Removes Azure AD authentication support from test configuration (TestDefaultConfiguration)
- Updates all Azure Storage test configurations to use the new container manager instead of environment variables
- Removes Azurite service definition from GitHub Actions workflow for Azure Storage tests
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/TestInfrastructure/TestExtensions/AzuriteContainerManager.cs | New singleton manager for Azurite Testcontainer with lazy initialization |
| test/TestInfrastructure/TestExtensions/TestDefaultConfiguration.cs | Removed Azure AD authentication properties and connection string configurations |
| test/TestInfrastructure/TestExtensions/TestUtils.cs | Updated CheckForAzureStorage to use AzuriteContainerManager |
| test/TestInfrastructure/TestExtensions/TestExtensions.csproj | Replaced Azure.Identity dependency with Testcontainers.Azurite |
| test/TesterInternal/GeoClusterTests/BasicLogTestGrainTests.cs | Migrated to use AzuriteContainerManager connection string |
| test/Tester/Forwarding/ShutdownSiloTests.cs | Migrated to use AzuriteContainerManager connection string |
| test/Extensions/TesterAzureUtils/Streaming/TestAzureTableStorageStreamFailureHandler.cs | Migrated to use AzuriteContainerManager connection string |
| test/Extensions/TesterAzureUtils/Streaming/AQStreamingTests.cs | Migrated to use AzuriteContainerManager connection string |
| test/Extensions/TesterAzureUtils/StorageEmulatorUtilities.cs | Updated to check AzuriteContainerManager connection string |
| test/Extensions/TesterAzureUtils/AzureStorageOperationOptionsExtensions.cs | Removed Azure AD auth logic and migrated all extension methods to use AzuriteContainerManager |
| test/Extensions/Tester.Cosmos/CosmosOptionsExtensions.cs | Simplified to remove Azure AD authentication support |
| test/Extensions/ServiceBus.Tests/EventHubConfigurationExtensions.cs | Simplified to remove Azure AD authentication support |
| test/Orleans.Journaling.Tests/JournalingAzureStorageTestConfiguration.cs | Updated to use AzuriteContainerManager connection string |
| test/Benchmarks/Transactions/TransactionBenchmark.cs | Migrated to use AzuriteContainerManager connection string |
| test/Benchmarks/GrainStorage/GrainStorageBenchmark.cs | Migrated to use AzuriteContainerManager connection string (with bug in AdoNet configurator) |
| NuGet.Config | Commented out Azure DevOps package feeds (should not be in this PR) |
| src/Directory.Build.props | Commented out Microsoft.DotNet.GenAPI.Task (should not be in this PR) |
| Directory.Packages.props | Added Testcontainers.Azurite package version |
| .github/workflows/ci.yml | Removed Azurite service and connection string from Azure Storage test job |
|
getting errors seems random 😔 |
|
@Meir017 the Consul Testcontainers commit is causing failures in our internal Azure DevOps, I suppose we aren't allowed to access docker.io from there. I'm trying to see if we can fix that (#9933). We might need to disable Testcontainers tests in AzDO. |
I believe they have some strict rate limits on dockerhub. So creating a custom container registry and copying specific images could work (like the aspire repo does) |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.
Suppressed comments (5)
test/TestInfrastructure/TestExtensions/TestUtils.cs:25
- CheckForEventHub now only considers EventHubConnectionString. This breaks the existing AAD path (EventHubConfigurationExtensions uses EventHubFullyQualifiedNamespace when UseAadAuthentication=true), causing AAD-based Event Hubs tests to be skipped even when correctly configured.
public static void CheckForEventHub()
{
if (string.IsNullOrWhiteSpace(EventHubConnectionString))
{
throw new SkipException("No connection string found. Skipping");
test/Extensions/Orleans.Azure.Tests/Streaming/AQStreamingTests.cs:53
- AQStreamingTests now always configures the Azure Queue stream provider via a connection string. This removes the existing AAD configuration path (ServiceUri/Credential + ServiceKey), so running these tests with UseAadAuthentication=true will no longer work.
ConfigureStreaming("ProviderType", "AzureQueueStorage");
ConfigureStreaming("ConnectionString", TestDefaultConfiguration.AzureStorageConnectionString);
test/Extensions/Orleans.Azure.Tests/StorageEmulatorUtilities.cs:12
- EnsureEmulatorIsNotUsed now reads AzureStorageConnectionString, which can start the Azurite container just to decide to skip. This adds unnecessary Docker startup overhead and side effects during precondition checks. Prefer checking the configured DataConnectionString (or its absence) without triggering container startup.
public static void EnsureEmulatorIsNotUsed()
{
if (TestDefaultConfiguration.AzureStorageConnectionString is { Length: > 0 } connectionString
&& (connectionString.Contains("UseDevelopmentStorage", StringComparison.OrdinalIgnoreCase)
|| connectionString.Contains("devstoreaccount", StringComparison.OrdinalIgnoreCase)))
test/TestInfrastructure/TestExtensions/TestDefaultConfiguration.cs:53
- AzureStorageConnectionString currently treats "UseDevelopmentStorage=true" as a real connection string (so tests will try to use a local emulator which is no longer auto-started) and it will also start Azurite even if UseAadAuthentication=true and a caller accidentally uses this property. Consider mapping the legacy value to Azurite and guarding against accidental use in AAD mode.
var connectionString = DataConnectionString;
return string.IsNullOrWhiteSpace(connectionString)
? AzuriteContainerManager.ConnectionString
: connectionString;
}
test/TestInfrastructure/TestExtensions/TestUtils.cs:17
- CheckForAzureStorage no longer validates required AAD configuration and also won’t start Azurite when DataConnectionString is set to the legacy "UseDevelopmentStorage=true" value. In AAD mode this can now throw (eg new Uri(null)) instead of skipping, and in non-AAD mode it can leave tests pointing at a non-running local emulator.
This issue also appears on line 21 of the same file.
public static void CheckForAzureStorage()
{
if (!UseAadAuthentication && string.IsNullOrWhiteSpace(DataConnectionString))
{
AzuriteContainerManager.EnsureStarted();
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Suppressed comments (5)
test/TestInfrastructure/TestExtensions/TestUtils.cs:26
- CheckForEventHub now only checks EventHubConnectionString, which breaks AAD-backed runs: EventHubConfigurationExtensions uses EventHubFullyQualifiedNamespace + TokenCredential when UseAadAuthentication is true. With the current precondition, AAD runs will be skipped (or will later throw due to null-forgiving usage).
Update the precondition to accept either (AAD + fully-qualified namespace) or (connection string), and skip only when the relevant config is missing.
{
if (string.IsNullOrWhiteSpace(EventHubConnectionString))
{
throw new SkipException("No connection string found. Skipping");
}
test/Extensions/Orleans.Azure.Tests/StorageEmulatorUtilities.cs:14
- EnsureEmulatorIsNotUsed now reads AzureStorageConnectionString, which can start the Azurite container. In AAD-backed runs (where DataConnectionString is typically unset), this can incorrectly require Docker and skip/fail tests even though real Azure configuration is available.
This utility is meant to guard against explicitly configured storage-emulator connection strings, so it should inspect the raw DataConnectionString value instead of the fallback connection string.
if (TestDefaultConfiguration.AzureStorageConnectionString is { Length: > 0 } connectionString
&& (connectionString.Contains("UseDevelopmentStorage", StringComparison.OrdinalIgnoreCase)
|| connectionString.Contains("devstoreaccount", StringComparison.OrdinalIgnoreCase)))
{
throw new SkipException("This test does not support the storage emulator.");
test/TestInfrastructure/TestExtensions/TestUtils.cs:18
- CheckForAzureStorage no longer validates AAD-based configuration. In AAD mode, tests can now proceed with missing Azure Storage endpoint settings and later fail (or start Azurite unnecessarily). This also regresses the stated goal of keeping AAD configuration supported.
Consider explicitly checking for required AAD keys (without evaluating the Uri properties) and skipping with a clear message when missing; otherwise, start Azurite only when not using AAD and no connection string was supplied.
This issue also appears on line 22 of the same file.
{
if (!UseAadAuthentication && string.IsNullOrWhiteSpace(DataConnectionString))
{
AzuriteContainerManager.EnsureStarted();
}
test/Extensions/Orleans.Azure.Tests/Streaming/AQStreamingTests.cs:53
- This change removes the AAD-auth configuration path for Azure Queue streaming tests (it now always configures a connection string). That conflicts with the PR description which says AAD configuration remains supported, and it can also cause these tests to require Docker even when real Azure/AAD configuration is present.
Restore the prior AAD branch (ServiceUri + Credential + ServiceKey) and use AzureStorageConnectionString only for the connection-string path.
ConfigureStreaming("ProviderType", "AzureQueueStorage");
ConfigureStreaming("ConnectionString", TestDefaultConfiguration.AzureStorageConnectionString);
test/TestInfrastructure/TestExtensions/TestcontainerManager.cs:45
- EnsureStartedAsync currently does all work synchronously (via _startSkipReason.Value) and then returns a completed Task. That makes the method name/return type misleading and can unexpectedly block async call paths.
If you want to keep the cached Lazy<string?> approach, consider offloading the potentially blocking initialization to the thread pool so callers can truly await it.
public Task<bool> EnsureStartedAsync() => Task.FromResult(_startSkipReason.Value is null);
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
test/TestInfrastructure/TestExtensions/TestUtils.cs:27
CheckForEventHubnow only checksEventHubConnectionString, but AAD-based Event Hubs tests configureEventHubFullyQualifiedNamespace(seeEventHubConfigurationExtensions.ConfigureTestDefaults). WithUseAadAuthentication=true, this will incorrectly skip tests even when AAD config is present.
public static void CheckForEventHub()
{
if (string.IsNullOrWhiteSpace(EventHubConnectionString))
{
throw new SkipException("No connection string found. Skipping");
}
}
test/Extensions/Orleans.Azure.Tests/StorageEmulatorUtilities.cs:15
EnsureEmulatorIsNotUsednow readsAzureStorageConnectionString, which can start an Azurite container as a side effect. That means tests which are about to be skipped for emulator usage may still spin up Docker, and AAD-based runs can be affected unnecessarily. This check should use the explicitly configured connection string (or AAD mode) rather than the Azurite fallback.
public static void EnsureEmulatorIsNotUsed()
{
if (TestDefaultConfiguration.AzureStorageConnectionString is { Length: > 0 } connectionString
&& (connectionString.Contains("UseDevelopmentStorage", StringComparison.OrdinalIgnoreCase)
|| connectionString.Contains("devstoreaccount", StringComparison.OrdinalIgnoreCase)))
{
throw new SkipException("This test does not support the storage emulator.");
}
test/TestInfrastructure/TestExtensions/TestcontainerManager.cs:4
DotNet.Testcontainers.Buildersis not used in this file. With code-style enforcement enabled, the unused using can fail the build (CS8019) because warnings are treated as errors.
using Docker.DotNet;
using DotNet.Testcontainers.Builders;
using DotNet.Testcontainers.Containers;
using Xunit;
test/TestInfrastructure/TestExtensions/TestUtils.cs:19
CheckForAzureStorageno longer validates the AAD configuration. IfUseAadAuthentication=truebutTableEndpoint(and related endpoints) are not set, tests will later throw whenTestDefaultConfiguration.TableEndpointis evaluated instead of being skipped cleanly.
public static void CheckForAzureStorage()
{
if (!UseAadAuthentication && string.IsNullOrWhiteSpace(DataConnectionString))
{
AzuriteContainerManager.EnsureStarted();
}
}
|
@ReubenBond ready for review |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Suppressed comments (5)
test/TestInfrastructure/TestExtensions/TestUtils.cs:25
CheckForEventHubnow only checksEventHubConnectionString, which breaks AAD-based Event Hubs test runs: whenUseAadAuthenticationis true, the tests useEventHubFullyQualifiedNamespace+TokenCredential, but this precondition method will always skip because the connection string is expected to be unset.
public static void CheckForEventHub()
{
if (string.IsNullOrWhiteSpace(EventHubConnectionString))
{
throw new SkipException("No connection string found. Skipping");
test/TestInfrastructure/TestExtensions/TestUtils.cs:17
CheckForAzureStorageno longer validates the AAD-auth configuration. WhenUseAadAuthenticationis true but required settings (e.g., TableEndpoint/DataBlobUri/DataQueueUri) are missing, tests will now fail later when thoseUriproperties are accessed (they throw on missing config) instead of skipping cleanly.
This issue also appears on line 21 of the same file.
public static void CheckForAzureStorage()
{
if (!UseAadAuthentication && string.IsNullOrWhiteSpace(DataConnectionString))
{
AzuriteContainerManager.EnsureStarted();
test/TestInfrastructure/TestExtensions/TestcontainerManager.cs:4
using DotNet.Testcontainers.Builders;appears unused in this file and will typically produce CS8019 (unused using) which can fail the build when warnings are treated as errors. Remove the unused using directive.
using Docker.DotNet;
using DotNet.Testcontainers.Builders;
using DotNet.Testcontainers.Containers;
test/Extensions/Orleans.Azure.Tests/StorageEmulatorUtilities.cs:12
EnsureEmulatorIsNotUsednow readsAzureStorageConnectionString, which can lazily start Azurite and then cause these tests to be skipped even whenUseAadAuthenticationis true (i.e., when the intent is to run against real Azure endpoints). This guard should not trigger container startup and should only check the explicitly supplied connection string.
public static void EnsureEmulatorIsNotUsed()
{
if (TestDefaultConfiguration.AzureStorageConnectionString is { Length: > 0 } connectionString
&& (connectionString.Contains("UseDevelopmentStorage", StringComparison.OrdinalIgnoreCase)
|| connectionString.Contains("devstoreaccount", StringComparison.OrdinalIgnoreCase)))
test/Extensions/Orleans.Azure.Tests/Streaming/AQStreamingTests.cs:53
- The host-side streaming provider configuration now always uses
AzureStorageConnectionString. InUseAadAuthenticationmode this will force a connection-string/Azurite path (and potentially start Azurite) instead of using the existing AAD configuration, which contradicts the goal of keeping AAD-based internal runs working and can also diverge from the client-side configuration which still usesConfigureTestDefaults()(AAD-aware).
ConfigureStreaming("ProviderType", "AzureQueueStorage");
ConfigureStreaming("ConnectionString", TestDefaultConfiguration.AzureStorageConnectionString);
|
thanks, @Meir017! Looks like GHA hit some issues. I'll retry the CI run soon |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.
Suppressed comments (5)
test/TestInfrastructure/TestExtensions/TestUtils.cs:25
CheckForEventHubnow only checksEventHubConnectionString, butEventHubConfigurationExtensionsusesEventHubFullyQualifiedNamespace!whenUseAadAuthenticationis true. In AAD mode (with no connection string), this method will incorrectly skip the precondition check and tests can fail with a null-forgiving dereference. The precondition should match the chosen auth mode.
public static void CheckForEventHub()
{
if (string.IsNullOrWhiteSpace(EventHubConnectionString))
{
throw new SkipException("No connection string found. Skipping");
test/Extensions/Orleans.Clustering.Consul.Tests/ConsulTestUtils.cs:40
IPC_LOCKcapability is now added unconditionally. The previous implementation avoided setting this when running against a Windows Docker daemon/container mode. Adding Linux capabilities in those environments can cause container creation to fail, turning a previously-skipped test into a hard failure.
.WithCreateParameterModifier(parameters =>
{
if (parameters.HostConfig is not null)
{
parameters.HostConfig.CapAdd = ["IPC_LOCK"];
test/Benchmarks/GrainStorage/AzureBlobWriteStateStreamingBenchmark.cs:110
- Benchmarks are not xUnit tests; using
AzureStorageConnectionStringcan start an Azurite Testcontainer and throwSkipExceptionwhen Docker is unavailable, which BenchmarkDotNet will report as a benchmark failure (and the message refers to "tests"). Consider keeping the benchmark’s dependency explicit by requiringDataConnectionStringfor the non-AAD path (as before) and throwing a clearInvalidOperationExceptionif it’s missing.
return new BlobServiceClient(TestDefaultConfiguration.AzureStorageConnectionString);
test/Benchmarks/GrainStorage/AzureBlobReadStateStreamingBenchmark.cs:121
- Benchmarks are not xUnit tests; using
AzureStorageConnectionStringcan start an Azurite Testcontainer and throwSkipExceptionwhen Docker is unavailable, which BenchmarkDotNet will report as a benchmark failure (and the message refers to "tests"). Consider keeping the benchmark’s dependency explicit by requiringDataConnectionStringfor the non-AAD path (as before) and throwing a clearInvalidOperationExceptionif it’s missing.
return new BlobServiceClient(TestDefaultConfiguration.AzureStorageConnectionString);
test/TestInfrastructure/TestExtensions/TestUtils.cs:17
CheckForAzureStorageno longer validates AAD-based configuration. WhenUseAadAuthenticationis true andTableEndpointis missing/empty, tests can now fail later withUriconstruction exceptions instead of being skipped. Consider restoring an explicit precondition check for the AAD path before proceeding.
public static void CheckForAzureStorage()
{
if (UseAzurite)
{
AzuriteContainerManager.EnsureStarted();
There was a problem hiding this comment.
🟡 Changes recommended
Docker startup failures after the preflight check can escape skip handling and fail Azure Storage tests.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 27/27 changed files
- Comments generated: 1
- Review effort level: Lite
1ea7e42 to
0fae3f0
Compare
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Review details
- Files reviewed: 27/27 changed files
- Comments generated: 4
- Review effort level: Lite
0fae3f0 to
1cae1c8
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Three moderate issues remain in benchmark configuration and Azure Storage precondition handling.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
test/Benchmarks/GrainStorage/GrainStorageBenchmark.cs:82
- This benchmark setup now resolves
AzureStorageConnectionString, which starts Azurite when no real connection string is configured. That makesGrainStorage.AzureBlobmeasure the emulator rather than the intended Azure backend and can surface an xUnitSkipExceptionin the console benchmark if Docker is unavailable. Keep benchmark setup on an explicitly configured connection with a clear guard.
options.BlobServiceClient = new(TestDefaultConfiguration.AzureStorageConnectionString);
test/Extensions/Orleans.Azure.Tests/AzureStorageOperationOptionsExtensions.cs:29
- Event Hubs tests call this helper through
EventHubConfigurationExtensions.ConfigureTestDefaults, but their fixture checks only Event Hubs configuration. With noOrleansDataConnectionString, this line now lazily starts Azurite; when Docker is unavailable, the resultingSkipExceptionis raised duringBaseTestClusterFixture.InitializeAsyncrather than being captured by the fixture precondition path, so the tests do not follow the intended clean-skip behavior. Add an Azure Storage precondition for those callers or keep this fallback scoped to Azure Storage fixtures.
: new(TestDefaultConfiguration.AzureStorageConnectionString, clientOptions);
- Files reviewed: 26/26 changed files
- Comments generated: 2
- Review effort level: Lite
This reverts commit 71e943b.
1cae1c8 to
81b0107
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Address the cancellation-token handling and retry-contract input declaration findings.
Review details
Suppressed comments (2)
.github/actions/run-tests/action.yml:46
- This new input is also forwarded in the retry-contract workflow, which replaces
dotnet-test/action.ymlwith.github/actions/run-tests/test-fixture/action.yml; that fixture does not declaremax-parallel-test-modules. Each retry-contract run will therefore report an unexpected-input warning, so update the fixture's input contract alongside this forwarding change.
max-parallel-test-modules: ${{ inputs.max-parallel-test-modules }}
test/TestInfrastructure/TestExtensions/TestContainerManager.cs:76
- Passing
CancellationToken.Noneto the actual container startup ignores the xUnit cancellation/hang token. If an image pull or Docker daemon stalls after the preflight succeeds, the test process cannot be canceled while waiting forStartAsync; pass the current test cancellation token (or thread a token throughEnsureStarted) so startup observes test cancellation.
var container = _container.Value;
await _startAsync(container, CancellationToken.None).ConfigureAwait(false);
- Files reviewed: 26/26 changed files
- Comments generated: 0 new
- Review effort level: Lite
Azure Storage tests currently depend on separately provisioned Azurite services and connection-string plumbing, making local runs and CI setup inconsistent.
This change introduces a shared, lazily started Azurite Testcontainer when no Azure Storage connection string or AAD configuration is supplied. Explicit connection-string and AAD settings select existing storage services, while standalone silo processes reuse the generated connection string. Testcontainers supplies the service bindings, dynamic ports, and cleanup, replacing the separate Azurite setup in the GitHub Actions Azure Storage job.
Local tests skip when the initial Docker availability check fails. CI requires Linux Docker, and container startup errors fail tests in every environment. Azure grain storage and transaction benchmarks require an explicitly configured storage connection so results reflect the selected backend.
Related to #9708.
Microsoft Reviewers: Open in CodeFlow