Emulator (preview) and TestContainers: not a happy marriage?

[OnnoH]

For our local integration tests we use Testcontainers (https://testcontainers.com/modules/cosmodb/).

Depending on the CPU (our pipeline agents run on Intel, locally I have an Apple Silicon), the container image is selected:

private CosmosDBEmulatorContainer databaseContainer;

String cosmosDBEmulatorImage = "mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest";
String osArch = System.getProperty("os.arch");
if ("aarch64".equalsIgnoreCase(osArch) || "arm64".equalsIgnoreCase(osArch)) {
    cosmosDBEmulatorImage = "mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview";
}
databaseContainer = new CosmosDBEmulatorContainer(DockerImageName.parse(cosmosDBEmulatorImage))
        .withStartupTimeout(Duration.ofSeconds(120));
if ("aarch64".equalsIgnoreCase(osArch) || "arm64".equalsIgnoreCase(osArch)) {
    databaseContainer.withCommand("--protocol", "https")
            .waitingFor(Wait.forLogMessage(".*listening.*", 1));
}

databaseContainer.start();

That seems to work:

06:47:04.000 [main] INFO tc.mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview -- Creating container for image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
06:47:04.042 [main] INFO tc.mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview -- Container mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview is starting: eeb9718ee468bd4f7d1e58db84a02a2fb168049f44447ab60d48786e24155c20
06:47:11.232 [main] INFO tc.mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview -- Container mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview started in PT7.232195S

and doing some certificate magic too.

String certificatePath = System.getProperty("java.io.tmpdir") + File.separator + "cosmosdev";
certList = KeyStoreHelper.extractPublicCerts(databaseContainer.getHost(), databaseContainer.getMappedPort(8081), certificatePath, ".pem");
KeyStoreHelper.createKeyStore(certList, trustStorePath, password);

However running the tests after spinning up the TestContainers ecosystem fail to connect to the CosmosDB container:

06:47:14.297+01:00  WARN 6676 --- [cosmosdb-app] [ctor-http-nio-3] c.a.c.i.RxGatewayStoreModel              : Network failure

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8081
Caused by: java.net.ConnectException: Connection refused

The repositories are found:

2025-01-29T06:47:12.872+01:00  INFO 6676 --- [cosmosdb-app] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data cosmos repositories in DEFAULT mode.
2025-01-29T06:47:12.962+01:00  INFO 6676 --- [cosmosdb-app] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 86 ms. Found 4 cosmos repository interfaces.
2025-01-29T06:47:13.056+01:00  INFO 6676 --- [cosmosdb-app] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data cosmos repositories in DEFAULT mode.
2025-01-29T06:47:13.064+01:00  INFO 6676 --- [cosmosdb-app] [           main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 7 ms. Found 0 cosmos repository interfaces.

Getting the database account also result in a response:

c.a.c.i.RxDocumentClientImpl             : Getting database account endpoint from https://localhost:61517
2025-01-29T06:47:14.239+01:00  INFO 6676 --- [cosmosdb-app] [ctor-http-nio-1] c.a.c.i.GlobalEndpointManager            : db account retrieved {"_self":"","id":"cosmosdev","_rid":"cosmosdev","media":"//media/","addresses":"//addresses/","_dbs":"//dbs/","enableMultipleWriteLocations":false,"writableLocations":[{"name":"Primary","databaseAccountEndpoint":"https://127.0.0.1:8081/"}],"readableLocations":[{"name":"Primary","databaseAccountEndpoint":"https://127.0.0.1:8081/"}],"userConsistencyPolicy":{"defaultConsistencyLevel":"Eventual"},"queryEngineConfiguration":"{\"sqlAllowSubQuery\":true,\"sqlAllowTop\":true,\"sqlAllowGroupByClause\":true,\"sqlAllowLike\":true,\"sqlAllowScalarSubQuery\":true,\"sqlAllowAggregateFunctions\":true,\"maxSpatialQueryCells\":2147483647,\"maxLogicalOrPerSqlQuery\":2147483647,\"maxLogicalAndPerSqlQuery\":2147483647,\"maxInExpressionItemsCount\":2147483647,\"enableSpatialIndexing\":true,\"sqlDisableOptimizationFlags\":0,\"sqlAllowNonFiniteNumbers\":false,\"spatialMaxGeometryPointCount\":256,\"queryMaxInMemorySortDocumentCount\":-500,\"maxUdfRefPerSqlQuery\":10,\"maxSqlQueryInputLength\":524288,\"maxQueryRequestTimeoutFraction\":0.9,\"maxJoinsPerSqlQuery\":10,\"allowNewKeywords\":true}"}

The client starts:

com.azure.cosmos.CosmosClientBuilder     : Cosmos Client with (Correlation) ID [00001] started up in [324] ms with the following configuration: serviceEndpoint [https://localhost:61517], preferredRegions [[]], excludedRegions [[]], connectionPolicy [ConnectionPolicy{httpNetworkRequestTimeout=PT1M, tcpNetworkRequestTimeout=PT5S, connectionMode=GATEWAY, maxConnectionPoolSize=100, idleHttpConnectionTimeout=PT24H, idleTcpConnectionTimeout=PT0S, userAgentSuffix='az-sd-cos/5.19.0', throttlingRetryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTime=PT30S}, endpointDiscoveryEnabled=true, preferredRegions=null, multipleWriteRegionsEnabled=true, proxyType=null, inetSocketProxyAddress=null, readRequestsFallbackEnabled=true, connectTimeout=PT5S, idleTcpEndpointTimeout=PT1H, maxConnectionsPerEndpoint=130, maxRequestsPerConnection=30, tcpConnectionEndpointRediscoveryEnabled=true, ioThreadPriority=5, ioThreadCountPerCoreFactor=2, tcpHealthCheckTimeoutDetectionEnabled=true, minConnectionPoolSizePerEndpoint=1, openConnectionsConcurrency=1, aggressiveWarmupConcurrency=12}], consistencyLevel [null], contentResponseOnWriteEnabled [true], sessionCapturingOverride [false], connectionSharingAcrossClients [false], clientTelemetryEnabled [false], proactiveContainerInit [null], diagnostics [{samplingRate=1.0, thresholds={pointOperationLatencyThreshold=PT1S, nonPointOperationLatencyThreshold=PT3S, requestChargeThreshold=1000.0, payloadSizeInBytesThreshold=2147483647, customFailureHandler=false}, clientCorrelationId=null, clientTelemetryEnabled=false, clientMetricsEnabled=true, transportLevelTracingEnabled=false, showQueryMode=None, customTracerProvided=false, customDiagnosticHandlers=(com.azure.cosmos.CosmosDiagnosticsLogger)}], tracing [true, false, com.azure.cosmos.implementation.DiagnosticsProvider.EnabledNoOpTracer, [com.azure.cosmos.implementation.clienttelemetry.ClientMetricsDiagnosticsHandler, com.azure.cosmos.CosmosDiagnosticsLogger]], nativeTransport [false] fastClientOpen [false] isRegionScopedSessionCapturingEnabled [false]

The property is set:

System.setProperty("spring.cloud.azure.cosmos.endpoint", "https://"+databaseContainer.getHost()+":"+databaseContainer.getMappedPort(8081));

Now the response from Getting database account endpoint worries me:

"databaseAccountEndpoint":"https://127.0.0.1:8081/"

What am I doing wrong?

[xgerman]

We are committed to make this work. We will review and see what we cna do to make this more smooth.

[OnnoH]

We are committed to make this work. We will review and see what we cna do to make this more smooth.

That would be nice ;-) Let me know if I can help.

[lionelc]

@OnnoH May I ask if you use direct mode or gateway mode to visit the localhost endpoint? This is an article about these two modes with Java SDK: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/tune-connection-configurations-java-sdk-v4?tabs=api-async#direct-connection-mode

[OnnoH]

@lionelc I'm using 'gateway mode' as suggested in #158 . Maybe this issue is also related to #159 ?

[bitxon]

Hello, @OnnoH
i was able to configure Java Testcontainers to be working with CosmosEmulator VNext image over https.
Take a look, might be it would be helpful to you

Key points:

1. Certificate should be extracted/copied from image /scripts/certs/domain.crt and /scripts/certs/rootCA.crt to host machine
2. Certificates should we saved to trustStore
3. Gateway mode should be used
4. ⚠️ Exposed port must be equals to internal port (by default testcontainer exposes on random port that doesn't match internal). this could be fixed using following snippet

           this.port = getRandomAvailablePort();
           this.withEnv(Map.of(
               "PORT", String.valueOf(port),
               "PROTOCOL", "https"
           ));
           this.withCreateContainerCmdModifier(cmd -> cmd.withHostConfig(
               new HostConfig().withPortBindings(new PortBinding(Ports.Binding.bindPort(port), new ExposedPort(port)))
           ));

   For some reason it is important for cosmos-client even if you run regular docker image without testcontainers

   docker run --publish 56001:56001 --publish 1234:1234 \
     mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview \
     --protocol https --port 56001

You can see full example here:
https://github.com/bitxon/java-cosmos-emulator-vnext-experiment

P.S for me this approach with VNext works on Ubuntu, Mac (arm), Windows

[OnnoH]

@bitxon Thanks for this update! I also managed to get it working by adding the custom container class. In my Spring Boot app, only a keystore with the extracted certificates did not work: I needed to add them to the cacerts.

[lionelc]

@bitxon Thank you for trying and working it around. We have been working to make the VNext emulator directly compatible with CosmosDBEmulatorContainer with testcontainers-java. While the certs handling needs to be explicit now, the old class CosmosDBEmulatorContainer could still be used with additional options:

new CosmosDBEmulatorContainer(
DockerImageName.parse("mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview")
)
.withExposedPorts(port) ...

Stay tuned as we will make our updates on this.