How to ensure the readiness of the Api Azure CosmosDb Emulator:vnext-preview?

[NelsonBN]

How to ensure the CosmosDB emulator:vnext-preview API is ready?

I'm using this configuration:

docker run -d -p 8081:8081 -e ENABLE_EXPLORER:false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview

I disabled the Explorer because this setup is for integration tests, and the UI is not needed.

The problem is that the API endpoint becomes available before the emulator is fully ready, causing flaky tests.

One idea is to ping the Explorer (UI) endpoint, as it takes longer to be ready.

Another option is to wait for ~1000-1500ms after the API endpoint is available. However, this is not an ideal solution and might fail in slower environments.

Here's an example of a .NET exception when accessing the API too early:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: InternalServerError (500); Substatus: 0; ActivityId: 8181dbbf-e94b-4a3a-a744-68a1bb1f26c1; Reason: ({"code":"InternalServerError","message":"schema \u0022cosmos_api\u0022 does not exist"}
RequestUri: http://127.0.0.1:62083/dbs/Demo;
RequestMethod: GET;
Header: Authorization Length: 86;
Header: x-ms-date Length: 29;
Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1;
Header: x-ms-activity-id Length: 36;
Header: Cache-Control Length: 8;
Header: User-Agent Length: 89;
Header: x-ms-version Length: 10;
Header: Accept Length: 16;

ActivityId: 8181dbbf-e94b-4a3a-a744-68a1bb1f26c1, Request URI: /dbs/Demo, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.26100 cosmos-netstandard-sdk/3.37.1);
   at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)
   at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)
--- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(500, 0)":1}},"name":"CreateDatabaseIfNotExistsAsync","start datetime":"2025-01-02T19:07:04.988Z","duration in milliseconds":527.5967,"data":{"Client Configuration":{"Client Created Time Utc":"2025-01-02T19:07:04.9887536Z","MachineId":"hashedMachineName:3d70b522-daa4-aa5c-ed33-fcc1b6764e3b","NumberOfClientsCreated":8,"NumberOfActiveClients":8,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.46.0|8|X64|Microsoft Windows 10.0.26100|.NET 9.0.0|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":12}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":344.9599,"children":[{"name":"Waiting for Initialization of client to complete","duration in milliseconds":10.0809},{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":334.8141,"data":{"System Info":{"systemHistory":[{"dateUtc":"2025-01-02T19:06:06.7262467Z","cpu":22.098,"memory":12126596.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":3.9645,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2025-01-02T19:06:16.8137848Z","cpu":12.261,"memory":12066520.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0409,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2025-01-02T19:06:26.9332150Z","cpu":13.300,"memory":12247012.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0583,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2025-01-02T19:06:37.0401833Z","cpu":12.582,"memory":12156652.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0439,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2025-01-02T19:06:47.1318151Z","cpu":10.674,"memory":12070400.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0702,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2025-01-02T19:06:57.2368031Z","cpu":12.186,"memory":12295008.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0419,"availableThreads":32766,"minThreads":12,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":334.8034,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":334.7943,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":334.755,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":334.7512,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":309.3186,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2025-01-02T19:07:04.9992145Z","DurationInMs":205.1235,"RequestUri":"http://127.0.0.1:62083/dbs/Demo","ResourceType":"Database","HttpMethod":"GET","ActivityId":"8181dbbf-e94b-4a3a-a744-68a1bb1f26c1","StatusCode":"InternalServerError","ReasonPhrase":"Internal Server Error"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"8181dbbf-e94b-4a3a-a744-68a1bb1f26c1","ResponseTimeUtc":"2025-01-02T19:07:05.3327347Z","StatusCode":500,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/Demo","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\"code\":\"InternalServerError\",\"message\":\"schema \\u0022cosmos_api\\u0022 does not exist\"}\r\nRequestUri: http://127.0.0.1:62083/dbs/Demo;\r\nRequestMethod: GET;\r\nHeader: Authorization Length: 86;\r\nHeader: x-ms-date Length: 29;\r\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\r\nHeader: x-ms-activity-id Length: 36;\r\nHeader: Cache-Control Length: 8;\r\nHeader: User-Agent Length: 89;\r\nHeader: x-ms-version Length: 10;\r\nHeader: Accept Length: 16;\r\n\r\nActivityId: 8181dbbf-e94b-4a3a-a744-68a1bb1f26c1, Request URI: /dbs/Demo, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.26100 cosmos-netstandard-sdk/3.37.1\r\n   at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\r\n   at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\r\n   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\r\n   at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\r\n   at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":"0:-1#1","BELatencyInMs":null}}}]}]}]}]}]}]}]}

here's an example tests and here a discussion in the Testcontainers community about implementing a better solution.

What is the best way to ensure the API is fully ready?

Is it possible to implement a more resilient health check strategy that works for future versions of emulator as well?

[christopheranderson]

We should change this to return a retriable status code that the SDK will automatically handle retry on. Ideally, we'd use one which the SDK would take a backoff signal for so we can avoid the SDK retrying constantly if we expect it to take 1000ms.

[argium]

Some guidance would be great. My team's test suite also uses TestContainers and I'm testing for readiness by trying to create a database.

        public async Task<bool> UntilAsync(IContainer container)
        {
            return await this.retry.ExecuteAsync(async () =>
            {
                var client = clientFactory();
                var result = await client.Object.CreateDatabaseIfNotExistsAsync("testdb");
                return result.StatusCode == HttpStatusCode.Created;
            });
        }

Best would be to have a health check page that returns 200 OK when all internal services are ready.

[kirankumarkolli]

@christopheranderson what's the cause?