CreateItemDuringTimeoutTest2  Source: ClientTests.cs line 962  Duration: 3.7 sec Message:  Test method Microsoft.Azure.Cosmos.SDK.EmulatorTests.ClientTests.CreateItemDuringTimeoutTest2 threw exception: Microsoft.Azure.Cosmos.CosmosException: Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 7ea13ae7-9730-4363-aec0-448f3f804d18; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:3/17/2025 11:29:11 PM; Total Duration:1006.5077 Ms; Request Timeout 65000 Ms; Http Client Timeout:65000 Ms; Activity id: 2303f861-f868-4a1f-9842-2107f92f2cff;);; Diagnostics:{"Summary":{},"name":"CreateItemAsync","start datetime":"2025-03-17T23:29:11.998Z","duration in milliseconds":1025.2522,"data":{"Client Configuration":{"Client Created Time Utc":"2025-03-17T23:29:11.6344289Z","MachineId":"vmId:e64a87f2-63dd-495c-bd6c-29a92bb238e2","VM Region":"westus2","NumberOfClientsCreated":2,"NumberOfActiveClients":2,"ConnectionMode":"Direct","User Agent":"cosmos-netstandard-sdk/3.47.2|2|X64|Microsoft Windows 10.0.26100|.NET 6.0.36|N|","ConnectionConfig":{"gw":"(cps:50, urto:2, p:False, httpf: False)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: Session, prgns:[], apprgn: )","ProcessorCount":16}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0145},{"name":"Get PkValue From Stream","duration in milliseconds":0.0736,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0004}]},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":1024.9866,"children":[{"name":"Get Collection Cache","duration in milliseconds":0.0003},{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":1024.9674,"data":{"System Info":{"systemHistory":[{"dateUtc":"2025-03-17T23:29:07.3560660Z","cpu":7.158,"memory":39482076.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":16,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":1024.9648,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":1024.9633,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":1024.9562,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":1024.9548,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":1024.8925,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[]}}}]}]}]}]}]}]},{"name":"Get Collection Cache","duration in milliseconds":0.0008}]} ---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 65 seconds elapsing. ---> System.TimeoutException: Task canceled exception ---> System.Threading.Tasks.TaskCanceledException: Task canceled exception ---> System.Threading.Tasks.TaskCanceledException: Inner exception Task canceled exception message Stack Trace:  ClientTests.SendingRequestEventHandlerScriptsVerifier(Object sender, SendingRequestEventArgs e) line 954 HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) line 592 HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- ExceptionHandlingUtility.HandleSpecialException(Exception e) line 25 AsyncCacheNonBlocking`2.GetAsync(TKey key, Func`2 singleValueInitFunc, Func`2 forceRefresh) line 120 GatewayAddressCache.TryGetAddressesAsync(DocumentServiceRequest request, PartitionKeyRangeIdentity partitionKeyRangeIdentity, ServiceIdentity serviceIdentity, Boolean forceRefreshPartitionAddresses, CancellationToken cancellationToken) line 285 AddressResolver.TryResolveServerPartitionAsync(DocumentServiceRequest request, ContainerProperties collection, CollectionRoutingMap routingMap, Boolean collectionCacheIsUptodate, Boolean collectionRoutingMapCacheIsUptodate, Boolean forceRefreshPartitionAddresses, CancellationToken cancellationToken) line 499 AddressResolver.ResolveAddressesAndIdentityAsync(DocumentServiceRequest request, Boolean forceRefreshPartitionAddresses, CancellationToken cancellationToken) line 316 AddressResolver.ResolveAsync(DocumentServiceRequest request, Boolean forceRefreshPartitionAddresses, CancellationToken cancellationToken) line 66 GlobalAddressResolver.ResolveAsync(DocumentServiceRequest request, Boolean forceRefresh, CancellationToken cancellationToken) line 224 AddressSelector.ResolveAddressesAsync(DocumentServiceRequest request, Boolean forceAddressRefresh) ConsistencyWriter.WritePrivateAsync(DocumentServiceRequest request, TimeoutHelper timeout, Boolean forceRefresh) BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback) ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback) BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback) ConsistencyWriter.WriteAsync(DocumentServiceRequest entity, TimeoutHelper timeout, Boolean forceRefresh, CancellationToken cancellationToken) <b__0>d.MoveNext() --- End of stack trace from previous location --- RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback) ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback) RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback) StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy) TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) line 135 TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) line 33 --- End of inner exception stack trace --- ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Nullable`1 openTelemetry, RequestOptions requestOptions, Nullable`1 resourceType) line 565 ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func`2 task, Nullable`1 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel, Nullable`1 resourceType) line 262 <b__1>d.MoveNext() line 1017 --- End of stack trace from previous location --- ClientTests.CreateItemDuringTimeoutTest2() line 1037 ClientTests.CreateItemDuringTimeoutTest2() line 1043 ThreadOperations.ExecuteWithAbortSafety(Action action) Standard Output:  With cache fix and 10K threads 4923736 bytes allocated in (Gen0:0 1:0 2:0) Frames=2