Consider releasing #107333 into dotnet 9

[djluck]

We've been observing high lock contention in ActivitySource when it comes to publishing activity start/ stop notifications. #107333 fixes this issue but I'm unable to comment in this PR (it was locked).

@tarekgh in that MR you outlined the higher threshold for merging this change into v9, citing impact. In our case we are seeing 10-20% of CPU time being contented in some of our high throughput services, contributing to severe latency spikes. Our hardware can have up to 192 cores which means we're far more likely to experience contention around a single shared mutex.

Hopefully this is data point is enough to consider merging this change into v9.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-activity
See info in area-owners.md if you want to be subscribed.

[tarekgh]

@djluck the bar is really very high now. I'll check if this can be justified at the current stage of the release. Or maybe we can look having it in the service release.

Could you please help answering the following questions:

You said In our case we are seeing 10-20% of CPU time being contented in some of our high throughput services,, did you check if the issue we have fixed is the root cause of that or could be other things contributing to that? how did you validate that? Why did you start seeing that now? Is there a way you test your scenario with .NET 10 https://github.com/dotnet/sdk/blob/main/documentation/package-table.md and validate the fix is really helping.

[tarekgh]

CC @noahfalk @ericstj

[djluck]

Thanks for the quick reply! In response:

did you check if the issue we have fixed is the root cause of that or could be other things contributing to that?

We took traces of our application and saw excessive CPU due to lock contention in the exact function this PR addresses. I see no other mutex use within the start/ stop code path and so I believe it can only be the listener invocations. A sample trace we captured:

How did you validate that?

See this trace:

Why did you start seeing that now?

We're relatively new to tracing within our organization and have been increasing adoption within our large scale service over the last few months.

Is there a way you test your scenario with .NET 10 https://github.com/dotnet/sdk/blob/main/documentation/package-table.md and validate the fix is really helping.

Is there a docker image available currently? One way I've been able to verify the lack of mutex impact is to create multiple ActivitySource's and distribute Start() calls across them- this has eliminated the contention we are seeing:

public class ActivitySources
    {
        private int _next = 0; 
        private ImmutableArray<ActivitySource> _sources;


        public ActivitySources(ActivitySpanSubsystemConfig config)
        {
            _sources = Enumerable.Range(1, 2048).Select(n => new ActivitySource(config.ActivitySourceName + n))
                .ToImmutableArray();
        }

        public ActivitySource Get()
        {
            return _sources[Interlocked.Increment(ref _next) % _sources.Length];

        }
    }

[tarekgh]

@djluck thanks for the info.

Is there a docker image available currently?

Could you try please reference the System.Diagnostics.DiagnosticSource package version 10.0.0-alpha.1.24521.2 from the NuGet feed https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10/nuget/v3/index.json.
https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet10/NuGet/System.Diagnostics.DiagnosticSource/overview/10.0.0-alpha.1.24521.2

I'll wait your validation then I'll try to check if we can get the approval for 9.0.

[djluck]

Thanks for giving me the pre release package feed, I've built a test version of our application with it and will test it tomorrow.

[djluck]

Unfortunately the testing has hit a wall: we're seeing MissingMethodExceptions:

 Connection id "0HN7ISM12NH0N", Request id "0HN7ISM12NH0N:00000009": An unhandled exception was thrown by the application.
ERROR:One or more errors occurred. (Method not found: 'Void System.Diagnostics.ActivityCreationOptions`1.set_TraceState(System.String)'.) || {"ExceptionStr":"System.AggregateException: One or more errors occurred. (Method not found: 'Void System.Diagnostics.ActivityCreationOptions`1.set_TraceState(System.String)'.)\n ---> System.MissingMethodException: Method not found: 'Void System.Diagnostics.ActivityCreationOptions`1.set_TraceState(System.String)'.\n   at OpenTelemetry.Trace.TracerProviderSdk.ComputeActivitySamplingResult(ActivityCreationOptions`1& options, Sampler sampler)\n   at System.Diagnostics.ActivitySource.<>c.<CreateActivity>b__23_1(ActivityListener listener, ActivityCreationOptions`1& data, ActivitySamplingResult& result, ActivityCreationOptions`1& unused)\n   at System.Diagnostics.ActivitySource.CreateActivity(String name, ActivityKind kind, ActivityContext context, String parentId, IEnumerable`1 tags, IEnumerable`1 links, DateTimeOffset startTime, Boolean startIt, ActivityIdFormat idFormat)\n   at TTD.Common.TelemetryLib.ActivitySpans.ActivitySpan.Start(IEnumerable`1 tags, ActivityContext parentContext, IEnumerable`1 links, String callerName, String callerFilePath, Int32 callerLineNum)

I'm not 100% why we're getting this:

• ActivityCreationOptions.TraceState isn't mutable (marked as init)...
• But we are using version 1.9.0 of OpenTelemetry that uses the with syntax to create a new immutable instance

It's possible it might be related to the fact we're targeting net 6.0 for our application? I haven't decompiled the netstandard2 versions of all the DLLs but there might be some differences here?

[tarekgh]

This is very weird, and I have no idea how this happened. I downloaded the NET10 package 10.0.0-alpha.1.24521.2 and opened the ns2.0 library and I am seeing the init of the ActivityCreationOptions.TraceState is missing. This shouldn't happen. I built the package locally and opened the ns2.0 library and I am seeing the init is there.

Could you please try the recent version of the package from https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet10/NuGet/System.Diagnostics.DiagnosticSource/overview/10.0.0-alpha.1.24522.5? I confirmed this version doesn't have this problem.

[djluck]

Thanks for digging in, I'll give that package a try 👍