[Internal] Client Telemetry Public API Proposal

[sourabh1007]

Background

As per latest discussion, to get align with Java API and avoid confusion with feature name. We decided to use "Client Telemetry" as a parent feature under which there will be 2 kind of telemetry covered as of now:
1. Send Client Diagnostic Metrics To Service : Sending telemetry metrics to Microsoft.

Read more about it here: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/docs/observability.md#send-telemetry-from-sdk-to-service-private-preview

2. Distributed Tracing: Sending Activities with operation level or network level information to customer's APM tool like AppInsights (May be in future metrics will also be covered).

Read more about it here: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/docs/observability.md#distributed-tracing-preview

We are fine if Java API and .Net API doesn't look exactly similar. Here idea is to make sure, it should not confuse customer.

Java Public API looks like this:

CosmosClientBuilder

CosmosClientTelemetryConfig

CosmosDiagnosticsThresholds

Proposed Public API in .Net SDK to control above features:

By default, both of the above features will be enabled in SDK.

Why?

1. Send Client Diagnostic Metrics To Service: It is controlled by portal hence enabling it in SDK, by default, will give seamless experience to the customer where they can just enable it from portal and we can see data flowing to us without any code change. There won't be any perf impact/overhead on sdk, if it is disabled on portal and enabled on SDK as Portal setting has higher priority.
2. Distributed Tracing: It works on subscription model, until unless there is no subscriber, it won't generate any activity. Hence no overhead. This feature flag is introduced to support appinsight SDK as customer cannot de-subscribe any activity source there.

Want to see distributed tracing in action?
OpenTelemetry Way:
https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/OpenTelemetry
AppInsight SDK Way:
https://github.com/Azure/azure-cosmos-dotnet-v3/tree/master/Microsoft.Azure.Cosmos.Samples/Usage/ApplicationInsights

In both the cases, there is a scenario where customer would want to disable individual telemetry from SDK.

Public APIs

CosmosClientBuilder

Class Name	Method Name	Return Type	Comment
CosmosClientBuilder	WithClientTelemetryOptions(CosmosClientTelemetryOptions options)	Returns the client telemetry config instance for this builder	Single function to control/configure any kind of telemetry

CosmosClientOptions

Class Name	Method Name	Return Type	Default value	Comment
CosmosClientOptions	CosmosClientTelemetryOptions	Set ClientTelemetry Options	new CosmosClientTelemetryOptions() (Loaded with default values)	Option to control/configure any kind of telemetry

CosmosClientTelemetryOptions

Class Name	Method Name	Return Type	Default value	Comment
CosmosClientTelemetryOptions	DisableSendingMetricsToService()	void	false	Disable sending telemetry data to the service i.e. Microsoft
CosmosClientTelemetryOptions	DisableDistributedTracing()	void	PREVIEW ==> false GA ==> true	Disable Distributed Tracing feature, it means it will stop generating activities even if there are subscribers
CosmosClientTelemetryOptions	CosmosThresholdOptions(CosmosThresholdOptions options) PREVIEW	void	new CosmosThresholdOptions()	Options to configure threshold for distributed tracing (Later we can use this same config for opentelemetry metrics, similar to Java)

CosmosThresholdOptions

Class Name	Method Name	Return Type	Default value	Comment
CosmosThresholdOptions	NonPointOperationLatencyThreshold(TimeSpan span)	void	3s	Set when LatencyOverThrehold event with diagnostic string should be created for non-point operations
CosmosThresholdOptions	PointOperationLatencyThreshold(TimeSpan span)	void	1s	Set when LatencyOverThrehold event with diagnostic string should be created for point operations

[jcocchi]

I recommend DisableSendingMetricsToService() instead of DisableClientTelemetryToService() to avoid overloading "Client Telemetry". This also aligns better with the proposed feature name of "Send Client Diagnostic Metrics To Service"

[FabianMeiswinkel]

Please check whether you really want to go down the road of putting the latency thresholds behind a class named explicity for tracing only

In Java DiagnosticThresholds have more meaning than for tracing - and I think it has proven to be very useful

• used even to decide whether to emit Request-Level metrics - metrics can have dimensions based on replicaId - allowing to also restrict when to emit these metrics allows to avoid overloading the metric system with metrics having way too high cardinality on dimensions
• used for distributed tracing
• and used for logging (probably not relevant for .Net)

So, main ask is to check whether you forsee any ask to also use the same thresholds for metrics - and of course whether .Net needs mroe grnularity on thresholds than just latency (java allows customizing whether to consider a threshold violation based on StatusCode+SubStatusCode), RU-usage, payload size and latency

[kirankumarkolli]

Eventually, Is distinction between metrics and push to client telemetry endpoint required?
If so do API needs to reflect that?

[kirankumarkolli]

PointOperation: Very unintutive, how is it done for JAVA?

[sourabh1007]

In Java API metrics means Open telemetry Metrics.

Thresholds are applicable only for the data which we are sending to customer

[sourabh1007]

PointOperation: Very unintutive, how is it done for JAVA?

there it has similar name. Please refer above Java API screenshots in the description.

[kirankumarkolli]

CosmosDiagnosticsThreshold: Is it applicable to all diagnostics or only for DistributedTracing?

[jcocchi]

Sounds like right now these thresholds will only apply to DistributedTracing, but eventually they may also apply to other metrics/ telemetry.

+1 on adding a threshold for RU charge and payload size.

How would a status code threshold work? We already filter out sending diagnostics for any successful request in Distributed Tracing, so would this be if they want further control over non-success scenarios or is the suggestion to give the user full control and we don't do any filtering by default?

[sourabh1007]

@kirankumarkolli These threshold configurations are applicable for events generated with diagnostics string as part of Distributed Tracing feature.

@jcocchi regarding, adding a threshold for RU charge and payload size. Right now, we don't have any requirement or ask for .Net SDK. We need to discuss if we want to implement it. cc @kirankumarkolli

Regarding status code threshold, are you talking about SetFailureHandler function? From description, it sounds like that only, customer can filter the events by statuscode and substatuscode. @FabianMeiswinkel can confirm it.

[sourabh1007]

@kirankumarkolli @jcocchi @FabianMeiswinkel are we closing on above proposal?