Ingestion service data delivery status

[rajkumar-rangaraj]

Ingestion service delivery status

Users could track ingestion service response by subscribing to transmission event. Notification is sent to subscriber whenever channel makes an attempt to send telemetry to the Ingestion service. The notification event includes information about the response, and also the telemetry items which were part of the transmission. Subscriber could extract information from ingestion response. For example, following metrics could be extracted.

• How much traffic was accepted/rejected by ingestion service?
• What instrumentation keys were associated with the traffic?

Changes

(Please provide a brief description of the changes here.)

• Event handler TransmissionStatusEvent in Transmission could be wired up through ServerTelemetryChannel. Subscriber will utilize a public property in ServerTelemetryChannel to attach an event

// Sample
class Program
    {
        static void Main(string[] args)
        {
            var config = TelemetryConfiguration.CreateDefault();

            ServerTelemetryChannel channel = new ServerTelemetryChannel();
            channel.Initialize(config);
            config.TelemetryChannel = channel;
            config.InstrumentationKey = "IKEY";
            channel.TransmissionStatusEvent += delegate (object sender, TransmissionStatusEventArgs transmissionStatusEventArgs)
            {
                var transmission = sender as Transmission;

                if (transmissionStatusEventArgs.Response.StatusCode != (int)HttpStatusCode.PartialContent)
                {
                    Console.WriteLine($"Status Code: {transmissionStatusEventArgs.Response.StatusCode}");
                }
                else
                {
                    BackendResponse backendResponse = GetBackendResponse(transmissionStatusEventArgs.Response.Content);

                    Console.WriteLine($"ItemsAccepted: {backendResponse.ItemsAccepted}");
                    Console.WriteLine($"ItemsReceived: {backendResponse.ItemsReceived}");
                    
                    foreach (var error in backendResponse.Errors)
                    {
                        Console.WriteLine($"Error Index: {error.Index}");
                        Console.WriteLine($"Error Message: {error.Message}");
                    }
                }

                var iKeyGroup = transmission.TelemetryItems.GroupBy(x => x.Context.InstrumentationKey);

                foreach (var telemetry in iKeyGroup)
                {
                    Console.WriteLine($"Ikey: {telemetry.Key}, Count: {telemetry.Count()}");
                }

            };

            var client = new TelemetryClient(config);
            SendTelemetryToApplicationInsights(client);
            Console.ReadLine();
        }

        private static void SendTelemetryToApplicationInsights(TelemetryClient client)
        {
            EventTelemetry eventTelemetry1 = new EventTelemetry("Event1");
            eventTelemetry1.Context.InstrumentationKey = "IKEY_1";
            client.TrackEvent(eventTelemetry1);

            EventTelemetry eventTelemetry2 = new EventTelemetry("Event2");
            eventTelemetry2.Context.InstrumentationKey = "IKEY_2";
            client.TrackEvent(eventTelemetry2);

            EventTelemetry eventTelemetry3 = new EventTelemetry("Event3");
            eventTelemetry3.Context.InstrumentationKey = "6c49c07c-e95c-48fe-8a7b-eff230955cc5";
            client.TrackEvent(eventTelemetry3);

            EventTelemetry eventTelemetry4 = new EventTelemetry("Event3");
            eventTelemetry4.Context.InstrumentationKey = "IKEY_2";
            client.TrackEvent(eventTelemetry4);

            EventTelemetry eventTelemetry5 = new EventTelemetry("Event5");
            eventTelemetry5.Context.InstrumentationKey = "IKEY_1";
            client.TrackEvent(eventTelemetry5);

            client.Flush();
        }

        [DataContract]
        private class BackendResponse
        {
            [DataMember(Name = "itemsReceived", IsRequired = true)]
            public int ItemsReceived { get; set; }

            [DataMember(Name = "itemsAccepted", IsRequired = true)]
            public int ItemsAccepted { get; set; }

            [DataMember(Name = "errors")]
            public Error[] Errors { get; set; }

            [DataContract]
            private class Error
            {
                [DataMember(Name = "index")]
                public int Index { get; set; }

                [DataMember(Name = "statusCode")]
                public int StatusCode { get; set; }

                [DataMember(Name = "message")]
                public string Message { get; set; }
            }
        }

        private static BackendResponse GetBackendResponse(string responseContent)
        {
            BackendResponse backendResponse = null;
            DataContractJsonSerializer Serializer = new DataContractJsonSerializer(typeof(BackendResponse));

            try
            {
                if (!string.IsNullOrEmpty(responseContent))
                {
                    using (MemoryStream ms = new MemoryStream(Encoding.Unicode.GetBytes(responseContent)))
                    {
                        backendResponse = Serializer.ReadObject(ms) as BackendResponse;
                    }
                }
            }
            catch
            {
                backendResponse = null;
            }

            return backendResponse;
        }
    }

For another example, please refer test method TestTransmissionStatusEventWithEventsFromMultipleIKey in TransmissionTest.cs

• Events Notification is initiated on SendAsync completion (once response is received from ingestion service) in Transmission. This passes Sender as Microsoft.ApplicationInsights.Channel.Transmission and TransmissionStatusEventArgs with Microsoft.ApplicationInsights.Extensibility.Implementation.HttpWebResponseWrapper to subscriber
• Subscriber is responsible to handle event and extract information from Transmission and HttpWebResponseWrapper
• If there are no subscriber, event notification won’t be initiated

Checklist

• I ran Unit Tests locally.
• CHANGELOG.md updated with one line description of the fix, and a link to the original issue if available.

[cijothomas]

left comments.

[cijothomas]

Left comments.
We need sample usage code in the PR description to help users onboard to this feature as well.

[ivshch]

I was looking through the PR code and got a question:
what if the request is failed with the HttpRequestException (in case of general network failure), it seems that the delegate is not invoked?
This probably suits the initial idea of gathering metric of successful requests, but if I'm correct it makes it unusable in our scenario #1974 - detecting the connection loss.

Would the metric be accurate if it does not include the failures?
Could it improve your solution if the delegate allows collecting failures also?

[rajkumar-rangaraj]

Left comments.
We need sample usage code in the PR description to help users onboard to this feature as well.

Plan was to add complete working solution in samples folder. I mentioned it n TODO section of PR description. Should we provide complete sample in this PR description itself?

[reyang]

@rajkumar-rangaraj if customers are using the in-memory channel, is there a way to subscribe to the transmission status update?

Event handler TransmissionStatusEvent in Transmission is wired up through ServerTelemetryChannel, subscriber will utilize a public property in ServerTelemetryChannel to attach the event

[rajkumar-rangaraj]

@rajkumar-rangaraj if customers are using the in-memory channel, is there a way to subscribe to the transmission status update?

Event handler TransmissionStatusEvent in Transmission is wired up through ServerTelemetryChannel, subscriber will utilize a public property in ServerTelemetryChannel to attach the event

Changes will work only for ServerTelemetryChannel. As there are no retry logic did not expose public property from InMemoryChannel. Changes would be simple to enable for InMemoryChannel, we just need to expose property in channel and serializer.

[reyang]

LGTM.