As per #350 our goal is to support various traffic flavors and support them in a neutral way.
The tricky thing here is that there is no standard way of having metrics for these traffic flows and we have to provide our own, use metrics from ingress controllers, or rely on Service Mesh Interface's Traffic Metrics API.
We should move to a standardized approach so that:
- We define and use standard for traffic metrics to have a unified approach, regardless of the traffic type/source
- In a later stage, we can propose this metric to TAG Network as an open standard beyond the scope of KEDA
- We unify all these metrics into a central place and use that as input for our external scaler
- Change our interceptor so that it publishes the metrics it has today in to our central place above
- There is a chance that we could remove this from the interceptor, but most likely we will still need it for our service-to-service support
Proposal
OpenTelemetry Metrics are about to go stable and is an open standard for using metrics in systems.
Standardizing on OpenTelemetry & its Collector
Our interceptor should be changed so that it can publish its metrics to an OpenTelemetry Collector so that we can bring the metrics where we need them and end-users can re-use these metrics for their own purposes:
These metrics should comply with the defined HTTP semantics as per this doc.
Once the metrics are available, we can choose one of the existing exporters (full overview) to consume the metrics by pushing metrics to our external scaler directly (HTTP-based or gRPC-based, preferred approach) or through an external system such as Prometheus (less preferred).
When end-users install the HTTP add-on, we should automatically install a collector, unless they opt-out and configure a different endpoint. However, ideally, we fully manage and configure the collector with all the bells and whistles that we need.
Bringing existing traffic metrics into our standardized metrics approach
In order to bring existing traffic metrics into our way of working we will need two components:
- An adapter per traffic source to pull the metrics and make them available in the collector
- A custom
processor to transform the source metrics format to our standardized metrics format (learn more)
Some traffic sources might already be supported through an existing receiver.
For example, it would make sense to have an SMI-receiver that we can rely on instead of rolling our own. (servicemeshinterface/smi-spec#199)
Traffic Metrics Spec
SMI has its Traffic Metrics spec and OpenTelemetry is defining semantics for HTTP metrics.
We should aim to use those before rolling our own standard.
As per #350 our goal is to support various traffic flavors and support them in a neutral way.
The tricky thing here is that there is no standard way of having metrics for these traffic flows and we have to provide our own, use metrics from ingress controllers, or rely on Service Mesh Interface's Traffic Metrics API.
We should move to a standardized approach so that:
Proposal
OpenTelemetry Metrics are about to go stable and is an open standard for using metrics in systems.
Standardizing on OpenTelemetry & its Collector
Our interceptor should be changed so that it can publish its metrics to an OpenTelemetry Collector so that we can bring the metrics where we need them and end-users can re-use these metrics for their own purposes:
These metrics should comply with the defined HTTP semantics as per this doc.
Once the metrics are available, we can choose one of the existing exporters (full overview) to consume the metrics by pushing metrics to our external scaler directly (HTTP-based or gRPC-based, preferred approach) or through an external system such as Prometheus (less preferred).
When end-users install the HTTP add-on, we should automatically install a collector, unless they opt-out and configure a different endpoint. However, ideally, we fully manage and configure the collector with all the bells and whistles that we need.
Bringing existing traffic metrics into our standardized metrics approach
In order to bring existing traffic metrics into our way of working we will need two components:
receiverprocessorto transform the source metrics format to our standardized metrics format (learn more)Some traffic sources might already be supported through an existing
receiver.For example, it would make sense to have an SMI-receiver that we can rely on instead of rolling our own. (servicemeshinterface/smi-spec#199)
Traffic Metrics Spec
SMI has its Traffic Metrics spec and OpenTelemetry is defining semantics for HTTP metrics.
We should aim to use those before rolling our own standard.