Secure the ksvc, QP prometheus endpoints etc

[skonto]

Describe the feature

Serving exposes prometheus metrics in the control plane for ksvcs ( via autoscaler and controller svc), and uses functionality from pkg metrics in order to create a prometheus exporter, that listens to a specific path/port by default. This is not protected in any way.
This is problematic for multi-tenant users as any pod can hit a svc in the same cluster and leak information about ksvcs running on cluster by listing the related metrics.
In general this affects all prometheus endpoints.

Some solutions:
a) when serving mesh is used use the provided capabilities for example for istio we can have both [encryption and basic auth](For mutual tls plus token auth (more complex) we could use what the mesh provides for example:
https://istio.io/latest/docs/tasks/security/authentication/authn-policy/#globally-enabling-istio-mutual-tls-in-strict-mode
https://istio.io/latest/blog/2020/proxy-cert/).

b) use a proxy as a side car to manage access, for example kube-rbac-proxy

/cc @vagababov @markusthoemmes @nak3 (credit to @nak3 who initially reported this). WDYT?

[skonto]

@evankanderson @yanweiguo WDYTH?

[evankanderson]

I'd like to move a lot of this to the opencensus collector or the equivalent OpenTelemetry component when available. This would change the pattern from a "pull" from potentially short-lived components to a "push" to a central component where ACLs and policy could be enforced. (The opencensus collector would also support more backends for metrics without needing to build a bunch of infrastructure in Knative to manage those different export options.)

Would that work for your use-case?

[mattmoor]

/assign @skonto

Assigning for more info.

[skonto]

@evankanderson @mattmoor in my use case I need to secure the existing APIs assuming a pull model using Prometheus.
Current target is Openshift where Prometheus is used for gathering metrics via scraping (/cc @s-urbaniak).
I understand that moving things towards a push model with a "generic" collector would decouple knative metrics acquisition from any specific system and would be a better fit for short-lived logic. However, this is a no go at the moment as there is no automatic integration with openCensus collector in knative and this would add more complexity for existing systems (the more you add the more you have to manage). Also I am wondering how security (authentication, encryption) is enforced between each component exporter <-> agent <-> collector <-> Prometheus, I didnt find much info, only a diagram here, any pointer would be helpful.

[tcnghia]

cc @igorbelianski

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[markusthoemmes]

/remove-lifecycle stale