Use new data layer by default

[elevran]

What would you like to be added:
Enable the new extensible data layer as default instead of the current implementation.
Allow existing to run for a release as deprecated and the remove the code.
The main tasked is scoped on changing the featuregate from Enable to Disable, so that the datalayer metrics implementation runs by default. This aspect is user facing and should be noted in the PR.

Why is this needed:
Transition to extensible data layer to provide more flexible data collection for EPP decisions and avoid duplicate fixes in both implementations.
The new data layer implementation has been vetted in performance and soak test (see below) and can be used as the default.

[elevran]

/milestone Data Layer Foundational Implementation

[elevran]

work items

• validate performance and scale via benchmarking (incl. memory and goroutine)
• flip on/off by default (mostly done, currently always enabled with feature gate scoped to metrics collection only)
• ensure configuration is passed to data layer for setup (done)
• create issue (and later PR) to remove existing backend/ package

[ahg-g]

What is the timeline to enable it by default? I am hoping we can do this the next release, is that possible?

[elevran]

yes, that's possible.
The data layer functionality is enabled by default to support extensibility. The feature gate only controls metric collection using the new/previous implementation.
So you can, for example, enable v1/models collection with the previous metrics collection method. The notification based source (datastore and k8s) can also be used with / without metrics changed.

[ahg-g]

The data layer functionality is enabled by default to support extensibility. The feature gate only controls metric collection using the new/previous implementation.

Can we enable it by default as well? as long as it is disabled, it will never get soak time. Also, we recently added multi-engine support, this currently only works if the datalayer feature gate is set, right?

[ahg-g]

Actually, how did we validate that metrics collection using the new path is equivalent to the old one?

[elevran]

Good point. While the core metrics code is refactoring of the original and (barring any issues not detected when authoring or reviewing) should be the same, it is worwthwhile to add a validation test for that explicitly.
Will do so.

[elevran]

Can we enable it by default as well? as long as it is disabled, it will never get soak time.

Agree. That's the goal. I want to run through some more scale and functional testing before flipping the switch.

Also, we recently added multi-engine support, this currently only works if the datalayer feature gate is set, right?

Not sure why they depend on the new metrics collection?
If the only difference is in metric names, these can still be defined on the CLI.

[ahg-g]

A couple of things as well:

• There are more than a few "todo" comments in https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp/datalayer; are there any critical ones? pls create issues for the ones that we should do and cleanup the ones that are not.
• We need to update our integration tests to rely on the datalayer for metrics instead of the legacy metrics probing