test: refactor integration test harness to support data layer testing by Mohamedma96 · Pull Request #2664 · kubernetes-sigs/gateway-api-inference-extension

Mohamedma96 · 2026-03-23T13:59:48Z

Summary

Refactors the integration test harness to support testing both the legacy backend/metrics and the new data layer architecture. This enables comprehensive validation of the data layer metrics pipeline using mock data sources.

What type of PR is this?
/kind test

What this PR does / why we need it:
The data layer architecture introduces a new way to collect and process metrics from model servers. We need integration tests that validate this new pipeline works correctly while maintaining test coverage for the existing legacy metrics approach.

Which issue(s) this PR fixes:

Partially Fixes #2638

Does this PR introduce a user-facing change?:

NONE

Key changes:

Extract common test cases into commonTestCases() function in util.go
Add new datalayer_integration_test.go with TestFullDuplexStreamed_DataLayer
Enhance harness.go with data layer configuration support (WithDataLayer option)
Add mock data source setup and configuration helpers

Pull Request Title
test: refactor integration test harness to support data layer testing

Pull Request Description

Summary

Refactors the integration test harness to support testing both the standard metrics client and the new data layer architecture. This enables comprehensive validation of the data layer metrics pipeline using mock data sources.

Motivation

The data layer architecture introduces a new way to collect and process metrics from model servers. We need integration tests that validate this new pipeline works correctly while maintaining test coverage for the existing standard metrics client approach.

Changes

Test Infrastructure

test/integration/epp/util.go: Extracted common test cases into commonTestCases() function that can be shared across test suites
test/integration/epp/harness.go:
- Added WithDataLayer() option to configure test harness with mock data sources
- Added data layer configuration constants (mockDataSourceType, testDLConfig)
- Enhanced harness to support both standard and data layer execution modes

New Test Suite

test/integration/epp/datalayer_integration_test.go: New integration test suite that:
- Runs the same test scenarios as hermetic tests
- Uses mock DataSource instead of FakePodMetricsClient
- Validates the complete data layer metrics pipeline

Test Refactoring

test/integration/epp/hermetic_test.go: Updated to use shared commonTestCases() function

k8s-ci-robot · 2026-03-23T13:59:51Z

@Mohamedma96: The label(s) kind/test cannot be applied, because the repository doesn't have them.

Details

In response to this:

Summary

Refactors the integration test harness to support testing both the legacy backend/metrics and the new data layer architecture. This enables comprehensive validation of the data layer metrics pipeline using mock data sources.

What type of PR is this?
/kind test

What this PR does / why we need it:
The data layer architecture introduces a new way to collect and process metrics from model servers. We need integration tests that validate this new pipeline works correctly while maintaining test coverage for the existing legacy metrics approach.

Which issue(s) this PR fixes:

Partially Fixes #2638

Does this PR introduce a user-facing change?:
NONE
Key changes:

Extract common test cases into commonTestCases() function in util.go

Add new datalayer_integration_test.go with TestFullDuplexStreamed_DataLayer

Enhance harness.go with data layer configuration support (WithDataLayer option)

Add mock data source setup and configuration helpers

Pull Request Title
test: refactor integration test harness to support data layer testing

Pull Request Description

Summary

Refactors the integration test harness to support testing both the standard metrics client and the new data layer architecture. This enables comprehensive validation of the data layer metrics pipeline using mock data sources.

Motivation

The data layer architecture introduces a new way to collect and process metrics from model servers. We need integration tests that validate this new pipeline works correctly while maintaining test coverage for the existing standard metrics client approach.

Changes

Test Infrastructure

test/integration/epp/util.go: Extracted common test cases into commonTestCases() function that can be shared across test suites

test/integration/epp/harness.go:

Added WithDataLayer() option to configure test harness with mock data sources

Added data layer configuration constants (mockDataSourceType, testDLConfig)

Enhanced harness to support both standard and data layer execution modes

New Test Suite

test/integration/epp/datalayer_integration_test.go: New integration test suite that:

Runs the same test scenarios as hermetic tests

Uses mock DataSource instead of FakePodMetricsClient

Validates the complete data layer metrics pipeline

Test Refactoring

test/integration/epp/hermetic_test.go: Updated to use shared commonTestCases() function

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2026-03-23T13:59:54Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`7ba5b05`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69c8ef7ad063b000080395ab
😎 Deploy Preview	https://deploy-preview-2664--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2026-03-23T13:59:59Z

Hi @Mohamedma96. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

elevran · 2026-03-23T15:53:08Z

/ok-to-test

LukeAVanDrie

This looks great, thanks!

elevran · 2026-03-23T17:04:54Z

+		fwkplugin.Register(mockType, func(_ string, _ json.RawMessage, _ fwkplugin.Handle) (fwkplugin.Plugin, error) {
+			return mockDataSource, nil
+		})
+		defer delete(fwkplugin.Registry, mockType)


this might be unavoidable as there is no Unregister, but seems odd.

There is no lazy initialization, so this is currently safe, but it is a bit brittle.

I agree it seems odd, yet I find no way to prevents registry pollution between different tests, and right, this assumes that registry map is read only during setup function and not any stage later (which is true for now), so for now this will work.

I can remove the defer delete, since only datalayer tests "know" about the mock-metrics-plugin and in case it's not deleted the tests would just overwrite or re-register the plugin which has no actual difference (re-register with same factory func).

What do you suggest?

elevran · 2026-03-23T18:32:42Z

+	mockDataSourceType = "mock-metrics-source"
+
+	// testDLConfig configures the data layer using the mock data source type.
+	testDLConfig = `


nit: might be something you want to pull in via //go:embed from an external file

elevran · 2026-03-23T18:34:10Z

+
+// mockDataSourceBackend wraps the mock DataSource to implement metricsBackend.
+// It also sets metrics on the legacy FakePodMetricsClient because Prometheus metrics
+// (e.g. inference_pool_ready_pods) are still populated by the legacy refresh loop.


Q: is this functionality missing in the data layer or just not configured by the test? If missing, please open an issue to complement.

I think setting metrics functionality is missing in datalayer

can you please review pkg/epp/datalayer/logger/logger.go and create an issue to add missing metrics?
Specifically, I think ready pods are already set on L114.

true, changing the comment, and actually the two loggers are at full parity — both record the same three metrics. The gap is not between datalayer and backend loggers, but between what fwkdl.Metrics has and what either logger publishes. RunningRequestsSize is collected per-pod but neither logger records it created this issue to handle missing logs:
#2689

elevran · 2026-03-23T18:37:02Z

+			Name: mockDataSourceType,
+		})
+		mgr, dataStore, err = eppRunner.NewTestRunnerSetup(ctx, testEnv.Config, eppOptions, fakePmc, mockDataSource)
+		require.NoError(t, err, "failed to create manager")


Q: is it common practice to use testify/require in files that are _test.go? I realize this decision was not made in the PR - seeking recommendation for future use.
It makes sense in this context since it is effectively a test file, just not named so (which is another question...)

we can change the name to harness_test.go. harness.go is only called from test code.

I can see files that has require.NoError() from before which are not _test.go like test/integration/util.go, not.

@LukeAVanDrie is in a better position to comment on the file name change as #2084 dropped the extension

Here is the original discussion thread that resulted in #2084. #2022 (comment)

I actually started with the _test suffix but was advised to remove it.

then keeping it as harness.go
Thanks

elevran · 2026-03-24T12:26:05Z

+// non-nil, its plugin type is registered as a factory that returns the provided instance, so the
+// YAML config can reference it by type name and the runner wires it into the endpoint factory
+// automatically. Pass nil to fall back to the legacy metrics system with pmc.
+func NewTestRunnerSetup(


nit: please do not break parameter list across multiple lines unless necessary (e.g., see deleted L30 for acceptable length)

elevran · 2026-03-24T12:31:47Z

+			Name: mockDataSourceType,
+		})
+		mgr, dataStore, err = eppRunner.NewTestRunnerSetup(ctx, testEnv.Config, eppOptions, fakePmc, mockDataSource)
+		require.NoError(t, err, "failed to create manager")


@LukeAVanDrie is in a better position to comment on the file name change as #2084 dropped the extension

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

elevran · 2026-03-29T10:01:31Z

/lgtm

ahg-g

Thanks, I left a couple of questions not blocking to this PR.

/approve

ahg-g · 2026-03-30T08:39:07Z

+	mockDataSourceType = "mock-metrics-source"
 )

+//go:embed testdata/default-config.yaml


this is very cool, I didn't know about it!

ahg-g · 2026-03-30T08:43:48Z

+)
+
+// TestFullDuplexStreamed_DataLayer runs integration tests through the datalayer metrics pipeline.
+// It mirrors a representative subset of hermetic_test.go test cases but uses the data layer


Will we be able to run the rest of the tests using the datalayer once the legacy metrics collection code is removed?

yes, there are common tests that are shared among the legacy metrics collection and datalayer, those common test can be used to run the tests for datalayer, the other tests in harness.go (not common) are suited for the legacy metrics collection and I would drop them when the code is removed.

k8s-ci-robot · 2026-03-30T08:44:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, Mohamedma96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…kubernetes-sigs/gateway-api-inference-extension#2664) * Create integration tests for datalayer metrics Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> * formatting Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> * review comments and code enhancements Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> * use go:embed, some code lint fixes Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> * use CommonTest and code cleanup Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> * remove unnecessary comment Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com> --------- Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

k8s-ci-robot requested review from ahg-g and robscott March 23, 2026 13:59

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 23, 2026

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 23, 2026

LukeAVanDrie reviewed Mar 23, 2026

View reviewed changes

Mohamedma96 force-pushed the test/integration-harness-refactor branch from 4f401f2 to 5ec0154 Compare March 23, 2026 18:03

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 23, 2026

elevran reviewed Mar 23, 2026

View reviewed changes

elevran reviewed Mar 24, 2026

View reviewed changes

Mohamedma96 mentioned this pull request Mar 25, 2026

Enable DataLayer by default, Automatically add Data section if missing, when datalayer is enabled #2639

Merged

Mohamedma96 force-pushed the test/integration-harness-refactor branch from 9168677 to 1fa4a5e Compare March 25, 2026 19:36

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 27, 2026

Mohamedma96 added 6 commits March 29, 2026 12:20

Create integration tests for datalayer metrics

7a723e4

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

formatting

c258eda

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

review comments and code enhancements

da7ddd1

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

use go:embed, some code lint fixes

6e314a2

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

use CommonTest and code cleanup

65aacbf

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

remove unnecessary comment

7ba5b05

Signed-off-by: mohamedmahameed <mohamed.mahameed@ibm.com>

Mohamedma96 force-pushed the test/integration-harness-refactor branch from db142e3 to 7ba5b05 Compare March 29, 2026 09:23

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 29, 2026

k8s-ci-robot assigned elevran Mar 29, 2026

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 29, 2026

ahg-g reviewed Mar 30, 2026

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 30, 2026

k8s-ci-robot merged commit b941a91 into kubernetes-sigs:main Mar 30, 2026
11 checks passed

Uh oh!

Conversation

Mohamedma96 commented Mar 23, 2026

Summary

Summary

Motivation

Changes

Test Infrastructure

New Test Suite

Test Refactoring

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Summary

Summary

Motivation

Changes

Test Infrastructure

New Test Suite

Test Refactoring

Uh oh!

netlify Bot commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Uh oh!

elevran commented Mar 23, 2026

Uh oh!

LukeAVanDrie left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elevran commented Mar 29, 2026

Uh oh!

ahg-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

netlify Bot commented Mar 23, 2026 •

edited

Loading