[Test] Add load tests and behavioral checks to incremental upgrade E2E

[JiangJiaWei1103]

Why are these changes needed?

The existing RayService Incremental Upgrade E2E test is implemented as a single, monolithic functional test and doesn't include load testing. Hence, it doesn't verify system behavior under real traffic during upgrades. In addition, the current functional test doesn't provide comprehensive behavioral checks.

This PR introduces Locust-based load tests to validate incremental upgrade behavior under continuous traffic. It also adds more thorough behavioral checks to the functional test. Both test cases are executed across multiple upgrade strategies to improve test coverage.

Test Summary

RayCluster Setup

To run the Locust load tests, two RayClusters are required: one acting as the client (Locust) and the other managed by the RayService.

Locust Cluster (Client-Side)

The client cluster configuration follows the example defined here.

Component	Replicas	CPU (req/limit)	Memory (req/limit)
Head	1	300m / 500m	1Gi / 2Gi
Worker	0 (head-only)	-	-

RayService Cluster (Server-Side)

Worker resource limits are intentionally not set to align with the original example proposed by @Future-Outlier here.

UPDATE:

Without resource limits, workers may consume more CPU and memory than their requested resources. This can lead to node-level resource contention and cause the incremental upgrade process to time out. To avoid this issue, resource limits are kept for worker pods, particularly in CI environments where compute resources are constrained (e.g., Buildkite runners with 8 vCPUs).

In addition, setting explicit CPU requests for the head pod can make the pod unschedulable in single-node test environments due to Insufficient cpu. To ensure the test environment remains schedulable, the head is configured with rayStartParams["num-cpus"]: 0 while keeping minimal resource requests.

Component	Replicas (min/max)	CPU (req/limit)	Memory (req/limit)
Head	1	- (rayStartParams["num-cpus"]: 0)	-
Worker	1 / 4	2 / 2	2Gi / 2Gi

Test Matrix

Both test cases are executed with multiple upgrade strategies to improve coverage.

Upgrade Strategy	(maxSurgePercent, stepSizePercent, intervalSeconds)	Description
BlueGreen	(100, 100, 1)	Instant traffic cutover
AggressiveGradual	(50, 25, 2)	Larger traffic migration steps with shorter intervals
ConservativeGradual	(25, 5, 10)	Smaller traffic migration steps with longer intervals

Behavioral Checks

To improve the robustness of the e2e tests, we introduce additional behavioral checks, including verification that TargetCapacity and TrafficRoutedPercent progress monotonically. See the Change Summary below for details.

Test Results

Throughput

The CI throughput (~500 RPS, and sometimes ~450) is approximately half of the local test (~1000+ RPS). This is likely due to the smaller compute capacity of the Buildkite hosted runners:

Buildkite large instances provide 8 vCPUs, where each vCPU typically maps to one logical thread (hyper-thread) on a physical core. This means the available compute might be roughly half that of our local test machine. Furthermore, CI providers may enforce cgroup limits or CPU throttling, which can cause additional performance degradation. For full instance specifications, refer to the Buildkite hosted Linux sizes documentation.

NOTE: The greatest instance supported by the Ray ecosystem CI is large. For error on selecting xlarge, please see this CI failure.

Local (~1000+ RPS)	CI (~500 RPS)

Overall E2E

TestRayServiceIncrementalUpgrade	TestRayServiceIncrementalUpgradeWithLocust	TestRayServiceIncrementalUpgradeRollback

Change Summary

TestRayServiceIncrementalUpgrade

Add comprehensive behavioral checks covering:

• The current state value matches the expected value, or the RayService has already finished upgrading
• Both old and new versions serve traffic during the upgrade, ensuring no requests are dropped
• Traffic migration respects the configured interval seconds
• Active TargetCapacity is monotonically decreasing while Pending TargetCapacity is monotonically increasing
• Active TrafficRoutedPercent is monotonically decreasing while Pending TrafficRoutedPercent is monotonically increasing

TestRayServiceIncrementalUpgradeWithLocust

• Add a new E2E test case TestRayServiceIncrementalUpgradeWithLocust that runs Locust load tests
  • Run Locust in a background goroutine
  • Warm up Locust first (entering a steady state) before triggering the upgrade
  • Verify that no requests are dropped during the upgrade under load
• Add RayCluster and ConfigMap manifests required to run the Locust load tests
• Add a new ServeConfigV2 configuration with a lightweight Serve application that directly returns a response
  • This is designed for high-RPS load testing scenarios

Limitations and Future Improvements

• The Serve application source code is currently hosted in my personal repo. It should be migrated to an official ray-project organization repository.
  • We also considered mounting the Serve app via a ConfigMap, but working_dir currently does not support local paths.
  • (Solved) We've changed the working_dir to https://github.com/ray-project/serve_config_examples.
• Ray and image versions are currently hardcoded in locust-cluster.incremental-upgrade.yaml, following the same pattern used in the existing HA E2E tests here.
• Locust warm-up parameters are currently defined as named constants. We may need to further tune the ServeConfigV2 implementation and the RayCluster configuration to support higher RPS in CI:
  • Current throughput: ~500 RPS
  • Target throughput: 1000+ RPS

Related issue number

#3209

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[JiangJiaWei1103]

Current Local E2E Test Results

The next steps are:

• Address flaky parts (see TODO in the e2e test file)
• Add more test cases with diverse (stepSize, interval, maxSurge) combinations

[JiangJiaWei1103]

The setup order is rearranged to align with the official Ray docs example, so developers can follow along without wondering why the steps differ.

[JiangJiaWei1103]

Reverted at dc98432 due to duplicate installation of Istio GatewayClass. We might revisit this in the future.

[cursor]

BlueGreen scenario may fail both-versions-served assertion

Medium Severity

For the BlueGreen scenario (stepSize=100, interval=1, maxSurge=100), the upgrade may complete during the preceding validation steps (pending head pod readiness, HTTPRoute backend checks at lines 106–123), since the entire traffic shift happens in a single 1-second step. When the behavioral check loop starts, g.Eventually can succeed via the !IsRayServiceUpgrading(svc) escape clause, the curl returns only "8" (new version), and the loop breaks — leaving oldVersionServed as false. The assertion at line 205 then fails. This is a race condition that could cause flaky test failures.

Additional Locations (1)

• ray-operator/test/e2eincrementalupgrade/rayservice_incremental_upgrade_test.go#L140-L166

 

[JiangJiaWei1103]

This can happen when the upgrade completes too quickly, even before the test enters the upgradeSteps loop. In practice, this scenario is rare.

For now, I suggest keeping the current behavior unchanged. We can revisit whether it is reasonable to assert that both clusters should serve traffic during the upgrade for the Blue/Green strategy, which is effectively a single-step upgrade rather than a gradual traffic migration.

[JiangJiaWei1103]

We increase the timeout to deflake the e2e test.

[JiangJiaWei1103]

For resources setup for the locust RayCluster, we follow practices here:

kuberay/ray-operator/test/e2erayservice/testdata/locust-cluster.const-rate.yaml

Lines 13 to 19 in 44ae9e2

	resources:
	requests:
	cpu: 300m
	memory: 1G
	limits:
	cpu: 500m
	memory: 2G

[JiangJiaWei1103]

Thanks Ryan!! I'll change the URL once the PR is merged.

[JiangJiaWei1103]

The resource setup is mainly constrained by buildkite hardware limitation (8 vCPU). For details, please refer to the PR description.

[JiangJiaWei1103]

ditto

[ryanaoleary]

I recommend adding:

WithRayClusterDeletionDelaySeconds(0).

here since the default deletion delay is 60 seconds, which adds unnecessary lag to the test since we check for cluster deletion. We could lower it to 0 or even just a value like 10 seconds to speed up these tests.

[JiangJiaWei1103]

Hi @ryanaoleary,

I noticed that only TestRayServiceIncrementalUpgradeRollback verifies whether the pending cluster is deleted after the rollback completes.

For TestRayServiceIncrementalUpgrade and TestRayServiceIncrementalUpgradeWithLocust, neither test checks whether the previous active clusters are cleaned up. Instead, they focus on verifying that traffic is correctly served by the new cluster (i.e., the newly promoted active cluster). Therefore, the RayClusterDeletionDelaySeconds setup wouldn't help speed up the e2e tests at this stage.

I suggest the following adjustments in this PR:

1. Remove the rollback E2E tests for now, since this PR will be merged before the rollback logic itself.
2. Reintroduce the rollback logic along with the corresponding rollback E2E tests. For each test, we should:
   • Run it under Locust load
   • Enable WithRayClusterDeletionDelaySeconds to accelerate verification that the pending cluster is cleaned up.

WDYT? If I misunderstood anything, please let me know. Thanks!

[JiangJiaWei1103]

The rollback e2e has been removed at baf2d6c. I'll merge master again once #4604 is merged. Thanks!

[JiangJiaWei1103]

We add the basic rollback e2e back at 44fdb7e. And, we'll enhance rollback e2e coverage in the follow-up PRs.

[ryanaoleary]

LGTM - just one small comment on the config that gets applied and the test path needs to be updated when ray-project/serve_config_examples#15 is merged.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

[cursor]

Warmup stableCount not reset on error retries

Medium Severity

In warmupLocust, when the stats query fails (stderr non-empty/stdout empty), the stats slice is too short, or float parsing fails, the loop continues without resetting stableCount. Since locustWarmupStableWindowSeconds represents consecutive seconds of stability, intermittent failures during the stable window are silently skipped, and the function can prematurely declare steady state based on non-consecutive stable checks.

 

[JiangJiaWei1103]

We reset stableCount only when the RPS value is successfully queried and parsed, and is below the rpsThreshold:

https://github.com/ray-project/kuberay/pull/4541/changes#diff-e421bf294e3026a8e3ee0aad96d7d11b2e1714e705fb859d1889bceac8cd2ba5R426-R430

The three cases you mentioned are commonly observed formatting/parsing issues, which don't reliably indicate that the actual RPS is below the threshold. Therefore, we don't reset stableCount in those scenarios.

[ryanaoleary]

Since #4604 is closed I think we should actually keep the changes from 44fdb7e

[win5923]

Sorry for the late review, I spent some time getting familiar with the incremental upgrade PR.
Overall LGTM. Although I wasn’t able to reach 400 RPS in my local tests, the CI results look stable, which should be sufficient.