Skip to content

Commit 0db4b57

Browse files
authored
[CI] Make build-start-operator.sh fail fast on setup errors (#4893)
* [CI] Make build-start-operator.sh fail fast on setup errors Closes #4879. The script previously chained: IMG=kuberay/operator:nightly make docker-image && kind load docker-image kuberay/operator:nightly && echo "Deploying operator with test overrides ..." IMG=kuberay/operator:nightly make deploy-with-override If `make docker-image` failed (e.g. transient 504 from registry-1.docker.io), the `&&` chain short-circuited but `make deploy-with-override` still ran as a separate command. The script returned the exit code of the last command (0), so downstream `kubectl wait deployment kuberay-operator` then timed out because the operator image was never built/loaded, hiding the real cause. Fix: enable `set -euo pipefail` and remove the `&&` chain so any failing step aborts the script immediately and propagates a non-zero exit code to callers. Also use the `:-0` default for IS_FROM_RAY_RELEASE_AUTOMATION so `set -u` does not error when the variable is unset. This affects all Buildkite e2e steps that source this script. Behavior is unchanged on the success path; on failure, the real cause now surfaces instead of a misleading readiness timeout downstream. * [CI] Address bot feedback: drop set -u to avoid nounset leaking to callers Both Codex and Cursor Bugbot flagged that `set -u` propagates from this sourced script to the calling Buildkite shell. Downstream steps in `.buildkite/test-e2e.yml` and `.buildkite/test-historyserver-e2e.yml` reference `${KUBERAY_TEST_RAY_IMAGE}` without a default value, which would error under nounset when the non-release path leaves the variable unset. Drop `-u` while keeping `-e` and `-o pipefail`, which still gives us the fail-fast behavior needed to fix #4879 without breaking downstream references. Add an inline comment explaining the constraint so this is not reintroduced. Revert the unneeded `${IS_FROM_RAY_RELEASE_AUTOMATION:-0}` default since nounset is no longer in effect. * ci: retrigger build The previous run failed in Test Apiserver E2E (nightly operator) on TestCreateClusterAutoscaler, a known-flaky apiserver e2e test tracked upstream (kuberay#3293, kuberay#3670). The failure is unrelated to this PR's changes to build-start-operator.sh; the operator and apiserver both started successfully in that run. Retriggering CI via an empty commit to confirm the flake.
1 parent a284b82 commit 0db4b57

1 file changed

Lines changed: 10 additions & 3 deletions

File tree

.buildkite/build-start-operator.sh

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,27 @@
11
#!/bin/bash
2+
set -eo pipefail
23

34
# This script is used to start the operator in the buildkite test-e2e steps.
45

56
# When starting from the ray ci release automation, we want to install the latest
67
# released version from helm as actual users might. Ray ci is also always expected
78
# to kick off from the release branch so tests should match up accordingly.
9+
#
10+
# NOTE: Do not enable `set -u` (nounset) here. This script is sourced by the
11+
# Buildkite steps, so shell options propagate to the caller; downstream steps
12+
# reference $KUBERAY_TEST_RAY_IMAGE without a default and would fail under -u
13+
# when the variable is not set (e.g. in the non-release path).
814

915
if [ "$IS_FROM_RAY_RELEASE_AUTOMATION" = 1 ]; then
1016
helm repo update
1117
echo "Installing helm chart with test override values (feature gates enabled as needed)"
1218
# NOTE: The override file is CI/test-only. It is NOT part of the released chart defaults.
1319
helm install kuberay-operator kuberay/kuberay-operator -f ../.buildkite/values-kuberay-operator-override.yaml
14-
KUBERAY_TEST_RAY_IMAGE="rayproject/ray:nightly-extra-py310-cpu" && export KUBERAY_TEST_RAY_IMAGE
20+
KUBERAY_TEST_RAY_IMAGE="rayproject/ray:nightly-extra-py310-cpu"
21+
export KUBERAY_TEST_RAY_IMAGE
1522
else
16-
IMG=kuberay/operator:nightly make docker-image &&
17-
kind load docker-image kuberay/operator:nightly &&
23+
IMG=kuberay/operator:nightly make docker-image
24+
kind load docker-image kuberay/operator:nightly
1825
echo "Deploying operator with test overrides (feature gates via test-overrides overlay)"
1926
IMG=kuberay/operator:nightly make deploy-with-override
2027
fi

0 commit comments

Comments
 (0)