Skip to content

ci: split orchestrator integrity into parallel jobs for faster validation #809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
frostebite wants to merge 3 commits into from
Closed

ci: split orchestrator integrity into parallel jobs for faster validation #809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9789eb5
ci: split orchestrator integrity into parallel jobs for faster valida…
frostebite Mar 5, 2026
d21188e
ci: set macOS builds to continue-on-error
frostebite Mar 5, 2026
9579230
fix: replace orchestrator-develop branch references with main
frostebite Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/build-tests-mac.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ jobs:
buildForAllPlatformsMacOS:
name: ${{ matrix.targetPlatform }} on ${{ matrix.unityVersion }}
runs-on: macos-latest
continue-on-error: true
Copy link

@coderabbitai coderabbitai bot Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, check if the files exist and their structure
echo "=== Checking .github/workflows/build-tests-mac.yml ===" 
head -20 .github/workflows/build-tests-mac.yml

echo -e "\n=== Checking build-tests-ubuntu.yml lines 136-195 ===" 
sed -n '136,195p' .github/workflows/build-tests-ubuntu.yml

echo -e "\n=== Checking build-tests-windows.yml lines 72-143 ===" 
sed -n '72,143p' .github/workflows/build-tests-windows.yml

Repository: game-ci/unity-builder

Length of output: 5909

🏁 Script executed:

# Get the full build-tests-mac.yml to see complete step definitions
cat .github/workflows/build-tests-mac.yml

Repository: game-ci/unity-builder

Length of output: 3097

Move continue-on-error to step scope with an explicit retry mechanism.

The job-level continue-on-error: true suppresses all macOS matrix failures at the job level, making the workflow pass even when builds consistently fail. This also means macOS gets zero retry attempts, unlike Ubuntu and Windows workflows which implement a three-attempt retry with escalating delays (60s/120s → 240s) between each try. The correct pattern is step-level continue-on-error on the first two attempts only, with a final third attempt without continue-on-error to preserve the failure signal. This allows transient failures to self-heal while still catching real issues.

Mirror the retry structure from .github/workflows/build-tests-ubuntu.yml:136-195 and .github/workflows/build-tests-windows.yml:72-143:

  • Build attempt 1: id: build-1, continue-on-error: true, always runs
  • Sleep step: runs on steps.build-1.outcome == 'failure' (60–120 sec delay)
  • Build attempt 2: id: build-2, continue-on-error: true, conditional on build-1 failure
  • Sleep step: runs on steps.build-2.outcome == 'failure' (240 sec delay)
  • Build attempt 3: id: build-3, no continue-on-error, conditional on build-2 failure (final attempt, lets failure propagate)

Remove the job-level continue-on-error: true and implement the step-based retry pattern instead.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build-tests-mac.yml at line 15, Remove the job-level
continue-on-error and implement a step-scoped retry sequence mirroring
Ubuntu/Windows workflows: create three build steps with ids build-1 (always
runs, continue-on-error: true), build-2 (runs only if steps.build-1.outcome ==
'failure', continue-on-error: true), and build-3 (runs only if
steps.build-2.outcome == 'failure' and has no continue-on-error so failures
propagate); add sleep steps after build-1 and build-2 conditioned on their
failure outcomes with delays 60–120s and 240s respectively; ensure step ids and
conditional expressions (steps.build-1.outcome, steps.build-2.outcome) are used
exactly as referenced so the final build-3 signals a real failure.

strategy:
fail-fast: false
matrix:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/orchestrator-async-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
# AWS_STACK_NAME: game-ci-github-pipelines
CHECKS_UPDATE: ${{ github.event.inputs.checksObject }}
run: |
git clone -b orchestrator-develop https://github.com/game-ci/unity-builder
git clone -b main https://github.com/game-ci/unity-builder
cd unity-builder
yarn
ls
Expand Down
1,031 changes: 577 additions & 454 deletions .github/workflows/orchestrator-integrity.yml
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 2 additions & 4 deletions dist/index.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/index.js.map
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/model/orchestrator/tests/e2e/orchestrator-end2end-caching.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ describe('Orchestrator Caching', () => {
targetPlatform: 'StandaloneLinux64',
cacheKey: `test-case-${uuidv4()}`,
containerHookFiles: `debug-cache`,
orchestratorBranch: `orchestrator-develop`,
orchestratorBranch: `main`,
orchestratorDebug: true,
};

Expand Down
3 changes: 1 addition & 2 deletions src/model/orchestrator/workflows/async-workflow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,7 @@ if [ -n "$(git ls-remote --heads "$REPO" "$BRANCH" 2>/dev/null)" ]; then
git clone -q -b "$BRANCH" "$REPO" /builder
else
echo "Remote branch $BRANCH not found in $REPO; falling back to a known branch"
git clone -q -b orchestrator-develop "$REPO" /builder \
|| git clone -q -b main "$REPO" /builder \
git clone -q -b main "$REPO" /builder \
|| git clone -q "$REPO" /builder
fi
git clone -q -b ${Orchestrator.buildParameters.branch} ${OrchestratorFolders.targetBuildRepoUrl} /repo
Expand Down
3 changes: 1 addition & 2 deletions src/model/orchestrator/workflows/build-automation-workflow.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,7 @@ if [ -n "$(git ls-remote --heads "$REPO" "$BRANCH" 2>/dev/null)" ]; then
git clone -q -b "$BRANCH" "$REPO" "$DEST"
else
echo "Remote branch $BRANCH not found in $REPO; falling back to a known branch"
git clone -q -b orchestrator-develop "$REPO" "$DEST" \
|| git clone -q -b main "$REPO" "$DEST" \
git clone -q -b main "$REPO" "$DEST" \
|| git clone -q "$REPO" "$DEST"
fi
chmod +x ${builderPath}`;
Expand Down
Loading