Skip to content

LTS Infrastructure -- Combined Integration #816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
frostebite wants to merge 53 commits into from
Closed

LTS Infrastructure -- Combined Integration #816

Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
5268630
feat(orchestrator): enterprise feature support — CLI provider, submod…
frostebite Mar 5, 2026
d856336
feat(orchestrator): add experimental GCP Cloud Run and Azure ACI prov…
frostebite Mar 5, 2026
f4bc5d2
feat(orchestrator): multi-storage support for GCP and Azure providers
frostebite Mar 5, 2026
786ee37
feat(orchestrator): automatic provider fallback with runner availabil…
frostebite Mar 5, 2026
8194790
feat(orchestrator): add retry-on-fallback and provider init timeout
frostebite Mar 5, 2026
d17b099
style: format changed files with prettier
frostebite Mar 5, 2026
cfac5f1
test(orchestrator): expand local cache service test coverage
frostebite Mar 5, 2026
7e9d0bf
test(orchestrator): add runner availability service tests
frostebite Mar 5, 2026
17a0ea3
test(orchestrator): add unit tests for untested core services
frostebite Mar 5, 2026
f445106
ci(orchestrator): add fast unit test gate to integrity workflow
frostebite Mar 5, 2026
a0c79bd
test(orchestrator): expand unit tests for enterprise services
frostebite Mar 5, 2026
8a41533
fix(orchestrator): use http.extraHeader for secure git authentication
frostebite Mar 5, 2026
e4c156e
feat(orchestrator): add premade secret sources and YAML definitions
frostebite Mar 5, 2026
7f89530
feat(secrets): add HashiCorp Vault as first-class premade secret source
frostebite Mar 5, 2026
cf3478c
feat(lfs): add built-in elastic-git-storage support with auto-install
frostebite Mar 5, 2026
26903e9
feat(hooks): add Unity Git Hooks integration and runHookGroups
frostebite Mar 5, 2026
12f2871
feat(orchestrator): CI platform providers — Remote PowerShell, GitHub…
frostebite Mar 5, 2026
7db70a7
style: fix prettier formatting and eslint errors on test files
frostebite Mar 5, 2026
4f07508
feat(orchestrator): build reliability features — git integrity, reser…
frostebite Mar 5, 2026
47670cf
feat(reliability): implement build reliability service with git integ…
frostebite Mar 5, 2026
ff56194
test(providers): add comprehensive unit tests for GitHub Actions, Git…
frostebite Mar 5, 2026
007852a
feat(cache): add child workspace isolation for multi-product CI build…
frostebite Mar 5, 2026
fe63d7b
fix(cli-provider): add timeout protection for external CLI processes
frostebite Mar 5, 2026
1f3affe
fix(secrets): prevent shell injection in secret key names and mask va…
frostebite Mar 5, 2026
f06f99b
chore: rebuild dist for cli-provider timeout changes
frostebite Mar 5, 2026
cff7597
fix(load-balancing): add pagination limits and rate-limit detection
frostebite Mar 5, 2026
40dd436
fix(reliability): add disk space validation before build archival
frostebite Mar 5, 2026
120c3c5
fix(providers): add polling timeouts, fix credential parsing, validat…
frostebite Mar 5, 2026
4d7e871
chore: rebuild dist for provider timeout and credential fixes
frostebite Mar 5, 2026
b3bd405
fix: prettier formatting for orchestrator-folders-auth test
frostebite Mar 5, 2026
9789eb5
ci: split orchestrator integrity into parallel jobs for faster valida…
frostebite Mar 5, 2026
3976b7c
style: fix prettier formatting
frostebite Mar 5, 2026
e9c247f
style: fix prettier formatting
frostebite Mar 5, 2026
79ae558
style: fix prettier formatting
frostebite Mar 5, 2026
81ba9c3
style: fix prettier formatting
frostebite Mar 5, 2026
b4ffa3e
ci: split orchestrator integrity into 4 parallel jobs to fix timeout
frostebite Mar 5, 2026
02d4ec0
Merge origin/feature/orchestrator-enterprise-support into release/lts…
frostebite Mar 5, 2026
6c548cd
Merge remote-tracking branch 'origin/fix/secure-git-token-usage' into…
frostebite Mar 5, 2026
2ef2275
Merge remote-tracking branch 'origin/feature/orchestrator-unit-tests'…
frostebite Mar 5, 2026
3e15471
Merge remote-tracking branch 'origin/feature/build-reliability' into …
frostebite Mar 5, 2026
7307bea
Merge remote-tracking branch 'origin/feature/provider-load-balancing'…
frostebite Mar 5, 2026
f77a135
Merge remote-tracking branch 'origin/feature/premade-secret-sources' …
frostebite Mar 5, 2026
67fd293
Merge remote-tracking branch 'origin/feature/cloud-run-azure-provider…
frostebite Mar 5, 2026
52a5bc4
Merge remote-tracking branch 'origin/feature/ci-platform-providers' i…
frostebite Mar 5, 2026
18e20aa
Merge remote-tracking branch 'origin/ci/orchestrator-integrity-speedu…
frostebite Mar 5, 2026
f05cfe7
chore: add integration branch update scripts for release/lts-infrastr…
frostebite Mar 5, 2026
c08d13e
Merge remote-tracking branch 'origin/feature/orchestrator-enterprise-…
frostebite Mar 6, 2026
54a6c80
ci: mark failed macOS builds as neutral instead of failure
frostebite Mar 7, 2026
49c3bcf
fix(ci): restore missing k3d cluster step in orchestrator-integrity
frostebite Mar 7, 2026
6f751bf
revert: restore build-tests-mac.yml to match main
frostebite Mar 9, 2026
cdb17b2
fix(test): add gitAuthMode to orchestrator-folders test mock
frostebite Mar 9, 2026
5f63107
fix(ci): bump node version to 20 in integrity-check
frostebite Mar 9, 2026
936ca76
fix: revert Node version to 18 in CI workflow
frostebite Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion .github/workflows/build-tests-mac.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,14 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

permissions:
checks: write

jobs:
buildForAllPlatformsMacOS:
name: ${{ matrix.targetPlatform }} on ${{ matrix.unityVersion }}
runs-on: macos-latest
continue-on-error: true
Copy link

@coderabbitai coderabbitai bot Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n .github/workflows/build-tests-mac.yml | head -100

Repository: game-ci/unity-builder

Length of output: 3751

🌐 Web query:

GitHub Actions matrix fail-fast continue-on-error per entry scoping documentation

💡 Result:

  • jobs.<job_id>.strategy.fail-fast is matrix-wide: when true (default), if any matrix job fails, GitHub cancels the other in-progress or queued matrix jobs. This setting lives under the job’s strategy. [1]
  • jobs.<job_id>.continue-on-error is per job run (per matrix entry): it’s a job-level key (sibling to runs-on, steps, etc.), and it can be an expression that references matrix.*, so each matrix leg can decide whether its own failure should be tolerated. [2] [3]
  • Per-entry scoping pattern: add a boolean (or similar) field in strategy.matrix.include, then read it from jobs.<job_id>.continue-on-error. This is explicitly supported/documented and commonly shown as “experimental” legs. [2] [4]
  • Context availability (“scoping”): the matrix context is available when evaluating jobs.<job_id>.continue-on-error. [5]

Example:

jobs:
  test:
    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.allow_failure }}
    strategy:
      fail-fast: true
      matrix:
        python: [ "3.11", "3.12" ]
        allow_failure: [ false ]
        include:
          - python: "3.13"
            allow_failure: true

Docs to cite/quote in your own notes

  • Matrix “Handling failures” section (shows continue-on-error: ${{ matrix.experimental }} + strategy.fail-fast). [2]
  • Workflow syntax reference entries for jobs.<job_id>.strategy.fail-fast and jobs.<job_id>.continue-on-error. [1]
  • Contexts table confirming matrix is allowed for jobs.<job_id>.continue-on-error. [5]

References:
[1] GitHub Docs — Workflow syntax for GitHub Actions (docs.github.com)
[2] GitHub Docs — Run job variations (matrix) (“Handling failures” section) (docs.github.com)
[3] GitHub Blog changelog — expressions in job continue-on-error with matrix.* (github.blog)
[4] GitHub Docs — matrix example using matrix.experimental to drive continue-on-error (docs.github.com)
[5] GitHub Docs — Contexts reference (shows jobs.<job_id>.continue-on-error supports matrix) (docs.github.com)

Citations:

Scope continue-on-error to opt-in matrix rows, not the entire job.

strategy.fail-fast: false already keeps the rest of the matrix running. Setting continue-on-error: true at the job level makes every failing macOS/iOS build non-blocking, allowing the workflow to pass even when a platform build regresses. GitHub's matrix documentation recommends scoping continue-on-error per matrix entry using an expression like matrix.allowFailure to keep diagnostic collection for experimental versions without weakening CI signal for stable ones.

♻️ Suggested change 
-    continue-on-error: true
+    continue-on-error: ${{ matrix.allowFailure == true }}
     strategy:
       fail-fast: false
       matrix:
         projectPath:
           - test-project
         unityVersion:
           - 2021.3.45f1
           - 2022.3.13f1
           - 2023.2.2f1
         targetPlatform:
           - StandaloneOSX
           - iOS
         include:
           - unityVersion: 6000.0.36f1
             targetPlatform: StandaloneOSX
+            allowFailure: true
           - unityVersion: 6000.0.36f1
             targetPlatform: StandaloneOSX
             buildProfile: 'Assets/Settings/Build Profiles/Sample macOS Build Profile.asset'
+            allowFailure: true
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build-tests-mac.yml at line 15, Move the job-level
continue-on-error: true to a per-matrix-entry expression so only opt-in rows are
non-blocking: remove or set continue-on-error at the job level and add
continue-on-error: ${{ matrix.allowFailure }} (or similar) alongside your matrix
definition; keep strategy.fail-fast: false as-is and ensure the matrix includes
an allowFailure boolean for experimental macOS/iOS rows to preserve diagnostics
while preventing regressions from masking CI failures.

strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -82,6 +86,40 @@ jobs:
###########################
- uses: actions/upload-artifact@v4
with:
name: Build ${{ matrix.targetPlatform }} on MacOS (${{ matrix.unityVersion }})${{ matrix.buildProfile && ' With Build Profile' || '' }}
name:
Build ${{ matrix.targetPlatform }} on MacOS (${{ matrix.unityVersion }})${{ matrix.buildProfile && ' With
Build Profile' || '' }}
path: build
retention-days: 14

markUnstableBuilds:
name: Mark unstable macOS builds
needs: buildForAllPlatformsMacOS
if: always()
runs-on: ubuntu-latest
steps:
- uses: actions/github-script@v7
with:
script: |
const { data: checkRuns } = await github.rest.checks.listForRef({
owner: context.repo.owner,
repo: context.repo.repo,
ref: context.sha,
per_page: 100,
});

for (const run of checkRuns.check_runs) {
if (run.conclusion !== 'failure' || run.app?.slug !== 'github-actions') continue;
if (!run.name.includes('StandaloneOSX') && !run.name.includes('iOS')) continue;

await github.rest.checks.update({
owner: context.repo.owner,
repo: context.repo.repo,
check_run_id: run.id,
conclusion: 'neutral',
output: {
title: `${run.name} (unstable)`,
summary: 'This macOS build is unstable and its failure does not block merging.',
},
Copy link

@coderabbitai coderabbitai bot Mar 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

find . -name "build-tests-ubuntu.yml" -type f

Repository: game-ci/unity-builder

Length of output: 107

🏁 Script executed:

fd -t f "build-tests-ubuntu.yml"

Repository: game-ci/unity-builder

Length of output: 47

🏁 Script executed:

git ls-files | grep -i "build.*ubuntu"

Repository: game-ci/unity-builder

Length of output: 105

🏁 Script executed:

cat -n .github/workflows/build-tests-ubuntu.yml | sed -n '39,62p'

Repository: game-ci/unity-builder

Length of output: 1118

🏁 Script executed:

cat -n .github/workflows/build-tests-mac.yml | sed -n '95,123p'

Repository: game-ci/unity-builder

Length of output: 1376

🌐 Web query:

GitHub API checks.listForRef scope all workflows commit

💡 Result:

octokit.checks.listForRef() calls the REST endpoint “List check runs for a Git reference”:

  • Endpoint: GET /repos/{owner}/{repo}/commits/{ref}/check-runs where ref can be a commit SHA, branch, or tag. It lists check runs associated with that commit (this includes check runs produced by GitHub Actions workflows, plus other GitHub Apps that create checks). [1]
  • All vs latest: by default it uses filter=latest (most recent check runs). To return everything available for that commit, use filter=all. [1]
  • Token scope / permissions:
    • PAT classic / OAuth: needs repo scope for private repos. [1]
    • Fine-grained PAT / GitHub App tokens: needs repository permission “Checks: read”. [1]
    • Public repo: can be called without auth if you’re only requesting public resources. [1]
  • Gotchas: if the commit has >1000 check suites, results are limited to the 1000 most recent suites; to iterate beyond that you must list check suites for the ref and then list runs per suite. Also, checks endpoints only look for pushes in the repo where the suite/run was created (fork pushes won’t be detected). [1]

Sources: GitHub REST API “Check runs” docs (List check runs for a Git reference). [1]

🌐 Web query:

GitHub API actions.listJobsForWorkflowRun permissions required

💡 Result:

For the REST endpoint behind actions.listJobsForWorkflowRun (GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs):

  • Repository access: anyone with read access to the repo can use it. [1]
  • Fine-grained PAT / GitHub App tokens: require Repository permissions → Actions: Read. [1]
  • Classic PAT / OAuth app token (private repos): requires the repo scope. [1]
  • Public repos: can be called without authentication (subject to unauthenticated rate limits). [1]

Sources:
[1] GitHub Docs – “REST API endpoints for workflow jobs” (List jobs for a workflow run)

Scope the check run updates to only this workflow run.

checks.listForRef returns check runs from all workflows for the commit. The current filter—checking if job names include StandaloneOSX or iOS—will match Ubuntu workflow jobs (which use the same matrix values at .github/workflows/build-tests-ubuntu.yml:54-58). This causes unrelated Ubuntu job failures to be neutralized, weakening branch protection for real regressions.

Use the Workflow Jobs API instead, which scopes to the current workflow run via context.runId and directly exposes each job's check_run_url. Add permissions: { actions: read, checks: write } to support the API calls.

Suggested change 
   markUnstableBuilds:
     name: Mark unstable macOS builds
     needs: buildForAllPlatformsMacOS
     if: always()
     runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      checks: write
     steps:
       - uses: actions/github-script@v7
         with:
           script: |
-            const { data: checkRuns } = await github.rest.checks.listForRef({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              ref: context.sha,
-              per_page: 100,
-            });
-
-            for (const run of checkRuns.check_runs) {
-              if (run.conclusion !== 'failure' || run.app?.slug !== 'github-actions') continue;
-              if (!run.name.includes('StandaloneOSX') && !run.name.includes('iOS')) continue;
-
-              await github.rest.checks.update({
+            const jobs = await github.paginate(
+              github.rest.actions.listJobsForWorkflowRun,
+              {
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                run_id: context.runId,
+                per_page: 100,
+              },
+            );
+
+            for (const job of jobs) {
+              if (job.conclusion !== 'failure') continue;
+              if (!job.name.includes('StandaloneOSX') && !job.name.includes('iOS')) continue;
+
+              const checkRunId = Number(job.check_run_url.split('/').pop());
+
+              await github.rest.checks.update({
                 owner: context.repo.owner,
                 repo: context.repo.repo,
-                check_run_id: run.id,
+                check_run_id: checkRunId,
                 conclusion: 'neutral',
                 output: {
-                  title: `${run.name} (unstable)`,
+                  title: `${job.name} (unstable)`,
                   summary: 'This macOS build is unstable and its failure does not block merging.',
                 },
               });
             }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/build-tests-mac.yml around lines 95 - 123, Replace the
global checks.listForRef approach with the Workflow Jobs API scoped to this run:
call github.rest.actions.listJobsForWorkflowRun with run_id: context.runId to
get only jobs for the current workflow run, iterate the returned jobs (not
check_runs), filter by job.name including "StandaloneOSX" or "iOS", extract the
check_run id from each job.check_run_url (parse the final path segment) and call
github.rest.checks.update with that id to set conclusion: 'neutral' and the
unstable output; also add a top-level permissions entry to the workflow YAML:
permissions: { actions: read, checks: write } so the API calls are permitted.

});
}
2 changes: 1 addition & 1 deletion .github/workflows/orchestrator-async-checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
# AWS_STACK_NAME: game-ci-github-pipelines
CHECKS_UPDATE: ${{ github.event.inputs.checksObject }}
run: |
git clone -b orchestrator-develop https://github.com/game-ci/unity-builder
git clone -b main https://github.com/game-ci/unity-builder
cd unity-builder
yarn
ls
Expand Down
Loading
Loading