Orchestrator Extraction — Remove Orchestrator Code, Add Plugin Loader#819
Open
frostebite wants to merge 104 commits intomainfrom
Open
Orchestrator Extraction — Remove Orchestrator Code, Add Plugin Loader#819frostebite wants to merge 104 commits intomainfrom
frostebite wants to merge 104 commits intomainfrom
Conversation
…ule profiles, caching, LFS, hooks Add generic enterprise-grade features to the orchestrator, enabling Unity projects with complex CI/CD pipelines to adopt game-ci/unity-builder with built-in support for: - CLI provider protocol: JSON-over-stdin/stdout bridge enabling providers in any language (Go, Python, Rust, shell) via the `providerExecutable` input - Submodule profiles: YAML-based selective submodule initialization with glob patterns and variant overlays (`submoduleProfilePath`, `submoduleVariantPath`) - Local build caching: Filesystem-based Library and LFS caching for local builds without external cache actions (`localCacheEnabled`, `localCacheRoot`) - Custom LFS transfer agents: Register external transfer agents like elastic-git-storage (`lfsTransferAgent`, `lfsTransferAgentArgs`, `lfsStoragePaths`) - Git hooks support: Detect and install lefthook/husky with configurable skip lists (`gitHooksEnabled`, `gitHooksSkipList`) Also removes all `orchestrator-develop` branch references, replacing with `main`. 13 new action inputs, 13 new files, 14 new CLI provider tests, 17 submodule tests, plus cache/LFS/hooks unit tests. All 452 tests pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iders Add two new cloud provider implementations for the orchestrator, both marked as experimental: - **GCP Cloud Run Jobs** (`providerStrategy: gcp-cloud-run`): Executes Unity builds as Cloud Run Jobs with GCS FUSE for large artifact storage. Supports configurable machine types, service accounts, and VPC connectors. 7 new inputs (gcpProject, gcpRegion, gcpBucket, gcpMachineType, gcpDiskSizeGb, gcpServiceAccount, gcpVpcConnector). - **Azure Container Instances** (`providerStrategy: azure-aci`): Executes Unity builds as ACI containers with Azure File Shares (Premium FileStorage) for large artifact storage up to 100 TiB. Supports configurable CPU/memory, VNet integration, and subscription targeting. 9 new inputs (azureResourceGroup, azureLocation, azureStorageAccount, azureFileShareName, azureSubscriptionId, azureCpu, azureMemoryGb, azureDiskSizeGb, azureSubnetId). Both providers use their respective CLIs (gcloud, az) for infrastructure management and support garbage collection of old build resources. No tests included as these require real cloud infrastructure to validate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both providers now support four storage backends via gcpStorageType / azureStorageType: GCP Cloud Run: - gcs-fuse: Mount GCS bucket as POSIX filesystem (unlimited, best for large sequential I/O) - gcs-copy: Copy artifacts in/out via gsutil (simpler, no FUSE overhead) - nfs: Filestore NFS mount (true POSIX, good random I/O, up to 100 TiB) - in-memory: tmpfs (fastest, volatile, up to 32 GiB) Azure ACI: - azure-files: SMB file share mount (up to 100 TiB, premium throughput) - blob-copy: Copy artifacts in/out via az storage blob (no mount overhead) - azure-files-nfs: NFS 4.1 file share mount (true POSIX, no SMB lock overhead) - in-memory: emptyDir tmpfs (fastest, volatile, limited by container memory) New inputs: gcpStorageType, gcpFilestoreIp, gcpFilestoreShare, azureStorageType, azureBlobContainer. Constructor validates storage config and warns on missing prerequisites (e.g. NFS requires VPC connector/subnet). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ity check Adds built-in load balancing: check GitHub runner availability before builds start, auto-route to a fallback provider when runners are busy or offline. Eliminates the need for a separate check-runner job. New inputs: fallbackProviderStrategy, runnerCheckEnabled, runnerCheckLabels, runnerCheckMinAvailable. Outputs providerFallbackUsed and providerFallbackReason for workflow visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds retryOnFallback (retry failed builds on alternate provider) and providerInitTimeout (swap provider if init takes too long). Refactors run() into run()/runWithProvider() to support retry loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds tests for cache hit restore (picks latest tar), LFS cache restore/save, garbage collection age filtering, and edge cases like permission errors and empty directories. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers: no token skip, no runners fallback, busy/offline runners, label filtering (case-insensitive), minAvailable threshold, fail-open on API error, mixed runner states. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 64 new mock-based unit tests covering orchestrator services that previously had zero test coverage: - TaskParameterSerializer: env var format conversion, round-trip, uniqBy deduplication, blocked params, default secrets - FollowLogStreamService: build output message parsing — end of transmission, build success/failure detection, error accumulation, Library rebuild detection - OrchestratorNamespace (guid): GUID generation format, platform name normalization, nanoid uniqueness - OrchestratorFolders: path computation for all folder getters, ToLinuxFolder conversion, repo URL generation, purge flag detection All tests are pure mock-based and run without any external infrastructure (no LocalStack, K8s, Docker, or AWS). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a fast-fail unit test step at the top of orchestrator-integrity, right after yarn install and before any infrastructure setup (k3d, LocalStack). Runs 113 mock-based orchestrator tests in ~5 seconds. If serialization, path computation, log parsing, or provider loading is broken, the workflow fails immediately instead of spending 30+ minutes setting up LocalStack and k3d clusters. Tests included: orchestrator-guid, orchestrator-folders, task-parameter-serializer, follow-log-stream-service, runner-availability-service, provider-url-parser, provider-loader, provider-git-manager, orchestrator-image, orchestrator-hooks, orchestrator-github-checks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add comprehensive tests for CLI provider (cleanupWorkflow, garbageCollect, listWorkflow, watchWorkflow, stderr forwarding, timeout handling), local cache service (saveLfsCache full path and error handling), git hooks service (husky install, failure logging, edge cases), and LFS agent service (empty storagePaths, validate logging). 73 tests across 4 test files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace token-in-URL pattern with http.extraHeader for git clone and LFS operations. The token no longer appears in clone URLs, git remote config, or process command lines. Add gitAuthMode input (default: 'header', legacy: 'url') so users can fall back to the old behavior if needed. Closes #785 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SecretSourceService with premade secret source integrations:
- aws-secrets-manager (with --query SecretString for direct value)
- aws-parameter-store (with --with-decryption)
- gcp-secret-manager (latest version)
- azure-key-vault (via $AZURE_VAULT_NAME env var)
- env (environment variables, no shell command needed)
- Custom commands (any string with {0} placeholder)
- YAML file definitions for custom sources
Add secretSource input that takes precedence over inputPullCommand.
Backward compatible — existing inputPullCommand behavior unchanged.
Closes #776
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds three Vault entries: hashicorp-vault (KV v2), hashicorp-vault-kv1 (KV v1), and vault (short alias). Uses VAULT_ADDR for server address and VAULT_MOUNT env var for configurable mount path (defaults to 'secret'). Refs #776 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
First-class support for elastic-git-storage as a custom LFS transfer agent. When lfsTransferAgent is set to "elastic-git-storage" (or "elastic-git-storage@v1.0.0" for a specific version), the service automatically finds or installs the agent from GitHub releases, then configures it via git config. Supports version pinning via @Version suffix in the agent value, eliminating the need for a separate version parameter. Platform and architecture detection handles linux/darwin/windows on amd64/arm64. 37 unit tests covering detection, PATH lookup, installation, version parsing, and configuration delegation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Built-in support for Unity Git Hooks (com.frostebite.unitygithooks): - Auto-detect UPM package in Packages/manifest.json - Run init-unity-lefthook.js before hook installation - Set CI-friendly env vars (disable background project mode) New gitHooksRunBeforeBuild input runs specific lefthook groups before the Unity build, allowing CI to trigger pre-commit or pre-push checks that normally only fire on git events. 35 unit tests covering detection, init, CI env, group execution, and failure handling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Initial scaffold for the test workflow engine service directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Initial scaffold for the runner registration and hot editor provider module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, and collection service Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ut, and storage-backed sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add scheduled workflow that validates community Unity packages compile and build correctly using unity-builder. Runs weekly on Sunday. Includes: - YAML plugin registry (community-plugins.yml) for package listings - Matrix expansion across plugins and platforms - Automatic failure reporting via GitHub issues - Manual trigger with plugin filter and Unity version override Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… Actions, GitLab CI, Ansible Add four new providers that delegate builds to external CI platforms: - remote-powershell: Execute on remote machines via WinRM/SSH - github-actions: Dispatch workflow_dispatch on target repository - gitlab-ci: Trigger pipeline via GitLab API - ansible: Run playbooks against managed inventory Each follows the CI-as-a-provider pattern: trigger remote job, pass build parameters, stream logs, report status. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ved filename cleanup, archival Add three optional reliability features for hardening CI pipelines: - Git corruption detection & recovery (fsck, stale lock cleanup, submodule backing store validation, auto-recovery) - Reserved filename cleanup (removes Windows device names that cause Unity asset importer infinite loops) - Build output archival with configurable retention policy All features are opt-in and fail gracefully with warnings only. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rity, reserved filename cleanup, and build archival Adds BuildReliabilityService with the following capabilities: - checkGitIntegrity(): runs git fsck --no-dangling and parses output for corruption - cleanStaleLockFiles(): removes stale .lock files older than 10 minutes - validateSubmoduleBackingStores(): validates .git files point to valid backing stores - recoverCorruptedRepo(): orchestrates fsck, lock cleanup, re-fetch, retry fsck - cleanReservedFilenames(): removes Windows reserved filenames (con, prn, aux, nul, com1-9, lpt1-9) - archiveBuildOutput(): creates tar.gz archive of build output - enforceRetention(): deletes archives older than retention period - configureGitEnvironment(): sets GIT_TERMINAL_PROMPT=0, http.postBuffer, core.longpaths Wired into action.yml as opt-in inputs, with pre-build integrity checks and post-build archival in the main entry point. Includes 29 unit tests covering success and failure cases for all methods. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Lab CI, PowerShell, and Ansible providers (#806) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… monitoring, and job dispatch (#791) Adds persistent Unity editor instance support to reduce build iteration time by eliminating cold-start overhead. Includes: - HotRunnerTypes: interfaces for config, status, job request/result, transport - HotRunnerRegistry: in-memory runner management with file-based persistence - HotRunnerHealthMonitor: periodic health checks, idle recycling, job-count recycling - HotRunnerDispatcher: job routing with wait-for-runner, timeout, and output streaming - HotRunnerService: high-level API integrating registry, health, and dispatch - 34 unit tests covering registration, filtering, health, dispatch, timeout, fallback - action.yml inputs for hot runner configuration (7 new inputs) - Input/BuildParameters integration for hot runner settings - index.ts wiring with cold-build fallback when hot runner unavailable Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…s, tests, and action integration (#798) - Add ArtifactUploadHandler with support for github-artifacts, storage (rclone), and local copy upload targets, including large file chunking for GitHub Artifacts - Add 44 unit tests covering OutputTypeRegistry, OutputService, and ArtifactUploadHandler (config parsing, upload coordination, file collection) - Add 6 new action.yml inputs for artifact configuration - Add artifactManifestPath action output - Wire artifact collection and upload into index.ts post-build flow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…omy filtering, and structured results (#790) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te management, and tests (#799) - Add storage-pull strategy: rclone-based sync from remote storage with overlay and clean modes, URI parsing (storage://remote:bucket/path), transfer parallelism, and automatic rclone availability checking - Add SyncStateManager: persistent state load/save with configurable paths, workspace hash calculation via SHA-256 of key project files, and drift detection for external modification awareness - Add action.yml inputs: syncStrategy, syncInputRef, syncStorageRemote, syncRevertAfter, syncStatePath with sensible defaults - Wire sync into Input (5 getters), BuildParameters (5 fields), index.ts (local build path), and RemoteClient (orchestrator path) with post-job overlay revert when syncRevertAfter is true - Add 42 unit tests covering all strategies, URI parsing, state management, hash calculation, drift detection, error handling, and edge cases (missing rclone, invalid URIs, absent state, empty diffs) - Add root:true to eslintrc to prevent plugin resolution conflicts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The orchestrator tests need compiled output (dist/index.js) to exist before running integration tests that spawn containers/k8s jobs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…estrator workflows The refactor/orchestrator-extraction branch was not matching the feature/** pattern, preventing the integration workflow from running after fix commits were pushed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
frostebite
added a commit
to game-ci/documentation
that referenced
this pull request
Mar 10, 2026
Restores dedicated example pages for AWS and Kubernetes that were removed during the docs restructure. These complement the provider reference pages with copy-paste workflow examples. Related: game-ci/unity-builder#819, #541 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
frostebite
added a commit
to game-ci/orchestrator
that referenced
this pull request
Mar 10, 2026
- Rewrite README with architecture diagram, feature list, provider table, project structure, quick start guides, and cross-links to documentation PR #541 and extraction PR #819 - Add CLAUDE.md with project conventions and architecture guide - Add .claude/agents.md to direct agents to read CLAUDE.md Related: game-ci/unity-builder#819, game-ci/documentation#541 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
Author
|
Cross-references:
|
3 tasks
…nightly exhaustive suite
validate-orchestrator.yml (per-PR, ~5 min):
- Plugin architecture health: compilation, unit tests, plugin loader
graceful degradation, installed service validation, type declaration checks
validate-orchestrator-integration.yml (daily 3 AM UTC cron, ~1-2h):
- 5 parallel jobs mirroring orchestrator-integrity.yml:
plugin-interface, k8s (5 tests), aws (10 tests),
local-docker (9 tests), rclone (1 test)
- Full LocalStack + k3d integration coverage
- continue-on-error on known flaky end2end tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ensure orchestrator validation runs when yarn.lock changes, since dependency updates can affect plugin compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Install scripts now live at game-ci/orchestrator where the CLI releases are published. Removed from unity-builder to avoid duplication. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…in permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This was referenced Mar 10, 2026
The orchestrator is a plugin, not an enterprise feature. Renamed loadEnterpriseServices -> loadPluginServices and all related variables, types, log messages, and test descriptions to use "plugin" terminology. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…adPluginServices CI workflows still referenced the old function name after the rename. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously both loadOrchestrator() and loadPluginServices() caught all errors, masking real failures like syntax errors or missing transitive dependencies. Now only MODULE_NOT_FOUND / ERR_MODULE_NOT_FOUND errors are suppressed; all other exceptions are rethrown. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verifies end-to-end that loadOrchestrator().run() is correctly wired to Orchestrator.run(), BuildParameters.create() produces valid config, and plugin services resolve to real implementations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
|
Do we do a CLI as part of this? |
- Add workflow_call trigger to validate-orchestrator-integration.yml so other workflows can invoke the exhaustive test suite - Add orchestrator-integration job to integrity-check.yml that runs on pushes to main (skipped on PRs to avoid 1-2h CI time) - Daily cron + manual dispatch remain as fallback triggers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
localstack:latest (v4.14+) returns JSON responses for some S3 operations, but @aws-sdk/client-s3 v3.779+ uses AwsRestXmlProtocol which expects XML. This breaks all SharedWorkspaceLocking tests (locking, e2e caching, retaining). Pin to v3.8.1 (last v3 release) where the S3 provider returns proper XML responses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3 deserialization issue was caused by @aws-sdk/client-s3 v3.1005 (schema-based AwsRestXmlProtocol), not LocalStack's version. The SDK is now pinned to ~3.779.0 in the orchestrator repo, so localstack:latest works correctly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move mandatory tests (caching, locking-core, locking-get-locked) before continue-on-error e2e tests. The e2e tests can corrupt the workspace (delete package.json), which was causing subsequent mandatory tests to fail with "Couldn't find a package.json". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
Author
There is a CLI provided by orchestrator now it's standalone. It's a pretty good entrypoint for a CLI generally for Game-CI. This change specifically doesn't add it or require it, because this just implements orchestrator as a standalone plugin. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Orchestrator Extraction — Remove Orchestrator Code, Add Plugin Loader
Extracts the orchestrator subsystem from unity-builder into the standalone
game-ci/orchestratorrepository. Unity-builder becomes a lean GitHub Action; all orchestrator, CLI, and cloud provider features live in the standalone repo.Why
src/model/orchestrator/BuildParametersis a god object mixing both concernsWhat Changes for Users
The orchestrator is now a separate package that lives in its own repository (
game-ci/orchestrator). What this means depends on your build strategy:Local builds (
providerStrategy: local) — No change requiredIf you use the default local Docker builds, nothing changes. Unity-builder works standalone with no extra install step:
Cloud builds (
providerStrategy: aws,k8s, etc.) — Install the orchestrator firstFor cloud provider strategies, you now need to install the orchestrator before running unity-builder. The install adds the
game-ciCLI and the@game-ci/orchestratorpackage so that unity-builder's plugin loader can detect it automatically.GitHub Actions example:
That is all that changes: one extra step to install the orchestrator before the build step. Unity-builder automatically detects the orchestrator via its plugin loader and enables cloud providers and all services.
Install commands by platform
curl -fsSL https://raw.githubusercontent.com/game-ci/orchestrator/main/install.sh | shirm https://raw.githubusercontent.com/game-ci/orchestrator/main/install.ps1 | iexStandalone CLI usage (optional)
The orchestrator can also be used directly from the command line without GitHub Actions:
What This PR Does
118 files changed, 3,741 insertions, 15,404 deletions
Deleted
src/model/orchestrator/directory (148.tsfiles)src/cli.tsandsrc/cli/(CLI moved to orchestrator repo)src/integration/orchestrator-github-checks-integration-test.tssrc/test-utils/orchestrator-test-helpers.ts.github/workflows/orchestrator-async-checks.yml.github/workflows/orchestrator-integrity.yml.github/workflows/release-cli.yml@aws-sdk/*(5),@kubernetes/client-node,async-wait-until,aws-sdk,base-64,kubernetes-client,shell-quote,uuidyargs,@types/yargs,pkginstall.sh,install.ps1) — moved to orchestrator repoAdded
src/model/orchestrator-plugin.ts): Dynamicimport('@game-ci/orchestrator')with graceful degradationsrc/types/game-ci-orchestrator.d.ts): Ambient module for optional@game-ci/orchestratorpackagesrc/model/orchestrator-plugin.test.ts): 15 tests covering both installed and not-installed paths.github/workflows/validate-orchestrator.yml): Per-PR health checks — builds both repos, tests plugin loader, verifies type declarations.github/workflows/validate-orchestrator-integration.yml): Nightly exhaustive suite — k8s (k3d), AWS (LocalStack), local-docker, rcloneChanged
build-parameters.ts: Replaced 44OrchestratorOptions.*withInput.getInput(), inlined constantsgithub.ts: Stripped to minimal classinput.ts: RemovedOrchestratorQueryOverrideinput-readers/: ReplacedOrchestratorSystem.Runwithchild_process.execindex.ts: Plugin loader pattern for servicesHow the Plugin Loader Works
Test Results
dist/bundle rebuilt (~4MB vs ~15MB before)Orchestrator Repo Status
Checklist