-
-
Notifications
You must be signed in to change notification settings - Fork 307
Description
Summary
Three optional reliability features that harden the build pipeline for enterprise and self-hosted runner environments. These are inspired by battle-tested patterns from production Unity CI pipelines running on self-hosted runners with massive monorepos.
All three features are opt-in — they do nothing unless explicitly enabled, and they never block the build on failure (warnings only).
Features
1. Git Corruption Detection & Recovery
Self-hosted runners accumulate git state across builds. Corrupted .git directories, stale lock files, and broken submodule backing stores cause mysterious build failures that waste developer time.
What it does:
- Pre-checkout integrity check — runs
git fsck --no-danglingto detect object corruption before the build starts - Stale lock file cleanup — removes
index.lock,shallow.lock,config.lock, and other lock files left by crashed git processes - Submodule
.gitfile validation — verifies that submodule.gitfiles point to existing backing stores in.git/modules/ - System git config corruption bypass — sets
GIT_CONFIG_NOSYSTEM=1when system-level config is corrupted - Automatic recovery — if corruption is detected, removes the corrupted
.gitdirectory and triggers a fresh clone
Inputs:
gitIntegrityCheck— Enable pre-checkout git integrity verification (default: false)gitAutoRecover— Automatically recover from detected corruption (default: true when integrity check enabled)
2. Reserved Filename Cleanup
Windows has reserved device names (nul, con, prn, aux, com1–com9, lpt1–lpt9) that cannot be used as filenames. When these appear in Unity project assets (often from cross-platform development or bad imports), they cause the Unity asset importer to enter infinite loops, hanging the build indefinitely.
What it does:
- Scans the project's
Assets/directory for files or directories matching Windows reserved names - Removes or renames them before Unity opens the project
- Logs warnings about each removed file so the source can be investigated
Inputs:
cleanReservedFilenames— Enable reserved filename cleanup before build (default: false)
3. Build Output Archival
For self-hosted runners with persistent storage, manage build output lifecycle with retention policies. Prevents disk space exhaustion from accumulated builds while keeping recent outputs accessible.
What it does:
- Moves completed build outputs to a configurable archive location after the build
- Maintains a retention count per location (e.g., keep 3 most recent builds)
- Supports tiered storage (fast local → bulk archive)
- Cleans up old archives beyond the retention limit
Inputs:
buildArchiveEnabled— Enable post-build output archival (default: false)buildArchivePath— Path to archive location (default:../builds-archive)buildArchiveRetention— Number of builds to retain per platform (default: 3)
Implementation
New Service: BuildReliabilityService
Location: src/model/orchestrator/services/reliability/build-reliability-service.ts
export class BuildReliabilityService {
// Git integrity
static async checkGitIntegrity(repoPath: string): Promise<boolean>;
static async cleanStaleLockFiles(repoPath: string): Promise<number>;
static async validateSubmoduleBackingStores(repoPath: string): Promise<string[]>;
static async recoverCorruptedRepo(repoPath: string): Promise<void>;
// Reserved filenames
static async cleanReservedFilenames(projectPath: string): Promise<string[]>;
// Build archival
static async archiveBuildOutput(outputPath: string, archivePath: string, retention: number): Promise<void>;
static async enforceRetention(archivePath: string, retention: number): Promise<number>;
}Integration Points
- Git integrity — runs in
src/index.tsbefore checkout/submodule init, and inremote-client/index.tsbeforebootstrapRepository() - Reserved filenames — runs after checkout, before Unity build starts
- Build archival — runs after build completes, in the cleanup phase
Testing
Unit tests at src/model/orchestrator/services/reliability/build-reliability-service.test.ts covering:
- Lock file detection and cleanup (mock filesystem)
- Reserved filename pattern matching
- Retention policy enforcement
- Recovery flow (mock git commands)
Related
- tracking: Orchestrator LTS (v2.0.0) — Long-Term Support orchestrator release #794 (Orchestrator LTS) — Part of the LTS feature set
- feat(orchestrator): QoL for large projects — CLI providers, caching, LFS, hooks #777 (Enterprise features) — Complements the existing enterprise support
- feat: Automation & CI Dispatch Providers — Remote PowerShell, GitHub Actions, GitLab CI, and Ansible #805 (Automation providers) — Reliability features are especially important for self-hosted infrastructure
| PR | Description | Status |
|---|---|---|
| #808 | feat(orchestrator): build reliability features | Draft |