Skip to content

feat: Build Reliability Features — git integrity, reserved filename cleanup, and build output archival #807

@frostebite

Description

@frostebite

Summary

Three optional reliability features that harden the build pipeline for enterprise and self-hosted runner environments. These are inspired by battle-tested patterns from production Unity CI pipelines running on self-hosted runners with massive monorepos.

All three features are opt-in — they do nothing unless explicitly enabled, and they never block the build on failure (warnings only).


Features

1. Git Corruption Detection & Recovery

Self-hosted runners accumulate git state across builds. Corrupted .git directories, stale lock files, and broken submodule backing stores cause mysterious build failures that waste developer time.

What it does:

  • Pre-checkout integrity check — runs git fsck --no-dangling to detect object corruption before the build starts
  • Stale lock file cleanup — removes index.lock, shallow.lock, config.lock, and other lock files left by crashed git processes
  • Submodule .git file validation — verifies that submodule .git files point to existing backing stores in .git/modules/
  • System git config corruption bypass — sets GIT_CONFIG_NOSYSTEM=1 when system-level config is corrupted
  • Automatic recovery — if corruption is detected, removes the corrupted .git directory and triggers a fresh clone

Inputs:

  • gitIntegrityCheck — Enable pre-checkout git integrity verification (default: false)
  • gitAutoRecover — Automatically recover from detected corruption (default: true when integrity check enabled)

2. Reserved Filename Cleanup

Windows has reserved device names (nul, con, prn, aux, com1com9, lpt1lpt9) that cannot be used as filenames. When these appear in Unity project assets (often from cross-platform development or bad imports), they cause the Unity asset importer to enter infinite loops, hanging the build indefinitely.

What it does:

  • Scans the project's Assets/ directory for files or directories matching Windows reserved names
  • Removes or renames them before Unity opens the project
  • Logs warnings about each removed file so the source can be investigated

Inputs:

  • cleanReservedFilenames — Enable reserved filename cleanup before build (default: false)

3. Build Output Archival

For self-hosted runners with persistent storage, manage build output lifecycle with retention policies. Prevents disk space exhaustion from accumulated builds while keeping recent outputs accessible.

What it does:

  • Moves completed build outputs to a configurable archive location after the build
  • Maintains a retention count per location (e.g., keep 3 most recent builds)
  • Supports tiered storage (fast local → bulk archive)
  • Cleans up old archives beyond the retention limit

Inputs:

  • buildArchiveEnabled — Enable post-build output archival (default: false)
  • buildArchivePath — Path to archive location (default: ../builds-archive)
  • buildArchiveRetention — Number of builds to retain per platform (default: 3)

Implementation

New Service: BuildReliabilityService

Location: src/model/orchestrator/services/reliability/build-reliability-service.ts

export class BuildReliabilityService {
  // Git integrity
  static async checkGitIntegrity(repoPath: string): Promise<boolean>;
  static async cleanStaleLockFiles(repoPath: string): Promise<number>;
  static async validateSubmoduleBackingStores(repoPath: string): Promise<string[]>;
  static async recoverCorruptedRepo(repoPath: string): Promise<void>;

  // Reserved filenames
  static async cleanReservedFilenames(projectPath: string): Promise<string[]>;

  // Build archival
  static async archiveBuildOutput(outputPath: string, archivePath: string, retention: number): Promise<void>;
  static async enforceRetention(archivePath: string, retention: number): Promise<number>;
}

Integration Points

  • Git integrity — runs in src/index.ts before checkout/submodule init, and in remote-client/index.ts before bootstrapRepository()
  • Reserved filenames — runs after checkout, before Unity build starts
  • Build archival — runs after build completes, in the cleanup phase

Testing

Unit tests at src/model/orchestrator/services/reliability/build-reliability-service.test.ts covering:

  • Lock file detection and cleanup (mock filesystem)
  • Reserved filename pattern matching
  • Retention policy enforcement
  • Recovery flow (mock git commands)

Related

PR Description Status
#808 feat(orchestrator): build reliability features Draft

Metadata

Metadata

Assignees

No one assigned

    Labels

    LTS 2.0Orchestrator LTS v2.0 milestonebuild-reliabilityBuild reliability and infrastructure hardeningenhancementNew feature or requestorchestratorOrchestrator module

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions