feat: Test Workflow Engine — suite definitions, taxonomy filters, structured results

[frostebite]

Summary

Add a test workflow engine to Orchestrator that supports sophisticated test suite definitions, taxonomy-based filtering, and structured result reporting. Modeled on battle-tested patterns from production Unity projects.

Motivation

Current Orchestrator test support is limited to running a custom buildMethod that happens to execute tests. There's no built-in concept of:

• Test suite profiles (different suites for PR vs push vs release)
• Taxonomy-based test filtering (scope, maturity, feedback speed, determinism, etc.)
• Structured test result collection and reporting
• Sequential suite execution with dependencies
• Running tests against built clients (not just editor tests)

Production Unity projects already solve this with YAML-based test suite configs that define exactly which tests run for each CI event, filtered by multi-dimensional taxonomy metadata.

Proposed Features

YAML Test Suite Definitions

• Define test suites as YAML configs (e.g., pull-request-suite.yml, release-suite.yml)
• Each suite contains ordered runs with taxonomy filters
• Runs can depend on previous runs (needs field) for sequential execution
• Suite selection by CI event (PR, push, release, scheduled, manual dispatch)

Taxonomy Filter System

• Multi-dimensional test taxonomy: Scope, Maturity, FeedbackSpeed, Execution, Rigor, Determinism, IsolationLevel, etc.
• Filter syntax: CSV values, regex patterns, hierarchical dot-notation
• Closed groups (fixed values) vs extensible groups (project-defined values)
• Selectors for domain and case-tag filtering

Test Execution Modes

• EditMode tests — Standard Unity Test Framework in edit mode
• PlayMode tests — Unity Test Framework in play mode
• Built-client tests — Run tests against a previously built game client
• Custom test runners — Extensible for project-specific test frameworks

Structured Results

• Machine-readable test results (JUnit XML, JSON)
• Per-suite and per-run result aggregation
• Integration with GitHub Checks for inline test failure reporting
• Test timing and performance tracking

Cache-Aware Test Execution

• Cache test results by suite + taxonomy hash
• Skip unchanged test suites on incremental builds
• Separate cache profiles for test artifacts vs build artifacts

Example Suite Definition

name: pull-request
description: Fast feedback for pull requests
runs:
  - name: fast
    editMode: true
    filters:
      Maturity: Trusted
      FeedbackSpeed: Fast,Moderate
      Scope: Unit,Integration
    timeout: 300

  - name: basic
    needs: [fast]
    editMode: true
    playMode: true
    filters:
      Maturity: Trusted,Adolescent
      Scope: Unit,Integration,System
    timeout: 600

  - name: extended
    needs: [basic]
    playMode: true
    filters:
      Rigor: Strict
      Scope: End To End
    timeout: 1200

New Inputs

Input	Description
testSuitePath	Path to YAML test suite definition file
testSuiteEvent	CI event name for suite selection (pr, push, release, etc.)
testTaxonomyPath	Path to taxonomy definition YAML
testResultFormat	Output format for results (junit, json, both)
testResultPath	Where to write structured results

Implementation Notes

• New service: TestWorkflowService in src/model/orchestrator/services/test/
• YAML parsing with existing js-yaml dependency
• Test suite configs can live in repo (.game-ci/test-suites/) or be passed as inputs
• Should work with all providers (AWS, K8s, Docker, Local, CLI)
• Built-client testing requires a prior build step — can reference build artifacts from cache

Related

• PR feat(orchestrator): QoL for large projects — CLI providers, caching, LFS, hooks #777 (orchestrator enterprise support)
• GameClient test taxonomy system (reference implementation)
• feat: Hot Runner Protocol — persistent process-based build/test providers #789 (Hot Runner Protocol)
• feat: Generic Artifact System — multi-type output management, manifests, and processing pipelines #792 (Generic Artifact System)

---

Implementation

PR	Description	Status
#790	feat(orchestrator): test workflow engine	Draft — 43 tests
game-ci/documentation#538	docs: test workflow engine page	Draft

Tracking

• Next-Gen Features — #795
• LTS (v2.0.0) — #794
• Orchestrator: The Endgame — #810

[frostebite]

Implementing PR: #790 (feature/test-workflow-engine)

[frostebite]

Implementation Complete

PR #790 is fully implemented with:

• TestSuiteParser — Kahn's topological sort for run dependency ordering
• TaxonomyFilterService — 7 built-in dimensions (Scope, Maturity, FeedbackSpeed, Rigor, Determinism, Execution, IsolationLevel) with regex and hierarchical matching
• TestResultReporter — JUnit XML + Unity JSON parsing, Markdown summary generation
• TestWorkflowService — Full orchestration of suite loading, filtering, execution, and result output
• 43 unit tests covering parsing, filtering, result reporting, and orchestration
• 5 action.yml inputs: testSuitePath, testSuiteEvent, testTaxonomyPath, testResultFormat, testResultPath
• Documentation: docs(orchestrator): next-gen features and advanced topics documentation#538 (11-test-workflow-engine.mdx)

[frostebite]

Closing — orchestrator code has been extracted to the standalone game-ci/orchestrator repository. See PR #819.