CI Failure Analysis #1192
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # yamllint disable rule:line-length | |
| name: CI Failure Analysis | |
| # Automatically analyze failed CI runs and provide actionable fix suggestions | |
| # Triggers when any lint workflow fails (CI, Markdown, YAML, Validate Workflows) | |
| # | |
| # Testing: Intentionally fail typecheck (e.g., add a type error to any .ts file) | |
| on: | |
| workflow_run: | |
| workflows: | |
| - "CI" | |
| - "Markdown Lint" | |
| - "YAML Lint" | |
| - "Validate Workflows" | |
| # Note: "Check Links" excluded - it creates its own issues on failure | |
| types: [completed] | |
| # Cancel in-progress analysis for same workflow run | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.event.workflow_run.id }} | |
| cancel-in-progress: true | |
| jobs: | |
| analyze-failure: | |
| name: Analyze CI Failure | |
| # Only run on failure, skip bot triggers and fork PRs | |
| if: | | |
| github.event.workflow_run.conclusion == 'failure' && | |
| github.event.workflow_run.actor.login != 'dependabot[bot]' && | |
| github.event.workflow_run.actor.login != 'claude[bot]' && | |
| github.event.workflow_run.head_repository.full_name == github.repository | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 10 | |
| permissions: | |
| contents: read | |
| pull-requests: write | |
| issues: write | |
| actions: read | |
| id-token: write | |
| steps: | |
| # NOTE: No checkout step - analyzing untrusted code from forks is a security risk | |
| # The workflow logs contain all information needed for analysis | |
| - name: Analyze failure with Claude | |
| uses: anthropics/claude-code-action@b47fd721da662d48c5680e154ad16a73ed74d2e0 # v1.0.93 | |
| with: | |
| claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} | |
| additional_permissions: | | |
| actions: read | |
| prompt: | | |
| A CI workflow has failed. Analyze the failure and post an actionable fix comment on the PR. | |
| ## Failed Workflow Details | |
| - **Workflow**: ${{ github.event.workflow_run.name }} | |
| - **Run ID**: ${{ github.event.workflow_run.id }} | |
| - **Run URL**: ${{ github.event.workflow_run.html_url }} | |
| - **Commit SHA**: ${{ github.event.workflow_run.head_sha }} | |
| - **Branch**: ${{ github.event.workflow_run.head_branch }} | |
| - **Repository**: ${{ github.repository }} | |
| ## Instructions | |
| ### Step 1: Get the failure logs | |
| Use the MCP CI tools to get structured failure data: | |
| 1. `mcp__github_ci__get_workflow_run_details` - Get job-level status for run ${{ github.event.workflow_run.id }} | |
| 2. `mcp__github_ci__download_job_log` - Download logs for the failed job | |
| If MCP tools fail, fall back to: | |
| ```bash | |
| gh run view ${{ github.event.workflow_run.id }} --json jobs \ | |
| --jq '.jobs[] | select(.conclusion == "failure") | {name, conclusion}' | |
| ``` | |
| ### Step 2: Identify the error type | |
| Based on which workflow failed, look for specific error patterns: | |
| | Workflow | Tool | Error Pattern | | |
| |----------|------|---------------| | |
| | CI - Type Check | tsc | `TS\d{4}:` errors with file:line | | |
| | CI - Lint | ESLint | `error .+ @typescript-eslint/` | | |
| | CI - Format | Prettier | `Forgot to run Prettier?` or diff output | | |
| | CI - Test | Vitest | `FAIL`, `AssertionError`, timeout | | |
| | CI - Build | tsc | `error TS` during compilation | | |
| | Markdown Lint | markdownlint | `filename:line:col MD\d{3}/rule-name` | | |
| | YAML Lint | yamllint | `[error] message (rule-name)` | | |
| | Validate Workflows | actionlint | Workflow YAML syntax errors | | |
| ### Step 3: Find the associated PR | |
| ```bash | |
| gh pr list \ | |
| --search "${{ github.event.workflow_run.head_sha }}" \ | |
| --state open \ | |
| --json number,title,url \ | |
| --jq '.[0]' | |
| ``` | |
| If no PR is found (e.g., push to main), summarize findings but don't attempt to comment. | |
| ### Step 4: Check for existing analysis comment | |
| Before posting, check if a CI analysis comment already exists for this run: | |
| ```bash | |
| gh pr view {pr_number} --json comments \ | |
| --jq '.comments[].body | select(contains("CI Failure Analysis") and contains("#${{ github.event.workflow_run.id }}"))' | |
| ``` | |
| If a comment for this run ID exists, DO NOT post a duplicate. | |
| ### Step 5: Post analysis comment | |
| If a PR was found and no duplicate exists, post a comment using: | |
| ```bash | |
| gh pr comment {pr_number} --body "..." | |
| ``` | |
| Use this format: | |
| ```markdown | |
| ## CI Failure Analysis: {Workflow Name} | |
| **Run**: [#{run_id}]({run_url}) | **Commit**: `{sha:7}` | |
| ### Summary | |
| {1-2 sentence explanation of what failed and why} | |
| ### Failures Found | |
| | File | Line | Issue | | |
| |------|------|-------| | |
| | `path/to/file.ts` | 42 | Description of the error | | |
| ### How to Fix | |
| {For TypeScript errors} | |
| Check the type definitions and ensure compatibility. | |
| {For ESLint errors} | |
| ```bash | |
| npm run lint:fix | |
| ``` | |
| {For Prettier errors} | |
| ```bash | |
| npm run format | |
| ``` | |
| {For Markdown errors} | |
| ```bash | |
| npx markdownlint-cli2 --fix "**/*.md" | |
| ``` | |
| {For test failures} | |
| Run `npm test` locally to debug. | |
| --- | |
| *Analyzed by [Claude](https://claude.ai/code)* | |
| ``` | |
| ### Step 6: Report outcome | |
| Summarize what you found and what action was taken (comment posted, no PR found, etc.). | |
| claude_args: '--model claude-opus-4-6 --allowedTools "mcp__github_ci__get_ci_status,mcp__github_ci__get_workflow_run_details,mcp__github_ci__download_job_log,Bash(gh run view:*),Bash(gh pr list:*),Bash(gh pr comment:*),Bash(gh pr view:*)"' |