ci: unified docs orchestrator workflow and metadata sync script by mrpollo · Pull Request #26285 · PX4/PX4-Autopilot

mrpollo · 2026-01-15T21:32:27Z

Summary

Consolidates docs CI into a single orchestrator workflow and replaces scattered metadata scripts with a unified metadata_sync.sh.

What changed

New docs orchestrator (.github/workflows/docs-orchestrator.yml):

Single workflow replaces 3 separate workflows (docs_deploy_aws.yml, docs_flaw_checker.yml, docs_pr_comment.yml)
Jobs organized in tiers (T1-T4) with clear dependency chain
PR flow: T1 Detect Changes → T2 PR Metadata (conditional) + T2 Link Check → T3 Build Site
Push flow: T2 Metadata Sync (auto-commit with [skip ci]) → T3 Build Site → T4 Deploy to AWS
Job-level permissions (least privilege per job)
Triggers on docs/** + source paths (src/**, msg/**, ROMFS/**, Tools/module_config/**)

New unified metadata script (Tools/ci/metadata_sync.sh):

Handles all 6 metadata types: parameters, airframes, modules, msg_docs, uorb_graphs, failsafe_web
Modes: --generate, --sync, --verbose
Pinned Emscripten to 3.1.64
Consistent whitespace normalization

New Makefile target:

msg_docs — generates uORB message documentation

New documentation:

docs/en/test_and_ci/continous_integration.md — documents the docs CI pipeline

Updated metadata (236 docs files):

Regenerated parameter, airframe, module, message, and uORB graph docs with latest source

Deleted workflows (replaced by orchestrator)

docs_deploy_aws.yml
docs_flaw_checker.yml
docs_pr_comment.yml

Kept separately

docs_crowdin_upload.yml — separate trigger on push to main
docs_crowdin_download.yml — scheduled weekly translation download
docs_deploy.yml — manual GitHub Pages deploy

Before / After

	Before	After
Workflows	4 docs workflows	1 orchestrator + 3 independent
Scripts	scattered per-type scripts	1 unified `metadata_sync.sh`
Metadata updates	manual or per-script CI	auto-commit on push to main
Permissions	workflow-level (broad)	job-level (least privilege)
Loop protection	none	`[skip ci]` on auto-commits

Test plan

PR triggers orchestrator (T1 Detect Changes, T2 Link Check, T3 Build Site)
T2 PR Metadata correctly skips when no source files changed
T2 Link Check posts sticky PR comment
T3 Build Site passes (VitePress build)
Push to main triggers T2 Metadata Sync with auto-commit
Auto-commit includes [skip ci] and does not retrigger workflow
T4 Deploy succeeds (AWS S3 sync)
Crowdin upload triggers independently on push to main

docs/en/test_and_ci/continous_integration.md

.github/workflows/docs-orchestrator.yml

hamishwillee

@mrpollo The approach looks good, however I am unable to test it effectively locally. I think it would be great to do this suggestion #26285 (comment) and then merge this and iterate.

Approving to make that possible.

mrpollo · 2026-02-04T15:33:55Z

@hamishwillee I've addressed all your comments:

✅ Prettier formatting step added before auto-commit
✅ Conditional metadata generation for PRs that touch both docs and source files
✅ Tested the version path logic locally - works correctly for main and release/* branches

Let me know if you're happy to merge so we can start enjoying the benefits! 🚀

hamishwillee · 2026-02-04T21:40:48Z

.github/workflows/docs-orchestrator.yml

+        run: |
+          cd docs
+          yarn install --frozen-lockfile
+          yarn prettier --write "en/**/*.md"


I think this will prettier absolutely everything, not just modified files. This will affect efficiency, but it not a bad thing.

hamishwillee

Thanks @mrpollo - looks good. The only "issue" I can see is that prettier runs on everything, and had not been run on this PR. So decided to do it so all changes in other PRs going forwards are deltas.

We probably should run this in v1.17 too, because it's metadata is likely to be out of date.

But, there is a depencency error https://github.com/PX4/PX4-Autopilot/actions/runs/21689629726/job/62545775038?pr=26285

error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.

I think we can fix this by removing the json package lock file - trying that now

.github/workflows/docs-orchestrator.yml

context: our docs need this metadata, this script generates the metadata then moves the files to the required destination. docs: update_uorb_graphs.sh — generate, compare, and sync uORB graph JSONs Usage: ./scripts/update_uorb_graphs.sh [--test-only] [--debug] Options: --test-only Run generation and comparison only; exit 1 if diffs found, without copying files --debug Echo debug info for missing or differing files Examples: # CI mode: fail if docs need updates ./scripts/update_uorb_graphs.sh --test-only # Developer mode: regenerate and sync JSONs ./scripts/update_uorb_graphs.sh Signed-off-by: Ramon Roche <[email protected]>

check if metadata needs to be rebuilt! Signed-off-by: Ramon Roche <[email protected]>

more debug tools Signed-off-by: Ramon Roche <[email protected]>

Signed-off-by: Ramon Roche <[email protected]>

test is broken Signed-off-by: Ramon Roche <[email protected]>

Signed-off-by: Ramon Roche <[email protected]>

Add conditional metadata generation for PRs that modify both docs and source files. This fixes broken link checker reports when a PR adds a new module and documents it simultaneously - previously the link checker would fail because metadata files are only generated on push to main. Changes: - Add docs-orchestrator.yml workflow with: - detect-changes job to check if PR touches source paths - pr-metadata-regen job that generates metadata and uploads as artifact - link-check job that downloads metadata artifact when available - build-site job for VitePress site generation - deploy-aws and crowdin-upload jobs for push events - Prettier formatting step before auto-commit on push - Add Prettier for markdown formatting: - Add prettier ^3.2.0 as devDependency in docs/package.json - Add docs/.prettierrc with prose-preserving config - Remove normalize_whitespace from metadata_sync.sh (Prettier handles whitespace normalization now) Signed-off-by: Ramon Roche <[email protected]>

Regenerate yarn.lock to include prettier dependency that was added to package.json but missing from the lockfile, causing CI to fail with --frozen-lockfile. Also add package-lock.json to .gitignore since the project uses yarn. Signed-off-by: Ramon Roche <[email protected]>

Add fetch-depth: 0 to the checkout step in docs_flaw_checker.yml. Without a full clone, tj-actions/changed-files cannot find the merge base between main and the PR branch, causing the job to fail with "no merge base". Signed-off-by: Ramon Roche <[email protected]>

adds docs Signed-off-by: Ramon Roche <[email protected]>

Pass the changed files JSON through an environment variable instead of direct interpolation to avoid shell quoting issues. Signed-off-by: Ramon Roche <[email protected]>

Signed-off-by: Ramon Roche <[email protected]>

- Add [skip ci] to auto-commit message to prevent infinite workflow loop - Add source paths (src/, msg/, ROMFS/, Tools/module_config/) to push triggers so metadata regenerates on source-only changes - Move permissions from workflow-level to job-level (least privilege) Signed-off-by: Ramon Roche <[email protected]>

After a force push, GitHub evaluates path filters against only the pushed commits, not the full PR diff. Changes to the orchestrator workflow file itself were not matching the docs/** path filter, preventing the orchestrator from running on the PR. Signed-off-by: Ramon Roche <[email protected]>

Prefix job names with T1-T4 to make the execution order and dependency chain visible at a glance in the GitHub Actions UI. Signed-off-by: Ramon Roche <[email protected]>

- Update tier numbering to match T1-T4 job names - Fix PR flow diagram to show T2 jobs running in parallel - Add [skip ci] note to push flow - Separate push vs PR path trigger tables - Add workflow file to PR path triggers - Remove Crowdin from orchestrator deploy tier (separate workflow) - Update infrastructure table with correct job names Signed-off-by: Ramon Roche <[email protected]>

sonarqubecloud · 2026-02-09T18:28:54Z

Quality Gate failed

Failed conditions
10 Security Hotspots

See analysis details on SonarQube Cloud

github-actions bot added the Documentation 📑 label Jan 15, 2026

mrpollo force-pushed the mrpollo/docs_metadata branch 4 times, most recently from 6ec679a to 1116bb8 Compare January 15, 2026 22:41

mrpollo requested a review from hamishwillee January 15, 2026 22:56

mrpollo marked this pull request as ready for review January 15, 2026 22:58

hamishwillee reviewed Jan 29, 2026

View reviewed changes

docs/en/test_and_ci/continous_integration.md Outdated Show resolved Hide resolved

hamishwillee reviewed Jan 29, 2026

View reviewed changes

.github/workflows/docs-orchestrator.yml Show resolved Hide resolved

hamishwillee reviewed Jan 29, 2026

View reviewed changes

.github/workflows/docs-orchestrator.yml Outdated Show resolved Hide resolved

hamishwillee reviewed Jan 29, 2026

View reviewed changes

.github/workflows/docs-orchestrator.yml Show resolved Hide resolved

hamishwillee previously approved these changes Jan 29, 2026

View reviewed changes

mrpollo dismissed hamishwillee’s stale review via ef1f2f4 February 4, 2026 14:50

mrpollo force-pushed the mrpollo/docs_metadata branch from 9566995 to ef1f2f4 Compare February 4, 2026 14:50

hamishwillee reviewed Feb 4, 2026

View reviewed changes

hamishwillee previously approved these changes Feb 4, 2026

View reviewed changes

hamishwillee dismissed their stale review via 709a138 February 4, 2026 21:56

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

.github/workflows/docs-orchestrator.yml Fixed Show fixed Hide fixed

.github/workflows/docs-orchestrator.yml Fixed Show fixed Hide fixed

.github/workflows/docs-orchestrator.yml Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

.github/workflows/docs-orchestrator.yml Fixed Show fixed Hide fixed

mrpollo force-pushed the mrpollo/docs_metadata branch from 0ef2051 to ae9d8a8 Compare February 9, 2026 16:14

mrpollo requested a review from hamishwillee February 9, 2026 18:20

PX4 deleted a comment from github-actions bot Feb 9, 2026

mrpollo added 6 commits February 9, 2026 10:25

ci: metadata checks for docs

b7a8cff

check if metadata needs to be rebuilt! Signed-off-by: Ramon Roche <[email protected]>

tools: update uorb graphs script

0c38d5f

more debug tools Signed-off-by: Ramon Roche <[email protected]>

ci: enable uorb graphs debug

3ece1aa

Signed-off-by: Ramon Roche <[email protected]>

ci: fix git ownership issues

00270f5

Signed-off-by: Ramon Roche <[email protected]>

tools: generate & check failsafe web metadata

d04e0ce

Signed-off-by: Ramon Roche <[email protected]>

mrpollo and others added 28 commits February 9, 2026 10:25

toos: remove unnecessary file

d0fcb56

Signed-off-by: Ramon Roche <[email protected]>

tools: metadata script leftover

e396557

ci: bump px4-dev container

d3200e4

Signed-off-by: Ramon Roche <[email protected]>

docs: update module reference metadata

d613906

docs: update parameter reference metadata

6ff0545

docs: metadata: update uORB graph JSONs

e11aaed

ci: disable the metadata failsafe web check

38d07b0

test is broken Signed-off-by: Ramon Roche <[email protected]>

docs: metadata: update uORB graph JSONs

19f2271

docs: update uORB message reference docs

ed6fc4f

tools: strip trailing whitespace from markdown

c5b4699

Signed-off-by: Ramon Roche <[email protected]>

docs: update parameter reference metadata

3224697

docs: update module reference metadata

847267b

tools: remove trailing whitespace

78a29db

Signed-off-by: Ramon Roche <[email protected]>

ci: debug with tmate

6b421c0

Signed-off-by: Ramon Roche <[email protected]>

tools: metadata modules check with diff

d557701

Signed-off-by: Ramon Roche <[email protected]>

tools: merge all scripts into a single one

f6e85f8

Signed-off-by: Ramon Roche <[email protected]>

Add result of running prettier on everything

fcdded6

REmove package-lock.json - no longer required

fe5a34d

docs: consolidate docs ci

6c212df

adds docs Signed-off-by: Ramon Roche <[email protected]>

ci: fix jq parse error in changed files list step

249adea

Pass the changed files JSON through an environment variable instead of direct interpolation to avoid shell quoting issues. Signed-off-by: Ramon Roche <[email protected]>

docs: fix flaw checker paths

0e70a98

Signed-off-by: Ramon Roche <[email protected]>

ci: rename orchestrator jobs with tier prefixes

bc11845

Prefix job names with T1-T4 to make the execution order and dependency chain visible at a glance in the GitHub Actions UI. Signed-off-by: Ramon Roche <[email protected]>

mrpollo force-pushed the mrpollo/docs_metadata branch from c27458a to ef9c735 Compare February 9, 2026 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: unified docs orchestrator workflow and metadata sync script#26285

ci: unified docs orchestrator workflow and metadata sync script#26285
mrpollo wants to merge 44 commits intomainfrom
mrpollo/docs_metadata

mrpollo commented Jan 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hamishwillee left a comment •

edited

Loading

Uh oh!

mrpollo commented Feb 4, 2026

Uh oh!

hamishwillee Feb 4, 2026

Uh oh!

hamishwillee left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mrpollo commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Deleted workflows (replaced by orchestrator)

Kept separately

Before / After

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hamishwillee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrpollo commented Feb 4, 2026

Uh oh!

hamishwillee Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

hamishwillee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 9, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrpollo commented Jan 15, 2026 •

edited

Loading

hamishwillee left a comment •

edited

Loading

hamishwillee left a comment •

edited

Loading