Skip to content

[ci] Scan C# and Java with CodeQL in a build-only nightly job - #40385

Draft
rolfbjarne wants to merge 2 commits into
mainfrom
rolfbjarne-codeql-build-only-job
Draft

rolfbjarne wants to merge 2 commits into
mainfrom
rolfbjarne-codeql-build-only-job

Conversation

@rolfbjarne

Copy link
Copy Markdown
Member

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Target Branch

This PR is for Initial base
New features, API changes, or .NET 12 work (including .NET 12 fixes) main
.NET 11 servicing fixes net11, for review
Older servicing, release integration, infrastructure, or documentation The branch designated for that work

For .NET 11 fixes, only a maintainer retargets the approved PR to the active
inflight lane and merges it after CI validates that final base. Do not merge
individual fixes directly into net11; agents must not retarget or merge.

This changes the official pipeline (dotnet-maui) on main, so the base is main.

Description of Change

CodeQL rarely finishes a scan of the compiled languages (C# and Java) in the official pipeline (dotnet-maui). This adds a nightly job that builds only the shipping projects, and starts CodeQL after the provisioning, so that only the build can make the scan fail.

Why the scan rarely finishes

  • CodeQL scans a language at most every 72 hours for each branch and BuildIdentifier, and a failed scan still blocks the next one for 72 hours.
  • The 1ES template starts CodeQL right after the checkout in every job, and CodeQL drops its database when the job has already failed by the time CodeQL finishes, whichever step failed.
  • The scan ran in Pack Windows. Since 2026-09-01 a scan was due 24 times there: 13 uploaded and 11 failed. None of the 11 failed while compiling the code. 9 failed in Build Workloads, Sign & Publish (NU1102, e.g. 3076230) and 2 in Provision SDK & Workloads (NU1301, e.g. 3135601, fixed by [ci] Restore official runtime manifests from transferred packages #40320). So no scan was uploaded from 2026-09-15 to 2026-10-16, or since 2026-11-10.
  • The result of the whole build doesn't matter (the builds are red because of OneLocBuild and still uploaded), only the result of the job that runs the scan.
  • The restore and build is the only part of Pack, Sign that the new job keeps. In the 224 builds of main since 2026-08-15, that restore and build failed twice, both times because of a XAML compile error in the code (3115897 and 3116305).

What changes

  • eng/pipelines/ci-official.yml:
    • A new CodeQL stage with the job CodeQL Windows, on the same pool as Pack Windows. It only exists in the scheduled and manual builds of main. The steps are: checkout, the same provisioning as Pack Windows, CodeQL3000Init@0 (C# and Java, with its own BuildIdentifier), build.cmd -restore -build of the shipping projects, CodeQL3000Finalize@0. There is no pack, signing, workload build or publishing. The binlog is published like the one of Pack Windows.
    • The stage is separate from Pack, so that a failure or a slow run can't hold up the publishing. It starts after Pack because the provisioning restores the runtime manifest packages from MacNativeArtifacts ([ci] Restore official runtime manifests from transferred packages #40320).
    • sdl.codeql.compiled.enabled: false turns off the compiled scan that 1ES injects into every job, including Pack Windows. The new job replaces the scan in Pack Windows. Turning off the compiled scan also turns off the interpreted scan, so interpreted.enabled: true keeps it running in SDL Sources.
  • eng/Build.props: /p:BuildPackageProjects=true selects the projects that -pack builds (the build tasks, then Microsoft.Maui.Packages.slnf) without running the Pack target.
  • .github/docs/maui-ci-facts.md: a short section about the stage.

Validation

I expanded the pipeline with the Azure DevOps preview API (nothing was queued), using this ci-official.yml as yamlOverride. The stage is inline, because the preview reads template files from the mirror.

Check Result
CodeQL3000Init@0 and CodeQL3000Finalize@0 in main 8 and 8 before, 2 and 2 after (SDL Sources and CodeQL Windows)
Lines that disappear only the injected pairs of Pack macOS, Pack Windows, Publish Assets, OneLocBuild, NuGet Validation, Signing Validation and Publish Using Darc
SDL Sources unchanged (LanguageDetectorFilter: interpreted, BuildMode: none)
Steps of CodeQL Windows those of Pack Windows up to its build, without the MicroBuild steps, then Initialize, Build, Finalize
Other branches net11.0, inflight/current, candidate/11.0.1xx-sr2 and release/11.0.1xx have no CodeQL stage
Build.Reason the preview is a manual build and has the stage; without Manual in the condition it doesn't
eng/Build.props, for Windows and macOS Pack=true and BuildPackageProjects=true select the same projects; with neither, the selection is unchanged

Not verified

The job hasn't run: it needs this change in the internal mirror. Things that can only be seen in the first real run:

  • The explicit CodeQL tasks have not run in this pipeline before. CodeQL Finalize runs while network isolation is on, and the injected Finalize ran after it ended. The isolation policy is Permissive, and the injected CodeQL Initialize already ran while it was on.
  • If the first scan fails after CodeQL Initialize, the next one is only due after 72 hours.
  • The Windows build of Microsoft.Maui.Packages.slnf without -pack hasn't run. It is the same project set as in Pack, Sign, without the Pack target.

Good to know

  • On most nights no scan is due. CodeQL then says skipped in the CodeQL Initialize log, and the job is just a build. A green job doesn't mean that a scan happened.
  • The nightly takes longer overall, because the stage starts after Pack. The publishing stages don't wait for it, but the job uses the same agent pool as the validation stage.
  • First run: queue a manual build of main. CodeQL Initialize should say New csharp database will be built now and database initialized, and CodeQL Finalize should say database uploaded.
  • To keep the old scan in Pack Windows for now (both would scan, with different BuildIdentifiers), set templateContext.sdl.codeql.compiled.enabled to true on the pack_net_Windows job in stage-pack.yml. I didn't preview that, because the preview can't read template changes before they are in the mirror.
  • ci-official-release.yml is unchanged.

Issues Fixed

None. There is no GitHub issue for this change.

rolfbjarne and others added 2 commits November 18, 2026 12:40
The 1ES template starts CodeQL right after the checkout in every job, so
any later failure in the job drops the database. CodeQL scans a language
at most every 72 hours, and a failed scan counts too. Since 2026-09-01,
11 of the 24 scans in Pack Windows failed, none of them in the compile.

Add a CodeQL stage to the official pipeline, for the scheduled and manual
builds of main. Its Windows job provisions first, then runs CodeQL around
a build of the shipping projects only (no pack, sign or publish). The
compiled scan that 1ES injects into every job is turned off; the
interpreted scan in SDL Sources stays.

eng/Build.props: BuildPackageProjects=true selects the projects of
Microsoft.Maui.Packages.slnf like -pack does, without running Pack.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40385

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40385"

@azure-pipelines

This comment has been minimized.

@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 18, 2026
@kubaflo

kubaflo commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

/rebase

@kubaflo

This comment has been minimized.

@azure-pipelines

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 0 findings

Reviewed commit 846f774.

@MauiBot

MauiBot commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@rolfbjarne — existing-test verification for commit 846f7747.

Result SKIPPED Platform Android Commit 846f7747


🧪 Gate analysis — click to expand

⚠ Skipped. No runnable PR tests were selected. The fix has not been verified by this gate.

No interpretable test-group summary was produced. No additional test execution is inferred.

PR head: 846f7747.

Without-fix baseline: 331056d8.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 18, 2026
kubaflo pushed a commit to kubaflo/maui that referenced this pull request Nov 18, 2026
…only nightly job

Copy of dotnet#40385 for a /review run-tests trial. Do not merge.
kubaflo added a commit that referenced this pull request Nov 20, 2026
…#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces #40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from #38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| #40442 | #40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| #40443 | #40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| #40444 | #40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| #40445 | #40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| #40446 | #40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| #40447 | #40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| #40448 | #40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| #40449 | #39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
@kubaflo

kubaflo commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

/review run-tests

@github-actions

Copy link
Copy Markdown
Contributor

Selected UI Tests

@kubaflo — AI-selected UI tests (GPT-6.1 Sol + Claude Opus 5.5) for commit 846f7747.

Result NOTHING TO RUN Platforms none Categories 0 of 77 Commit 846f7747


🧪 Test selection — click to expand

ℹ️ The AI found no UI-test-relevant changes, so no maui-pr-uitests run was queued.

GPT-6.1 Sol: No UI tests needed: changes affect only official CodeQL builds and documentation, with default UI-test builds unchanged.
Claude Opus 5.5: CI-only change adding a nightly CodeQL build-only job; no UI tests needed.

🧪 Running — nothing

No UI tests run for this PR.

⏭️ Skipped — 77 categories, 4 platforms

Platforms not run: Android, iOS, MacCatalyst, Windows

Each skipped platform and category is listed as a Skipped maui-pr-uitests (skipped) … check on this commit. Use /azp run maui-pr-uitests for the full suite.

🤖 Why these tests — AI rationale (GPT-6.1 Sol + Claude Opus 5.5)

Each model chose independently; every test that runs was selected by at least one of them.

Platform GPT-6.1 Sol Claude Opus 5.5
Android — —
iOS — —
MacCatalyst — —
Windows — —

GPT-6.1 Sol — No UI tests needed: changes affect only official CodeQL builds and documentation, with default UI-test builds unchanged.

  • eng/pipelines/ci-official.yml adds an official-only CodeQL job; docs describe it.
  • eng/Build.props changes project selection only with opt-in BuildPackageProjects=true; default builds are unchanged.
  • No product or UI test changes; all platforms omitted.

Claude Opus 5.5 — CI-only change adding a nightly CodeQL build-only job; no UI tests needed.

  • eng/pipelines/ci-official.yml: official/nightly pipeline only, not maui-pr-uitests.
  • eng/Build.props: new behavior only behind opt-in BuildPackageProjects=true; default build unchanged.
  • .github/docs/maui-ci-facts.md: docs only.
  • No product or test code changed, so no UI categories selected.

Generated by /review run-tests. A targeted run is a fast signal, not full-suite certification.

HarishKumarSF4517 pushed a commit to HarishKumarSF4517/maui that referenced this pull request Nov 20, 2026
…dotnet#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces dotnet#40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from dotnet#38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| dotnet#40442 | dotnet#40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| dotnet#40443 | dotnet#40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| dotnet#40444 | dotnet#40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| dotnet#40445 | dotnet#40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| dotnet#40446 | dotnet#40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| dotnet#40447 | dotnet#40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| dotnet#40448 | dotnet#40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| dotnet#40449 | dotnet#39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — 846f7747 Deployed Nov 18, 2026 by rolfbjarne via conclusion #3086
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants