You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[CI] /review run-tests: UI tests selected by ai, visible as PR checks - #40305
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
CI and workflow infrastructure, so it targets main. The slash command activates only once the workflow is on the default branch.
Description of Change
Adds /review run-tests. Two AI models, GPT-6.1 Sol and Claude Opus 5.5, each read the PR diff independently and pick the UI test categories and platforms it can affect. maui-pr-uitests then runs only the union of their picks. A CollectionView-only Android fix runs the CollectionView shards on Android, not all 77 categories on 4 platforms.
Only what the models selected: the per-platform union of both picks
Both models choose the full suite (e.g. core Element changes)
Full suite
Neither model produces a pick
Nothing is queued; the comment says so
/azp run maui-pr-uitests
Full suite, unchanged
Branch, scheduled and manual builds (main, release/*, candidate/*, …)
Full suite, unchanged. ci-uitests.yml only passes a selection when Build.SourceBranch is refs/pull/*.
Nothing is ever added that a model did not choose: no full-suite fallback, unknown category names run nothing, Android and the Material3 lane run only when selected (also enforced in the pipeline template), and only the five declared proposal fields are read.
What the requester sees
Native checks in the PR's checks list.maui-pr-uitests is queued through the Build API as a pull-request build (reason=pullRequest + triggerInfo.pr.sourceSha), so Azure Pipelines reports the run and every job as checks on the PR head, exactly like /azp run.
Skipped checks for everything not selected. One maui-pr-uitests (skipped) <Category> check per category and one maui-pr-uitests (skipped) <Platform> UITests check per platform, plus a maui-pr-uitests (test selection) summary check.
A summary comment in the PR Test Gate layout: title, requester line and Result / Platforms / Categories / Commit badges, with everything else in one collapsed Test selection section: build status, each model's summary, what ran, results with failed tests, the skipped list, and Why these tests with a per-platform table of each model's pick and both rationales. review-run-tests-results.yml updates it in place when the maui-pr-uitests check completes, with a half-hourly sweep as backup.
How it works
Piece
Role
copilot-review-run-tests.md (gh-aw, GPT-6.1 Sol)
Pre-activation authorizes an exact /review run-tests from a write+ user and collects the diff, the category inventory from eng/pipelines/common/ui-tests.yml, which categories have enabled tests per platform, and which the PR's own tests enable. The read-only agent (checkout: false) proposes platforms, categories, full_run, a summary and a rationale. The safe-output job saves the proposal and starts the Opus workflow.
copilot-review-run-tests-opus.md (gh-aw, Claude Opus 5.5, new)
workflow_dispatch only. Verifies the source run (run-tests lock workflow, default branch, same repo, under 3 hours old), reuses its context artifact, and makes an independent proposal with the same shared prompt (shared/run-tests-selection.md). Then it starts the dispatcher with both run ids. If either agent fails, the chain still hands over.
review-run-tests-dispatch.yml
workflow_dispatch on main, with no environment, so its OIDC subject is repo:dotnet/maui:ref:refs/heads/main like review-trigger.yml. Verifies both runs, downloads each proposal (a listed artifact that fails to download fails the job), checks each proposal's model and source run, and runs Invoke-RunTests.ps1.
Invoke-RunTests.ps1
Trusted. Validates each proposal against the inventory, prunes each platform to categories with enabled tests there, and merges the two per platform. When every selected platform runs the same categories they are sent as plain names; otherwise every entry is platform-qualified (android:Button, ios:Label), never a mix. Re-checks the requester's permission, the PR head and its target branch, waits for a verified test-merge commit, queues the pull-request build and cancels it if Azure didn't apply the selection. Then publishes the Skipped and selection checks and the comment.
review-run-tests-results.yml
Updates the comment when the azure-pipelinesmaui-pr-uitests check completes; sweeps every 30 minutes as backup.
eng/pipelines/**/ui-tests*.yml, ci-uitests.yml
Optional selectedRun, selectedPlatforms, selectedCategories parameters filter stages and matrix entries at compile time. CollectionView and Shell expand to their shards; <platform>:<Category> limits a category to one platform, and every stage guard, including the dedicated Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles both the plain and the qualified form. The sample build gets template-evaluated booleans, never parameter text. Defaults expand to exactly main's jobs.
/review run-tests is excluded from /review and from recovery. rebalance-ui-test-categories understands the typed categoryGroupsToTest parameter.
/azp run, /review, /review gate and /review tests are unchanged. Device tests are out of scope.
Model policy exception: the repo's GPT-only automation policy (CopilotPipeline.Tests.ps1, from #38040) gets one narrow exception: exactly claude-opus-5.5, only in the run-tests selection files (the Opus workflow and its lock file, the dispatcher, RunTests.Common.ps1 and the tests). Any other Claude/Anthropic model, or that model anywhere else, still fails the test. Opus only proposes a selection; trusted code validates it, and it can only add tests, never remove the other model's picks.
Security review notes: the new Opus workflow uses only existing secrets (the COPILOT_PAT_0..9 pool via copilot-pat-pool); the dispatcher uses the existing AZDO_TRIGGER_TENANT_ID / AZDO_TRIGGER_CLIENT_ID. No new third-party actions. All actions are pinned by SHA.
Validation
End to end in a fork (kubaflo/maui) through the real GPT agent → Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs succeeded; each PR got the comment with both models' picks and the Skipped checks. Azure is skipped there because the fork has no federated identity.
Real builds on dotnet/maui. Other PRs' merge refs don't have the new pipeline parameters until this merges, so each trial was a temporary draft PR: this branch plus one current PR's change. Invoke-RunTests.ps1 ran locally with both models' proposals from the fork run on the same diff, queued maui-pr-uitests, and posted the comment. Every build got exactly the selected platforms and categories.
The failures are the same tests that failed in an earlier round on the same copied changes, so they come from those PRs, not the selection. The comments were refreshed from the real timelines and test results with Update-RunTestsResults.ps1.
Azure previews of this PR's merge ref:
Defaults: identical to main (13 jobs, 157 matrix entries).
Android Button: 2 jobs, no Material3 lane. Button + Material3: the Material3 lane is included.
Qualified selections: android:Material3 + ios:Label runs only the Material3 lane on Android; android:Shell1 also runs the Shell renderer lane; android:SafeAreaEdges runs its dedicated lane; windows:CollectionView1 runs the CV1 lane; an Android-only CollectionView pick never starts the Windows CV1 lane.
selectedPlatforms containing "x; Remove-Item": only template booleans reach the build step.
Tests:RunTests.Tests.ps1 has 39 Pester tests covering the inventory and #if evaluator, pruning and qualification, the two-model union (one model empty, neither, full only when chosen), AI-only rules, the queue request and dropped-selection guard, the check plan, comment rendering and updates, and static template checks. The ReviewTrigger, Recover-MissedReviewCommands and CopilotPipeline suites pass. gh aw compile --strict succeeds.
Not verified before merge:
The slash command and the dispatcher's OIDC exchange on dotnet/maui itself. GitHub only runs them from the default branch. After merge, comment /review run-tests on a PR to confirm.
The check_run-triggered comment update on dotnet/maui (the script itself is verified; the half-hourly sweep is the backup).
The copilot-pat-pool token selection in the new Opus workflow (the fork used a single secret).
The agent picks platforms and UI test categories from the PR diff. A trusted
safe-output job validates the choice against the pipeline inventory, queues
maui-pr-uitests as a pull-request build so Azure Pipelines reports native
checks on the PR head, publishes a Skipped check for every platform and
category that was not selected, and posts a collapsible summary comment that
is updated when the run finishes.
The previous expression also enabled the zero-test guard in the default
full run, which failed every job whose test step did not produce results.
The dispatcher always sets selectedRun for targeted runs.
…sts only
The policy test now accepts exactly 'claude-opus-5.5' in the run-tests selection files. Any other Claude or Anthropic model, or that model in any other file, still fails.
Adds /review run-tests to .NET MAUI’s CI automation. GPT-6.1 Sol and Claude Opus 5.5 independently select affected UI tests, and trusted scripts combine their selections, queue Azure Pipelines, and report results on the PR.
Changes:
Adds the two-model selection workflow, validated dispatch, and automatic result updates.
Introduces platform/category filtering and zero-test guards while preserving full-suite defaults.
Adds regression tests, command-routing exclusions, and a narrowly scoped model-policy exception.
File
Description
eng/pipelines/common/ui-tests.yml
Filters build stages and test matrices by selection.
eng/pipelines/common/ui-tests-steps.yml
Requires executed test results for selected runs.
eng/pipelines/common/ui-tests-build-sample.yml
Limits shared sample builds to selected platforms.
…ted PRs
selectedCategories is now either all plain names (every selected platform runs the same categories) or all platform-qualified. Every stage guard in ui-tests.yml handles both forms, including the Android Shell renderer, SafeAreaEdges and Material3 lanes and the Windows CV1 lane, so another platform's entry never turns a stage on and a qualified shard is never dropped.
Invoke-RunTests.ps1 refuses to queue when the PR's target branch changed after the context was collected.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Active issues can omit Windows test coverage, leave unsuccessful agent handoffs without a build or explanatory PR comment, and merge distinct test failures by discarding fixture identity.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Preserve fully qualified test names when grouping failures
.github/scripts/RunTests.Common.ps1:612
automatedTestName is the fully qualified test name, so taking only its last segment discards the fixture identity. Failures such as Bugzilla39636.DoesNotCrash and Bugzilla39821.DoesNotCrash then merge into one row when their titles match, hiding which tests failed. Preserve the full identity and parameterized title when grouping, and update the fixtures in RunTests.Tests.ps1 to include fully qualified method names.
Handle unsuccessful agent handoffs without requiring failure
If the agent succeeds with only noop/missing_data, or is skipped by the daily-credit guard, this fallback also skips because it requires failure; the chain stops without a build or explanatory PR comment, even when the other model has a proposal. After validated context, handle an unsuccessful handoff without requiring agent failure, retain the cancellation guard, and regenerate both lock workflows. Same issue in .github/workflows/copilot-review-run-tests-opus.md:94.
Handlers/Items2 also contains the Windows CollectionView2 implementation, registered in Hosting/AppHostBuilderExtensions.cs:67–74. Calling the whole directory Apple-only tells both selectors to omit Windows coverage for those changes. Restrict this rule to the iOS subtree/files, explicitly recognize the Windows implementation, and regenerate both importing workflow locks.
The reason will be displayed to describe this comment to others. Learn more.
Stamping as requested by @kubaflo; no code review performed.
kubaflo
changed the title
[CI] /review run-tests: UI tests selected by GPT-6.1 Sol and Claude Opus 5.5, visible as PR checks
[CI] /review run-tests: UI tests selected by ai, visible as PR checks
Nov 19, 2026
Keep .NET 12 dependency versions. Combine dotnet#40305's selected-platform UI test
builds with the separate Android CoreCLR sample build: the shared stage builds
only the selected iOS/Mac Catalyst platforms, and the Android stage runs only
when Android is selected.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…dotnet#40305)
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!
### Target Branch
- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).
CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.
### Description of Change
Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.
Replaces dotnet#40253.
**When what runs**
| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |
Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.
**What the requester sees**
1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.
**How it works**
| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |
`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.
**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from dotnet#38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.
**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.
### Validation
**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.
**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.
| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| dotnet#40442 | dotnet#40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| dotnet#40443 | dotnet#40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| dotnet#40444 | dotnet#40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| dotnet#40445 | dotnet#40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| dotnet#40446 | dotnet#40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| dotnet#40447 | dotnet#40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| dotnet#40448 | dotnet#40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| dotnet#40449 | dotnet#39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |
The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.
**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.
**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.
**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).
---------
Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
…s permissions, plus rollout fixes (#40468)
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!
### Target Branch
- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).
CI workflow infrastructure follow-up to #40305, so it targets `main`.
### Description of Change
Fixes found by running `/review run-tests` on open PRs after #40305
merged, plus the reviewer findings that were still open on #40305.
**Found in the rollout**
**1. Every targeted run was refused with 403.** Since sprint 237, Azure
DevOps lets an identity with only *Queue builds* set runtime parameters
whose values come from a fixed list (booleans, `values:` strings).
Free-text and `object` parameters need *Edit queue build configuration*,
which the trigger identity doesn't have. `selectedPlatforms` and
`selectedCategories` were objects. Full-suite runs, which pass no
parameters, were queued fine.
`ci-uitests.yml` now takes the selection only as constrained parameters,
so no permission change is needed:
| Parameter | Type | Meaning |
| --- | --- | --- |
| `selectedRun` | boolean | A `/review run-tests` selection |
| `uitestPlatformAndroid`, `…Ios`, `…Catalyst`, `…Windows` | boolean |
Which platforms build and run |
| `uitest_<Category>` (one per category, 77) | string, `values: [none,
android, ios, …, android+ios+catalyst+windows]` | Where that category
runs |
At compile time it rebuilds the same platform-qualified
`selectedCategories` list (`android:Button`, `ios:Label`, …) for
`ui-tests.yml`, so no stage or matrix guard changed. A selected run
always includes a `none:none` sentinel, so an empty selection can never
fall back to "every category". A Pester test keeps the `uitest_*` list
in sync with the categories in `ui-tests.yml`.
**2. A stale test-merge commit returned 400.**
The first run (#40420) selected `Shell1`, `Shell2` and `TitleView` on
Android correctly, but the comment said only "The Azure DevOps request
failed". Azure's response was:
> Unexpected parameter 'selectedRun'; Unexpected parameter
'selectedPlatforms'; Unexpected parameter 'selectedCategories'
GitHub's test-merge commit for that PR still merged a `main` from before
#40305, so the pipeline being built had no selection parameters. GitHub
refreshes those commits lazily: right after the merge, 13 of 38 `main`
PRs still had a stale one. PRs targeting `net11`, `net11.0` and other
branches can never be queued this way, because those branches don't have
the parameters.
| Change | Effect |
| --- | --- |
| Check the test-merge commit's `eng/pipelines/ci-uitests.yml` before
queueing | A stale merge ref gets a clear message (push to the PR, then
comment again) instead of a failed request |
| Check the target branch in pre-activation | On a branch without the
parameters, neither model runs; the PR gets a one-line comment pointing
to `/azp run maui-pr-uitests` |
| Show Azure DevOps' validation messages | Any other rejection says why,
in the comment and the log |
**Open reviewer findings from #40305**
| Change | Effect |
| --- | --- |
| URL-encode changed test paths | A test file with `#` or `?` in its
name is read correctly |
| Read up to 100 changed test files; if any is skipped or unreadable,
don't prune | An AI-selected category enabled only by an unread test
file is no longer dropped as having no tests |
| `Get-RunTestsMergeState` helper | The stale test-merge check is unit
tested (old head, missing commit, retarget, close, conflict) |
| Single-model rationale | When only one model answers, the comment
shows its rationale instead of "No rationale provided" |
### Validation
- `RunTests.Tests.ps1` and `CopilotPipeline.Tests.ps1`: 65 tests pass,
including new ones for every change above.
- The new helpers were checked against real data: the Azure 400 response
from #40420 yields the three messages above; the pipeline check is false
for #40420's stale merge commit and for `net11`, and true for `main`.
- `gh aw compile --strict` succeeds.
- Azure DevOps preview of this branch:
| Parameters | Compiled to |
| --- | --- |
| none | identical to `main` (13 jobs, same matrices) |
| Android `Button` | Android build + 1 Android job (`Button`) |
| Android `Material3` + iOS `Label` | Material3 lane on Android, `Label`
on iOS only |
| Android `Shell1` | Android `Shell1` + the Shell renderer lane |
| Windows `CollectionView1` | WinUI `CollectionView1` + the CV1 lane |
| `Label` on all four | `Label` on each platform |
| `selectedRun` with no category | no stages (never queued by the
dispatcher) |
| `uitest_Button: android;rm` | rejected by Azure DevOps: not a valid
value |
- **Not verified before merge:** the trigger identity actually queueing
with these parameters. Only the default branch can use its federated
credential; after merge, comment `/review run-tests` on a PR.
- Rollout results are in the comments below.
---------
Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
… commit (#40483)
<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!
### Target Branch
- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).
CI workflow infrastructure, so it targets `main`.
### Description of Change
Follow-up to #40468, found while testing `/review run-tests` on open PRs
after it merged.
`maui-pr-uitests` builds GitHub's test-merge commit
(`refs/pull/N/merge`). GitHub does not refresh it when `main` moves; it
waits for activity on the PR. Right after #40468 merged, every open PR's
merge ref still merged an older `main` (21 merged #40305's `main`, 20
merged an even older one). Some had been stale for hours. The pipeline
in those commits does not declare the new `uitest_*` parameters, so the
dispatcher stopped straight away with "merges an older main".
Changes:
- **Wait instead of failing.** `Get-VerifiedPullRequest` now treats a
merge ref whose pipeline cannot take the selection as "not ready yet".
It keeps polling for up to 10 minutes, which covers a refresh GitHub is
already computing. A missing merge of a new head still gives up after
about 3 minutes, as before. The dispatcher job timeout goes from 15 to
25 minutes to fit.
- **Check every parameter that will be sent.**
`Test-RunTestsPipelineSupportsSelection` also requires each
`uitest_<Category>` / `uitestPlatform*` name in the queue request.
Otherwise a category added to `main` later would reach Azure as an
"Unexpected parameter" 400 on PRs whose merge ref predates it.
- If GitHub still hasn't refreshed after 10 minutes, the comment says so
and suggests commenting again later or updating the branch.
### Validation
- `RunTests.Tests.ps1` and `CopilotPipeline.Tests.ps1`: 66 passed. A new
test covers parameters that are missing from the built pipeline,
including regex metacharacters.
- Real `/review run-tests` comments after #40468 merged, using the
current `main` dispatcher (without this PR):
| PR | Merge ref merges | Selection | Result |
| --- | --- | --- | --- |
| #40437 | new `main` (head pushed after the merge) | 9 categories, all
platforms | **Queued targeted build 1692555** as the trigger identity
(no 403). Azure shows `selectedRun=true`, the four platform booleans and
only the 9 `uitest_*` values; only the CollectionView, CV1 and
per-platform stages started |
| #40418 | new `main` | full suite (both models) | Full build 1692548 |
| #40369, #40344, #40288 | #40305's `main` | 5 / 4 / 2 categories |
Stopped: stale merge ref |
| #40420 | older `main` | 4 categories, Android | Stopped: stale merge
ref |
A `/review run-tests` comment, an API read and a no-op base edit did not
refresh a stale merge ref within 10+ minutes; a push to the PR head
refreshes it at once. Until GitHub refreshes the older PRs, they need an
update or push. This PR waits for any refresh GitHub is already
computing, explains the situation in the comment, and keeps a later
category addition from becoming an Azure 400.
Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Target Branch
CI and workflow infrastructure, so it targets
main. The slash command activates only once the workflow is on the default branch.Description of Change
Adds
/review run-tests. Two AI models, GPT-6.1 Sol and Claude Opus 5.5, each read the PR diff independently and pick the UI test categories and platforms it can affect.maui-pr-uiteststhen runs only the union of their picks. A CollectionView-only Android fix runs the CollectionView shards on Android, not all 77 categories on 4 platforms.Replaces #40253.
When what runs
/review run-testson any PR (write access)Elementchanges)/azp run maui-pr-uitestsmain,release/*,candidate/*, …)ci-uitests.ymlonly passes a selection whenBuild.SourceBranchisrefs/pull/*.Nothing is ever added that a model did not choose: no full-suite fallback, unknown category names run nothing, Android and the Material3 lane run only when selected (also enforced in the pipeline template), and only the five declared proposal fields are read.
What the requester sees
maui-pr-uitestsis queued through the Build API as a pull-request build (reason=pullRequest+triggerInfo.pr.sourceSha), so Azure Pipelines reports the run and every job as checks on the PR head, exactly like/azp run.maui-pr-uitests (skipped) <Category>check per category and onemaui-pr-uitests (skipped) <Platform> UITestscheck per platform, plus amaui-pr-uitests (test selection)summary check.review-run-tests-results.ymlupdates it in place when themaui-pr-uitestscheck completes, with a half-hourly sweep as backup.How it works
copilot-review-run-tests.md(gh-aw, GPT-6.1 Sol)/review run-testsfrom a write+ user and collects the diff, the category inventory fromeng/pipelines/common/ui-tests.yml, which categories have enabled tests per platform, and which the PR's own tests enable. The read-only agent (checkout: false) proposes platforms, categories,full_run, a summary and a rationale. The safe-output job saves the proposal and starts the Opus workflow.copilot-review-run-tests-opus.md(gh-aw, Claude Opus 5.5, new)workflow_dispatchonly. Verifies the source run (run-tests lock workflow, default branch, same repo, under 3 hours old), reuses its context artifact, and makes an independent proposal with the same shared prompt (shared/run-tests-selection.md). Then it starts the dispatcher with both run ids. If either agent fails, the chain still hands over.review-run-tests-dispatch.ymlworkflow_dispatchonmain, with no environment, so its OIDC subject isrepo:dotnet/maui:ref:refs/heads/mainlikereview-trigger.yml. Verifies both runs, downloads each proposal (a listed artifact that fails to download fails the job), checks each proposal's model and source run, and runsInvoke-RunTests.ps1.Invoke-RunTests.ps1android:Button,ios:Label), never a mix. Re-checks the requester's permission, the PR head and its target branch, waits for a verified test-merge commit, queues the pull-request build and cancels it if Azure didn't apply the selection. Then publishes the Skipped and selection checks and the comment.review-run-tests-results.ymlazure-pipelinesmaui-pr-uitestscheck completes; sweeps every 30 minutes as backup.eng/pipelines/**/ui-tests*.yml,ci-uitests.ymlselectedRun,selectedPlatforms,selectedCategoriesparameters filter stages and matrix entries at compile time.CollectionViewandShellexpand to their shards;<platform>:<Category>limits a category to one platform, and every stage guard, including the dedicated Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles both the plain and the qualified form. The sample build gets template-evaluated booleans, never parameter text. Defaults expand to exactlymain's jobs.review-trigger.yml,ReviewCommandHelpers.ps1, rebalance skill/review run-testsis excluded from/reviewand from recovery.rebalance-ui-test-categoriesunderstands the typedcategoryGroupsToTestparameter./azp run,/review,/review gateand/review testsare unchanged. Device tests are out of scope.Model policy exception: the repo's GPT-only automation policy (
CopilotPipeline.Tests.ps1, from #38040) gets one narrow exception: exactlyclaude-opus-5.5, only in the run-tests selection files (the Opus workflow and its lock file, the dispatcher,RunTests.Common.ps1and the tests). Any other Claude/Anthropic model, or that model anywhere else, still fails the test. Opus only proposes a selection; trusted code validates it, and it can only add tests, never remove the other model's picks.Security review notes: the new Opus workflow uses only existing secrets (the
COPILOT_PAT_0..9pool viacopilot-pat-pool); the dispatcher uses the existingAZDO_TRIGGER_TENANT_ID/AZDO_TRIGGER_CLIENT_ID. No new third-party actions. All actions are pinned by SHA.Validation
End to end in a fork (kubaflo/maui) through the real GPT agent → Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs succeeded; each PR got the comment with both models' picks and the Skipped checks. Azure is skipped there because the fork has no federated identity.
Real builds on dotnet/maui. Other PRs' merge refs don't have the new pipeline parameters until this merges, so each trial was a temporary draft PR: this branch plus one current PR's change.
Invoke-RunTests.ps1ran locally with both models' proposals from the fork run on the same diff, queuedmaui-pr-uitests, and posted the comment. Every build got exactly the selected platforms and categories.WindowTitleView,LifecycleShell1/2on iOS, MacShell1/2,NavigationTabbedPageDisplayPromptAsyncElement/ VSM threadingButton,ImageButtonon iOS, MacThe failures are the same tests that failed in an earlier round on the same copied changes, so they come from those PRs, not the selection. The comments were refreshed from the real timelines and test results with
Update-RunTestsResults.ps1.Azure previews of this PR's merge ref:
main(13 jobs, 157 matrix entries).Button: 2 jobs, no Material3 lane.Button+Material3: the Material3 lane is included.android:Material3+ios:Labelruns only the Material3 lane on Android;android:Shell1also runs the Shell renderer lane;android:SafeAreaEdgesruns its dedicated lane;windows:CollectionView1runs the CV1 lane; an Android-only CollectionView pick never starts the Windows CV1 lane.selectedPlatformscontaining"x; Remove-Item": only template booleans reach the build step.Tests:
RunTests.Tests.ps1has 39 Pester tests covering the inventory and#ifevaluator, pruning and qualification, the two-model union (one model empty, neither, full only when chosen), AI-only rules, the queue request and dropped-selection guard, the check plan, comment rendering and updates, and static template checks. TheReviewTrigger,Recover-MissedReviewCommandsandCopilotPipelinesuites pass.gh aw compile --strictsucceeds.Not verified before merge:
dotnet/mauiitself. GitHub only runs them from the default branch. After merge, comment/review run-testson a PR to confirm.check_run-triggered comment update ondotnet/maui(the script itself is verified; the half-hourly sweep is the backup).copilot-pat-pooltoken selection in the new Opus workflow (the fork used a single secret).