Repository navigation
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Serialize runtime selections as JSON strings, confirm effective coverage and revision, and cancel unexpected runs before reporting them. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Broaden missing runtime patches and execution script changes, and match complete platform control names to avoid Image/ImageButton prefix collisions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40253Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40253" |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Expert Review — 6 findings
Reviewed commit a560c93.
See inline comments for details.
2 findings could not be attached inline; retained in run artifacts.
| Get-Content $PSCommandPath | ||
| $buildCommand = "./build.ps1 --target=uitests-apphost --configuration=${{ parameters.configuration }} --${{ parameters.platform }} --verbosity=diagnostic --usenuget=false --runtimevariant=${{ parameters.runtimeVariant }}" | ||
| $platformArguments = "--${{ parameters.platform }}" | ||
| if ("${{ join(',', parameters.platforms) }}" -ne "") { |
There was a problem hiding this comment.
[major] Input and Path Correctness - The ci-uitests.yml selectedPlatforms object parameter flows through ui-tests.yml into this double-quoted PowerShell source without element validation. A direct pipeline request containing android and an element with a PowerShell subexpression evaluates that expression here, before the Where-Object allowlist runs, rather than treating it as platform data. The hosted selector's whitelist does not cover direct pipeline requests. Transport the array as JSON through an environment variable and validate identifiers before constructing process arguments.
| schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber | ||
| headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha | ||
| platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device | ||
| runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0)) |
There was a problem hiding this comment.
[moderate] Logic and Correctness - A Windows-only WebViewHandler.Windows.cs change selects platforms=[windows] and uiCategories=[WebView], making runUi true. However, common/ui-tests.yml explicitly excludes Windows WebView, and its stage gates remove every stage for this combination. The dispatcher consequently submits a UI pipeline with no stages instead of omitting unsupported UI coverage, and the request fails. Resolve platform/category support before deciding to queue the UI pipeline.
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: | ||
| ${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}: |
There was a problem hiding this comment.
[moderate] Logic and Correctness - Accepted inventory identifiers do not all map to categoryGroupsToTest. For example, changing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests accepts that value, but no category group contains it, so this exact-match gate and its Apple/Windows equivalents emit no matching selected lane. Map every accepted selection to a nonempty matrix or widen unsupported selections before dispatch.
| ${{ categoryGroup }}: | ||
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: |
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - The new per-category matrix needs a platform-empty regression case. For a PR editing only shared Tests/Issues/Issue32989.cs, the no-AI fallback selects all four platforms and Essentials. The three Essentials fixtures are compiled only under ANDROID or WINDOWS, so the isolated iOS and MacCatalyst Essentials jobs execute zero tests and fail the new nonzero guard. Previously Essentials shared a group with categories that execute on Apple. Exclude unsupported pairs or widen their categories, and cover this case in a selector/matrix regression test.
pulls.base.sha is the base tip at the PR's last sync, while GitHub rebuilds merge_commit_sha on the current base tip. Require the merge's first parent to descend from the PR base instead of equal it, and freeze that parent as the selection base. Reject merges built on base commits outside the base history with an actionable message. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Real PRs widened to every platform because shared UI tests wrapped in a file-wide #if, HostApp issue pages, and screenshot baselines were treated as unresolved. Evaluate file-wide guards against each UI test project's DefineConstants (failing closed on #else, trailing code or unknown symbols), map issue pages and snapshot baselines to their owning UI test, and recognize [ShardedTestCategory]. Unit-test-only paths no longer add platforms because they run in maui-pr. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Feature-area runtime changes previously made the floor broad, discarding category narrowing. Shell, CollectionView/CarouselView, NavigationPage, TabbedPage and FlyoutPage now add their inventory-validated UI and device categories instead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/rebase |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Expert Review — 5 findings
Reviewed commit 406cb26.
See inline comments for details.
Previously reported findings were not duplicated.
1 findings could not be attached inline; retained in run artifacts.
| schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber | ||
| headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha | ||
| platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device | ||
| runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0)) |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], so runUi is true even without AI output. Queue-SelectedPRTests.ps1 consequently submits the UI pipeline, but common/ui-tests.yml excludes Windows WebView and removes every stage for this combination. The UI request has no runnable stages instead of retaining only supported device/build coverage. Resolve supported platform/category pairs before deciding to dispatch, and cover this negative case in a selector-to-pipeline regression test.
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: | ||
| ${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}: |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - Editing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests.ps1 accepts that declared inventory value, but categoryGroupsToTest contains no such category; this gate and its Apple/Windows equivalents emit no matrix entry for it. The selected run therefore cannot execute the changed fixture. Map accepted inventory values to runnable lanes or widen selections with no matching group, and add a discriminating regression case for an accepted category absent from the matrix.
| ${{ categoryGroup }}: | ||
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: |
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - File-wide guard detection fixes Issue32989.cs alone, but not a PR that also edits shared Tests/ButtonUITests.cs. The selector unions all four platforms with Button and Essentials; this loop and the MacCatalyst equivalent then emit standalone Essentials jobs on Apple. The only three Essentials fixtures are WINDOWS-only Issue32989 and ANDROID-only Issue36523/Issue37638, so those jobs execute zero tests and fail requireExecutedTests. Preserve platform/category associations or widen unsupported pairs, and add a multi-file selector-to-matrix regression test.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Expert Review — 6 findings
Reviewed commit e1bf4fd.
See inline comments for details.
Previously reported findings were not duplicated.
| <AndroidPackageName>%(_SelectedApk.PackageName)</AndroidPackageName> | ||
| <WorkItemTimeout>02:00:00</WorkItemTimeout> | ||
| <TestTimeout>01:00:00</TestTimeout> | ||
| <CustomCommands>xharness android test --app "$app" --package-name "$package_name" --output-directory "$output_directory" --timeout "$timeout" --expected-exit-code "$expected_exit_code" --arg "TestFilter=Category=%(_SelectedApk.Category)" && "$HELIX_PYTHONPATH" "$HELIX_CORRELATION_PAYLOAD/device-selection/verify-results.py" "$output_directory"</CustomCommands> |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - Splitting mobile coverage into single-category exclusion filters drops tests carrying multiple categories, even when every category is selected. Controls ButtonTests has class category Button and NestedButtonHasExpectedIconPosition has method category Layout. DeviceTestSharedHelpers.GetExcludedTestCategories turns Category=Button into exclusions including Layout, and Category=Layout into exclusions including Button; MauiProgram passes these exclusions to the headless runner. Neither selected Android work item executes this test, although an unrestricted run does. Preserve inclusive category matching or widen such selections rather than treating separate exclusion-filter runs as their union.
| schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber | ||
| headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha | ||
| platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device | ||
| runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0)) |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], so runUi is true even without AI output. Queue-SelectedPRTests.ps1 submits the UI pipeline, but common/ui-tests.yml excludes Windows WebView and removes every stage for this combination. The UI request has no runnable stages instead of retaining only supported device/build coverage. Resolve supported platform/category pairs before deciding to dispatch, and cover this negative case in a selector-to-pipeline regression test.
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: | ||
| ${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}: |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - Editing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests.ps1 accepts that declared inventory value, but categoryGroupsToTest contains no such category; this gate and its Apple/Windows equivalents emit no matrix entry for it. The selected run cannot execute the changed fixture. Map accepted inventory values to runnable lanes or widen selections with no matching group, and add a regression case for an accepted category absent from the matrix.
| ${{ categoryGroup }}: | ||
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: |
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - File-wide guard detection handles Issue32989.cs alone, but not a PR that also edits shared Tests/ButtonUITests.cs. Without AI output, the selector unions all four platforms with Button and Essentials; this loop and the MacCatalyst equivalent emit standalone Essentials jobs on Apple. The only three Essentials fixtures are WINDOWS-only Issue32989 and ANDROID-only Issue36523/Issue37638, so those jobs execute zero tests and fail requireExecutedTests. Preserve platform/category associations or widen unsupported pairs, and add a multi-file selector-to-matrix regression test.
| selection_ready: ${{ steps.context.outputs.ready }} | ||
| selection-fallback: | ||
| needs: [pre_activation, activation, agent] | ||
| if: ${{ !cancelled() && needs.pre_activation.outputs.selection_ready == 'true' && needs.agent.result == 'failure' }} |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - The fallback and normal dispatch paths are not mutually exclusive when the agent fails after emitting a proposal. The compiled queue_selected_tests condition accepts any non-skipped agent result with that output type, while this fallback also runs for failure and invokes the dispatcher without AgentOutputPath. An emitted proposal adding Layout to a Button floor followed by an agent/postprocessing failure therefore queues both the floor-only selection and the expanded selection. Find-SelectedRun only reuses exact parameter matches, so it does not suppress these overlapping UI/device runs. Select one dispatch path for a failed agent with collected output.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Tests Failure Analysis🧪 CI Analysis — click to expand📊 maui-pr
🧪 maui-pr-devicetests
🧪 maui-pr-uitestsMatched failures predate these changes on main; default shards/test code remain unchanged.
🧭 Follow-up — actions and refreshNext action: Restore missing diagnostics/results and inspect unresolved launch, setup, and image failures before attributing them to this PR.
|
…targeted-review-tests
Selected PR TestsNote Preview of
🎯 Detected coverage
✅ Results
🧪 Selection notes — click to expandSelected runs are not full-suite certification; queued does not mean passed. Coverage notes:
Generated by review run-tests. |
Selected PR TestsNote Preview of
🎯 Detected coverage
❌ Results
Failed tests (28)
Comment 🧪 Selection notes — click to expandSelected runs are not full-suite certification; queued does not mean passed. Coverage notes:
Generated by review run-tests. |
Embed queued Azure build IDs in the /review run-tests comment and add a scheduled updater that edits its results section with run status, pass/fail/not-run totals and failing tests verified through per-run result details. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
|
/rebase |
|
Unable to rebase this branch automatically. It may have conflicts with its base branch, or the PR author has not allowed maintainer edits. Please rebase manually. |
PR Test Gate |
There was a problem hiding this comment.
Expert Review — 7 findings
Reviewed commit a00e2dc.
See inline comments for details.
Previously reported findings were not duplicated.
| Get-Content $PSCommandPath | ||
| $buildCommand = "./build.ps1 --target=uitests-apphost --configuration=${{ parameters.configuration }} --${{ parameters.platform }} --verbosity=diagnostic --usenuget=false --runtimevariant=${{ parameters.runtimeVariant }}" | ||
| $platformArguments = "--${{ parameters.platform }}" | ||
| if ("${{ join(',', parameters.platforms) }}" -ne "") { |
There was a problem hiding this comment.
[major] Input and Path Correctness - The caller-supplied ci-uitests.yml selectedPlatforms object flows through common/ui-tests.yml into this double-quoted PowerShell source without element validation. A direct pipeline request containing android and an element with a PowerShell subexpression evaluates that expression here, before the Where-Object allowlist runs, instead of treating it as platform data. The hosted selector's whitelist does not cover direct pipeline requests. Pass the array as JSON through an environment variable and validate identifiers before constructing process arguments.
| <AndroidPackageName>%(_SelectedApk.PackageName)</AndroidPackageName> | ||
| <WorkItemTimeout>02:00:00</WorkItemTimeout> | ||
| <TestTimeout>01:00:00</TestTimeout> | ||
| <CustomCommands>xharness android test --app "$app" --package-name "$package_name" --output-directory "$output_directory" --timeout "$timeout" --expected-exit-code "$expected_exit_code" --arg "TestFilter=Category=%(_SelectedApk.Category)" && "$HELIX_PYTHONPATH" "$HELIX_CORRELATION_PAYLOAD/device-selection/verify-results.py" "$output_directory"</CustomCommands> |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - Separate single-category mobile work items are not an inclusive union. Controls ButtonTests has class category Button, while NestedButtonHasExpectedIconPosition also has method category Layout. DeviceTestSharedHelpers.GetExcludedTestCategories converts Category=Button into exclusions including Layout, and Category=Layout into exclusions including Button; MauiProgram passes these exclusions to the headless runner. Neither selected Android work item executes this test, even when both categories are selected, although an unrestricted run does. Other tests satisfy the nonzero-results guard and hide the omission. Preserve inclusive matching or widen such selections, and add a dual-category coverage regression case.
| schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber | ||
| headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha | ||
| platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device | ||
| runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0)) |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], making runUi true even without AI output. Queue-SelectedPRTests submits the UI pipeline, but common/ui-tests.yml explicitly excludes Windows WebView and removes every stage for this combination. The UI dispatch has no runnable stages instead of retaining only supported device/build coverage. Resolve platform/category support before deciding to queue UI coverage.
| CATEGORYGROUP: ${{ categoryGroup }} | ||
| ${{ else }}: | ||
| ${{ each category in split(categoryGroup, ',') }}: | ||
| ${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}: |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - Accepted inventory values do not all map to this matrix. Editing shared Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests accepts that declared value, but categoryGroupsToTest contains no MovedFromCompatibility entry; this gate and its Apple/Windows equivalents produce no matching selected lane. The selected run cannot execute the changed fixture. Map every accepted category to a runnable lane or widen selections with no matching group.
| selection_ready: ${{ steps.context.outputs.ready }} | ||
| selection-fallback: | ||
| needs: [pre_activation, activation, agent] | ||
| if: ${{ !cancelled() && needs.pre_activation.outputs.selection_ready == 'true' && needs.agent.result == 'failure' }} |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - The fallback and normal dispatch paths can both run when the agent fails after emitting a proposal. The compiled queue_selected_tests condition accepts any non-skipped agent result with that output type, while this fallback also runs for failure and invokes the dispatcher without AgentOutputPath. A proposal adding Layout to a Button floor followed by an agent/postprocessing failure therefore queues both the floor-only and expanded UI/device selections. Find-SelectedRun requires exact parameter equivalence, so it cannot deduplicate these overlapping runs. Make the two dispatch paths mutually exclusive.
| It 'replaces only the results block with verified Azure outcomes and marks completed runs final' { | ||
| $responses = @{ | ||
| 'https://dev.azure.com/dnceng-public/public/_apis/build/builds/22?api-version=7.1' = [pscustomobject]@{ status = 'completed'; result = 'failed'; definition = [pscustomobject]@{ id = 313 } } | ||
| 'https://vstmr.dev.azure.com/dnceng-public/public/_apis/testresults/resultsummarybybuild?buildId=22&api-version=7.1-preview.1' = [pscustomobject]@{ |
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - The only outcome-rendering fixture combines result=failed with two failed tests, so it cannot discriminate the formatter's independent failure signals. Removing the completed-build-result check would leave every new assertion satisfied, but a provisioning-failed build with zero published tests would receive a green Tests-Passed badge. Add separate fixtures for a failed build with zero test rows and a successful build with no failed tests, asserting both the failure-before-tests text and the non-failing badge case.
|
Superseded by #40305, which reworks /review run-tests from |
…#40305) <!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Target Branch - [x] I selected the applicable base branch using the [contribution guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting). CI and workflow infrastructure, so it targets `main`. The slash command activates only once the workflow is on the default branch. ### Description of Change Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude Opus 5.5**, each read the PR diff independently and pick the UI test categories and platforms it can affect. `maui-pr-uitests` then runs **only the union of their picks**. A CollectionView-only Android fix runs the CollectionView shards on Android, not all 77 categories on 4 platforms. Replaces #40253. **When what runs** | Trigger | What runs | | --- | --- | | `/review run-tests` on any PR (write access) | Only what the models selected: the per-platform union of both picks | | Both models choose the full suite (e.g. core `Element` changes) | Full suite | | Neither model produces a pick | Nothing is queued; the comment says so | | `/azp run maui-pr-uitests` | Full suite, unchanged | | Branch, scheduled and manual builds (`main`, `release/*`, `candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes a selection when `Build.SourceBranch` is `refs/pull/*`. | Nothing is ever added that a model did not choose: no full-suite fallback, unknown category names run nothing, Android and the Material3 lane run only when selected (also enforced in the pipeline template), and only the five declared proposal fields are read. **What the requester sees** 1. **Native checks in the PR's checks list.** `maui-pr-uitests` is queued through the Build API as a *pull-request* build (`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines reports the run and every job as checks on the PR head, exactly like `/azp run`. 2. **Skipped checks for everything not selected.** One `maui-pr-uitests (skipped) <Category>` check per category and one `maui-pr-uitests (skipped) <Platform> UITests` check per platform, plus a `maui-pr-uitests (test selection)` summary check. 3. **A summary comment in the PR Test Gate layout:** title, requester line and Result / Platforms / Categories / Commit badges, with everything else in one collapsed **Test selection** section: build status, each model's summary, what ran, results with failed tests, the skipped list, and **Why these tests** with a per-platform table of each model's pick and both rationales. `review-run-tests-results.yml` updates it in place when the `maui-pr-uitests` check completes, with a half-hourly sweep as backup. **How it works** | Piece | Role | | --- | --- | | `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation authorizes an exact `/review run-tests` from a write+ user and collects the diff, the category inventory from `eng/pipelines/common/ui-tests.yml`, which categories have enabled tests per platform, and which the PR's own tests enable. The read-only agent (`checkout: false`) proposes platforms, categories, `full_run`, a summary and a rationale. The safe-output job saves the proposal and starts the Opus workflow. | | `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) | `workflow_dispatch` only. Verifies the source run (run-tests lock workflow, default branch, same repo, under 3 hours old), reuses its context artifact, and makes an independent proposal with the same shared prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher with both run ids. If either agent fails, the chain still hands over. | | `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with **no environment**, so its OIDC subject is `repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`. Verifies both runs, downloads each proposal (a listed artifact that fails to download fails the job), checks each proposal's model and source run, and runs `Invoke-RunTests.ps1`. | | `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the inventory, prunes each platform to categories with enabled tests there, and merges the two per platform. When every selected platform runs the same categories they are sent as plain names; otherwise every entry is platform-qualified (`android:Button`, `ios:Label`), never a mix. Re-checks the requester's permission, the PR head and its target branch, waits for a verified test-merge commit, queues the pull-request build and cancels it if Azure didn't apply the selection. Then publishes the Skipped and selection checks and the comment. | | `review-run-tests-results.yml` | Updates the comment when the `azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30 minutes as backup. | | `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional `selectedRun`, `selectedPlatforms`, `selectedCategories` parameters filter stages and matrix entries at compile time. `CollectionView` and `Shell` expand to their shards; `<platform>:<Category>` limits a category to one platform, and every stage guard, including the dedicated Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles both the plain and the qualified form. The sample build gets template-evaluated booleans, never parameter text. Defaults expand to exactly `main`'s jobs. | | `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill | `/review run-tests` is excluded from `/review` and from recovery. `rebalance-ui-test-categories` understands the typed `categoryGroupsToTest` parameter. | `/azp run`, `/review`, `/review gate` and `/review tests` are unchanged. Device tests are out of scope. **Model policy exception:** the repo's GPT-only automation policy (`CopilotPipeline.Tests.ps1`, from #38040) gets one narrow exception: exactly `claude-opus-5.5`, only in the run-tests selection files (the Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1` and the tests). Any other Claude/Anthropic model, or that model anywhere else, still fails the test. Opus only proposes a selection; trusted code validates it, and it can only add tests, never remove the other model's picks. **Security review notes:** the new Opus workflow uses only existing secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` / `AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are pinned by SHA. ### Validation **End to end in a fork (kubaflo/maui)** through the real GPT agent → Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs succeeded; each PR got the comment with both models' picks and the Skipped checks. Azure is skipped there because the fork has no federated identity. **Real builds on dotnet/maui.** Other PRs' merge refs don't have the new pipeline parameters until this merges, so each trial was a temporary draft PR: this branch plus one current PR's change. `Invoke-RunTests.ps1` ran locally with both models' proposals from the fork run on the same diff, queued `maui-pr-uitests`, and posted the comment. Every build got exactly the selected platforms and categories. | Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result | | --- | --- | --- | --- | --- | --- | | #40442 | #40369 Windows title bar | `Window` | + `TitleView`, `Lifecycle` | Windows, 3 categories | Passed in 39 min | | #40443 | #40366 Shell modal stack | `Shell1/2` on iOS, Mac | same | iOS + MacCatalyst | Passed | | #40444 | #40363 Android EmptyView | CollectionView + CarouselView | same | Android, 8 categories | Passed in 41 min | | #40445 | #40344 Shell tab reselect | `Shell1/2`, `Navigation` | + `TabbedPage` | All platforms, 4 categories | Running at time of writing | | #40446 | #40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet | same | All platforms, 3 categories | 1 test failed on MacCatalyst | | #40447 | #40234 `Element` / VSM threading | Full suite | Full suite | Full suite | Running at time of writing | | #40448 | #40385 CodeQL CI-only | Nothing | Nothing | Nothing (no build) | Comment only | | #40449 | #39487 Button corner radius | `Button`, `ImageButton` on iOS, Mac | same | iOS + MacCatalyst | 1 Button test failed | The failures are the same tests that failed in an earlier round on the same copied changes, so they come from those PRs, not the selection. The comments were refreshed from the real timelines and test results with `Update-RunTestsResults.ps1`. **Azure previews of this PR's merge ref:** - Defaults: identical to `main` (13 jobs, 157 matrix entries). - Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`: the Material3 lane is included. - Qualified selections: `android:Material3` + `ios:Label` runs only the Material3 lane on Android; `android:Shell1` also runs the Shell renderer lane; `android:SafeAreaEdges` runs its dedicated lane; `windows:CollectionView1` runs the CV1 lane; an Android-only CollectionView pick never starts the Windows CV1 lane. - `selectedPlatforms` containing `"x; Remove-Item"`: only template booleans reach the build step. **Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the inventory and `#if` evaluator, pruning and qualification, the two-model union (one model empty, neither, full only when chosen), AI-only rules, the queue request and dropped-selection guard, the check plan, comment rendering and updates, and static template checks. The `ReviewTrigger`, `Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw compile --strict` succeeds. **Not verified before merge:** - The slash command and the dispatcher's OIDC exchange on `dotnet/maui` itself. GitHub only runs them from the default branch. After merge, comment `/review run-tests` on a PR to confirm. - The `check_run`-triggered comment update on `dotnet/maui` (the script itself is verified; the half-hourly sweep is the backup). - The `copilot-pat-pool` token selection in the new Opus workflow (the fork used a single secret). --------- Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
…dotnet#40305) <!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Target Branch - [x] I selected the applicable base branch using the [contribution guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting). CI and workflow infrastructure, so it targets `main`. The slash command activates only once the workflow is on the default branch. ### Description of Change Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude Opus 5.5**, each read the PR diff independently and pick the UI test categories and platforms it can affect. `maui-pr-uitests` then runs **only the union of their picks**. A CollectionView-only Android fix runs the CollectionView shards on Android, not all 77 categories on 4 platforms. Replaces dotnet#40253. **When what runs** | Trigger | What runs | | --- | --- | | `/review run-tests` on any PR (write access) | Only what the models selected: the per-platform union of both picks | | Both models choose the full suite (e.g. core `Element` changes) | Full suite | | Neither model produces a pick | Nothing is queued; the comment says so | | `/azp run maui-pr-uitests` | Full suite, unchanged | | Branch, scheduled and manual builds (`main`, `release/*`, `candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes a selection when `Build.SourceBranch` is `refs/pull/*`. | Nothing is ever added that a model did not choose: no full-suite fallback, unknown category names run nothing, Android and the Material3 lane run only when selected (also enforced in the pipeline template), and only the five declared proposal fields are read. **What the requester sees** 1. **Native checks in the PR's checks list.** `maui-pr-uitests` is queued through the Build API as a *pull-request* build (`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines reports the run and every job as checks on the PR head, exactly like `/azp run`. 2. **Skipped checks for everything not selected.** One `maui-pr-uitests (skipped) <Category>` check per category and one `maui-pr-uitests (skipped) <Platform> UITests` check per platform, plus a `maui-pr-uitests (test selection)` summary check. 3. **A summary comment in the PR Test Gate layout:** title, requester line and Result / Platforms / Categories / Commit badges, with everything else in one collapsed **Test selection** section: build status, each model's summary, what ran, results with failed tests, the skipped list, and **Why these tests** with a per-platform table of each model's pick and both rationales. `review-run-tests-results.yml` updates it in place when the `maui-pr-uitests` check completes, with a half-hourly sweep as backup. **How it works** | Piece | Role | | --- | --- | | `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation authorizes an exact `/review run-tests` from a write+ user and collects the diff, the category inventory from `eng/pipelines/common/ui-tests.yml`, which categories have enabled tests per platform, and which the PR's own tests enable. The read-only agent (`checkout: false`) proposes platforms, categories, `full_run`, a summary and a rationale. The safe-output job saves the proposal and starts the Opus workflow. | | `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) | `workflow_dispatch` only. Verifies the source run (run-tests lock workflow, default branch, same repo, under 3 hours old), reuses its context artifact, and makes an independent proposal with the same shared prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher with both run ids. If either agent fails, the chain still hands over. | | `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with **no environment**, so its OIDC subject is `repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`. Verifies both runs, downloads each proposal (a listed artifact that fails to download fails the job), checks each proposal's model and source run, and runs `Invoke-RunTests.ps1`. | | `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the inventory, prunes each platform to categories with enabled tests there, and merges the two per platform. When every selected platform runs the same categories they are sent as plain names; otherwise every entry is platform-qualified (`android:Button`, `ios:Label`), never a mix. Re-checks the requester's permission, the PR head and its target branch, waits for a verified test-merge commit, queues the pull-request build and cancels it if Azure didn't apply the selection. Then publishes the Skipped and selection checks and the comment. | | `review-run-tests-results.yml` | Updates the comment when the `azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30 minutes as backup. | | `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional `selectedRun`, `selectedPlatforms`, `selectedCategories` parameters filter stages and matrix entries at compile time. `CollectionView` and `Shell` expand to their shards; `<platform>:<Category>` limits a category to one platform, and every stage guard, including the dedicated Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles both the plain and the qualified form. The sample build gets template-evaluated booleans, never parameter text. Defaults expand to exactly `main`'s jobs. | | `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill | `/review run-tests` is excluded from `/review` and from recovery. `rebalance-ui-test-categories` understands the typed `categoryGroupsToTest` parameter. | `/azp run`, `/review`, `/review gate` and `/review tests` are unchanged. Device tests are out of scope. **Model policy exception:** the repo's GPT-only automation policy (`CopilotPipeline.Tests.ps1`, from dotnet#38040) gets one narrow exception: exactly `claude-opus-5.5`, only in the run-tests selection files (the Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1` and the tests). Any other Claude/Anthropic model, or that model anywhere else, still fails the test. Opus only proposes a selection; trusted code validates it, and it can only add tests, never remove the other model's picks. **Security review notes:** the new Opus workflow uses only existing secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` / `AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are pinned by SHA. ### Validation **End to end in a fork (kubaflo/maui)** through the real GPT agent → Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs succeeded; each PR got the comment with both models' picks and the Skipped checks. Azure is skipped there because the fork has no federated identity. **Real builds on dotnet/maui.** Other PRs' merge refs don't have the new pipeline parameters until this merges, so each trial was a temporary draft PR: this branch plus one current PR's change. `Invoke-RunTests.ps1` ran locally with both models' proposals from the fork run on the same diff, queued `maui-pr-uitests`, and posted the comment. Every build got exactly the selected platforms and categories. | Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result | | --- | --- | --- | --- | --- | --- | | dotnet#40442 | dotnet#40369 Windows title bar | `Window` | + `TitleView`, `Lifecycle` | Windows, 3 categories | Passed in 39 min | | dotnet#40443 | dotnet#40366 Shell modal stack | `Shell1/2` on iOS, Mac | same | iOS + MacCatalyst | Passed | | dotnet#40444 | dotnet#40363 Android EmptyView | CollectionView + CarouselView | same | Android, 8 categories | Passed in 41 min | | dotnet#40445 | dotnet#40344 Shell tab reselect | `Shell1/2`, `Navigation` | + `TabbedPage` | All platforms, 4 categories | Running at time of writing | | dotnet#40446 | dotnet#40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet | same | All platforms, 3 categories | 1 test failed on MacCatalyst | | dotnet#40447 | dotnet#40234 `Element` / VSM threading | Full suite | Full suite | Full suite | Running at time of writing | | dotnet#40448 | dotnet#40385 CodeQL CI-only | Nothing | Nothing | Nothing (no build) | Comment only | | dotnet#40449 | dotnet#39487 Button corner radius | `Button`, `ImageButton` on iOS, Mac | same | iOS + MacCatalyst | 1 Button test failed | The failures are the same tests that failed in an earlier round on the same copied changes, so they come from those PRs, not the selection. The comments were refreshed from the real timelines and test results with `Update-RunTestsResults.ps1`. **Azure previews of this PR's merge ref:** - Defaults: identical to `main` (13 jobs, 157 matrix entries). - Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`: the Material3 lane is included. - Qualified selections: `android:Material3` + `ios:Label` runs only the Material3 lane on Android; `android:Shell1` also runs the Shell renderer lane; `android:SafeAreaEdges` runs its dedicated lane; `windows:CollectionView1` runs the CV1 lane; an Android-only CollectionView pick never starts the Windows CV1 lane. - `selectedPlatforms` containing `"x; Remove-Item"`: only template booleans reach the build step. **Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the inventory and `#if` evaluator, pruning and qualification, the two-model union (one model empty, neither, full only when chosen), AI-only rules, the queue request and dropped-selection guard, the check plan, comment rendering and updates, and static template checks. The `ReviewTrigger`, `Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw compile --strict` succeeds. **Not verified before merge:** - The slash command and the dispatcher's OIDC exchange on `dotnet/maui` itself. GitHub only runs them from the default branch. After merge, comment `/review run-tests` on a PR to confirm. - The `check_run`-triggered comment update on `dotnet/maui` (the script itself is verified; the half-hourly sweep is the backup). - The `copilot-pat-pool` token selection in the new Opus workflow (the fork used a single secret). --------- Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Target Branch
mainnet11, for reviewThis is CI and workflow infrastructure, so it targets
main. The slash command can only activate after the workflow is on the default branch.For .NET 11 fixes, only a maintainer retargets the approved PR to the active
inflight lane and merges it after CI validates that final base. Do not merge
individual fixes directly into
net11; agents must not retarget or merge.Description of Change
Adds a
/review run-testsPR command that runs only the UI and device tests affected by the PR, instead of every test like/azp rundoes.copilot-review-run-tests.md) reads the frozen PR diff and proposes platforms, UI categories, device projects and device categories. Trusted code (Select-PRTests.ps1) checks the proposal against the real test inventories and combines it with a conservative coverage floor. For example, Android-only source selects Android, and.ios.csselects iOS and MacCatalyst. Shared, SDK, build or unmapped changes fall back to full coverage, and the AI can't drop coverage that the floor requires.Queue-SelectedPRTests.ps1) rechecks the caller and the PR head, base and test-merge identity, then queues the runs:maui-prruns unchanged.maui-pr-uitestsandmaui-pr-devicetestsrun with the selection as runtime parameters.selectedPlatforms,selectedCategories,selectedRun.selectedPlatforms,selectedProjects,selectedCategories.main. The device pipeline adds one validation stage.verify-results.py, Windows device runner)./azp run,/review,/review gateand/review testsbehave as before.review-trigger.ymland scheduled recovery exclude the new subcommand.Validation
These were live Azure DevOps runs, queued with the selection parameters at this branch's SHAs:
ButtonButtonCore.DeviceTests-ButtonandControls.DeviceTests-Buttonwork items: 64 tests on CoreCLR and 64 on Mono. iOS and MacCatalyst stages skipped.ButtonButtonControls.DeviceTests-{packaged,unpackaged}-Button, 16 tests. Every other platform stage skipped.Other checks:
main's stages, jobs and matrix keys.eng/Versions.propsand shared Windows handlers, was widened to all platforms and all device projects.gh aw compile(pinned v0.86.2, strict) passed, and the PowerShell, YAML, XML and Python syntax checks passed.Real-PR trials. The selector ran on six open PRs (local Gather, then a gpt-6.1-sol proposal, then trusted Validate). Two were also run live, each on a temporary branch combining this branch with the PR's commits.
Shell1,Shell2,Lifecycle; device ControlsShell,Memory,Lifecyclemain's Windows device run executes 2,263.For #39655, every UI failure except one also fails in the last three
mainruns. The exception is the PR's own new test,Issue29264.RepeatedRotationsPreserveSelectedRecipe(3,*), which failed on iOS 18.5 and latest. So the targeted run surfaced real PR feedback.Fixes found during these trials:
merge_commit_shaon newer base commits, so the identity check now freezes the merge's first parent (which must descend frombase.sha) instead ofbase.shaitself. PRs whose test-merge sits on an orphaned base fail closed, with an actionable "push or rebase" message.[ShardedTestCategory]attributes.#ifguards in shared UI tests, evaluated against each UI project'sDefineConstants; it fails closed on#else/#elifor unknown symbols.TestCases.HostApp/Issuespages, mapped to their paired test.Not yet verified:
/review run-testson a test PR to confirm it end to end.Issues Fixed
Fixes #