Skip to content

[CI] Add /review run-tests to run only affected UI and device tests - #40253

Closed
kubaflo wants to merge 18 commits into
mainfrom
kubaflo-targeted-review-tests
Closed

kubaflo wants to merge 18 commits into
mainfrom
kubaflo-targeted-review-tests

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Target Branch

This PR is for Initial base
New features, API changes, or .NET 12 work (including .NET 12 fixes) main
.NET 11 servicing fixes net11, for review
Older servicing, release integration, infrastructure, or documentation The branch designated for that work

This is CI and workflow infrastructure, so it targets main. The slash command can only activate after the workflow is on the default branch.

For .NET 11 fixes, only a maintainer retargets the approved PR to the active
inflight lane and merges it after CI validates that final base. Do not merge
individual fixes directly into net11; agents must not retarget or merge.

Description of Change

Adds a /review run-tests PR command that runs only the UI and device tests affected by the PR, instead of every test like /azp run does.

  • Selection: A read-only GPT agent (copilot-review-run-tests.md) reads the frozen PR diff and proposes platforms, UI categories, device projects and device categories. Trusted code (Select-PRTests.ps1) checks the proposal against the real test inventories and combines it with a conservative coverage floor. For example, Android-only source selects Android, and .ios.cs selects iOS and MacCatalyst. Shared, SDK, build or unmapped changes fall back to full coverage, and the AI can't drop coverage that the floor requires.
  • Dispatch: An isolated safe-output job (Queue-SelectedPRTests.ps1) rechecks the caller and the PR head, base and test-merge identity, then queues the runs:
    • maui-pr runs unchanged.
    • maui-pr-uitests and maui-pr-devicetests run with the selection as runtime parameters.
    • It confirms Azure applied those parameters, cancelling any run that ignored them, and posts a summary to the PR.
  • Pipelines: New optional parameters, all backward compatible:
    • UI: selectedPlatforms, selectedCategories, selectedRun.
    • Device: selectedPlatforms, selectedProjects, selectedCategories.
    • With no parameters, a preview of both pipelines expands to the same stages, jobs and matrix keys as main. The device pipeline adds one validation stage.
  • Zero-test guards: A selected run fails if it executes zero tests (UI step, Helix verify-results.py, Windows device runner).
  • Unchanged commands: /azp run, /review, /review gate and /review tests behave as before. review-trigger.yml and scheduled recovery exclude the new subcommand.

Validation

These were live Azure DevOps runs, queued with the selection parameters at this branch's SHAs:

Selection Build Result What ran
Android, UI Button maui-pr-uitests 1681287 succeeded Only the Android Button lane (107 tests: 105 passed, 2 not executed) and the Material3 lane (433 tests). No iOS, MacCatalyst or Windows stages.
Android, device Core and Controls Button maui-pr-devicetests 1681288 succeeded Only Core.DeviceTests-Button and Controls.DeviceTests-Button work items: 64 tests on CoreCLR and 64 on Mono. iOS and MacCatalyst stages skipped.
Windows, UI Button maui-pr-uitests 1681307 succeeded Only Windows stages: 1 matrix job instead of 29, 104 tests. The guard logged "Selected UI tests executed 103 tests".
Windows, device Controls Button maui-pr-devicetests 1681308 succeeded Only Controls.DeviceTests-{packaged,unpackaged}-Button, 16 tests. Every other platform stage skipped.

Other checks:

  • Default coverage: Azure previews of both pipelines with no parameters match main's stages, jobs and matrix keys.
  • Fallback: An over-narrow "Android Button" proposal for PR [Windows] Restore retained wrapper state and shadow visibility on handler reconnect #40156, which touches eng/Versions.props and shared Windows handlers, was widened to all platforms and all device projects.
  • Rejected input: Malformed filter expressions in a proposal are rejected.
  • Compilation: gh aw compile (pinned v0.86.2, strict) passed, and the PowerShell, YAML, XML and Python syntax checks passed.

Real-PR trials. The selector ran on six open PRs (local Gather, then a gpt-6.1-sol proposal, then trusted Validate). Two were also run live, each on a temporary branch combining this branch with the PR's commits.

PR Selected Build Result What ran
#39591 (Windows Shell) Windows; UI Shell1, Shell2, Lifecycle; device Controls Shell, Memory, Lifecycle uitests 1681371, devicetests 1681372 both succeeded 3 UI jobs (360 tests). The device run executed 179 tests (Shell 144, Memory 24, Lifecycle 10); main's Windows device run executes 2,263.
#39655 (iOS CarouselView) iOS and MacCatalyst; UI CarouselView, CollectionView1–7, Layout, Lifecycle; device Controls with the same categories uitests 1681373, devicetests 1681374 UI failed, device succeeded 30 UI runs (2,234 tests). Device ran 69 tests per leg on iOS and MacCatalyst (CoreCLR and Mono). No Android or Windows stages.

For #39655, every UI failure except one also fails in the last three main runs. The exception is the PR's own new test, Issue29264.RepeatedRotationsPreserveSelectedRecipe(3,*), which failed on iOS 18.5 and latest. So the targeted run surfaced real PR feedback.

Fixes found during these trials:

  • Test-merge base. GitHub rebuilds merge_commit_sha on newer base commits, so the identity check now freezes the merge's first parent (which must descend from base.sha) instead of base.sha itself. PRs whose test-merge sits on an orphaned base fail closed, with an actionable "push or rebase" message.
  • Mapping precision. The selector now handles:
    • [ShardedTestCategory] attributes.
    • File-wide #if guards in shared UI tests, evaluated against each UI project's DefineConstants; it fails closed on #else/#elif or unknown symbols.
    • TestCases.HostApp/Issues pages, mapped to their paired test.
    • Snapshot PNGs, mapped to the test method that produces them.
    • Unit-test paths, which no longer add platforms.
  • Feature areas. Shell, CollectionView/CarouselView (Items, Items2), NavigationPage, TabbedPage and FlyoutPage now map to their trusted categories. Before, these were treated as shared code and forced full coverage, discarding category narrowing.

Not yet verified:

  • The hosted slash-command path (GPT analysis, then OIDC dispatch). GitHub won't dispatch a workflow that isn't on the default branch yet. After merge, comment /review run-tests on a test PR to confirm it end to end.
  • The Material3 Android lane still runs its full suite whenever Android is selected, on purpose (conservative).
  • Selected mobile device builds still build the shared multi-platform apps; only test execution is narrowed.

Issues Fixed

Fixes #

kubaflo and others added 4 commits November 14, 2026 13:08
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Serialize runtime selections as JSON strings, confirm effective coverage and revision, and cancel unexpected runs before reporting them.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Broaden missing runtime patches and execution script changes, and match complete platform control names to avoid Image/ImageButton prefix collisions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:43 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40253

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40253"

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:43 — with GitHub Actions Inactive
@azure-pipelines

This comment has been minimized.

@kubaflo

This comment has been minimized.

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:45 — with GitHub Actions Inactive
@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 14, 2026
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:47 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:48 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 14, 2026 13:49 — with GitHub Actions Inactive
@MauiBot

This comment has been minimized.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 14, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 6 findings

Reviewed commit a560c93.

See inline comments for details.

2 findings could not be attached inline; retained in run artifacts.

Get-Content $PSCommandPath
$buildCommand = "./build.ps1 --target=uitests-apphost --configuration=${{ parameters.configuration }} --${{ parameters.platform }} --verbosity=diagnostic --usenuget=false --runtimevariant=${{ parameters.runtimeVariant }}"
$platformArguments = "--${{ parameters.platform }}"
if ("${{ join(',', parameters.platforms) }}" -ne "") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Input and Path Correctness - The ci-uitests.yml selectedPlatforms object parameter flows through ui-tests.yml into this double-quoted PowerShell source without element validation. A direct pipeline request containing android and an element with a PowerShell subexpression evaluates that expression here, before the Where-Object allowlist runs, rather than treating it as platform data. The hosted selector's whitelist does not cover direct pipeline requests. Transport the array as JSON through an environment variable and validate identifiers before constructing process arguments.

schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber
headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha
platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device
runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness - A Windows-only WebViewHandler.Windows.cs change selects platforms=[windows] and uiCategories=[WebView], making runUi true. However, common/ui-tests.yml explicitly excludes Windows WebView, and its stage gates remove every stage for this combination. The dispatcher consequently submits a UI pipeline with no stages instead of omitting unsupported UI coverage, and the request fails. Resolve platform/category support before deciding to queue the UI pipeline.

CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:
${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness - Accepted inventory identifiers do not all map to categoryGroupsToTest. For example, changing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests accepts that value, but no category group contains it, so this exact-match gate and its Apple/Windows equivalents emit no matching selected lane. Map every accepted selection to a nonempty matrix or widen unsupported selections before dispatch.

${{ categoryGroup }}:
CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - The new per-category matrix needs a platform-empty regression case. For a PR editing only shared Tests/Issues/Issue32989.cs, the no-AI fallback selects all four platforms and Essentials. The three Essentials fixtures are compiled only under ANDROID or WINDOWS, so the isolated iOS and MacCatalyst Essentials jobs execute zero tests and fail the new nonzero guard. Previously Essentials shared a group with categories that execute on Apple. Exclude unsupported pairs or widen their categories, and cover this case in a selector/matrix regression test.

kubaflo and others added 3 commits November 14, 2026 15:11
pulls.base.sha is the base tip at the PR's last sync, while GitHub rebuilds
merge_commit_sha on the current base tip. Require the merge's first parent
to descend from the PR base instead of equal it, and freeze that parent as
the selection base. Reject merges built on base commits outside the base
history with an actionable message.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Real PRs widened to every platform because shared UI tests wrapped in a
file-wide #if, HostApp issue pages, and screenshot baselines were treated as
unresolved. Evaluate file-wide guards against each UI test project's
DefineConstants (failing closed on #else, trailing code or unknown symbols),
map issue pages and snapshot baselines to their owning UI test, and
recognize [ShardedTestCategory]. Unit-test-only paths no longer add
platforms because they run in maui-pr.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Feature-area runtime changes previously made the floor broad, discarding
category narrowing. Shell, CollectionView/CarouselView, NavigationPage,
TabbedPage and FlyoutPage now add their inventory-validated UI and device
categories instead.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kubaflo

kubaflo commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

/rebase

@kubaflo

This comment has been minimized.

@azure-pipelines

This comment has been minimized.

@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 14, 2026
@MauiBot

This comment has been minimized.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 14, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 5 findings

Reviewed commit 406cb26.

See inline comments for details.

Previously reported findings were not duplicated.

1 findings could not be attached inline; retained in run artifacts.

schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber
headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha
platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device
runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], so runUi is true even without AI output. Queue-SelectedPRTests.ps1 consequently submits the UI pipeline, but common/ui-tests.yml excludes Windows WebView and removes every stage for this combination. The UI request has no runnable stages instead of retaining only supported device/build coverage. Resolve supported platform/category pairs before deciding to dispatch, and cover this negative case in a selector-to-pipeline regression test.

CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:
${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - Editing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests.ps1 accepts that declared inventory value, but categoryGroupsToTest contains no such category; this gate and its Apple/Windows equivalents emit no matrix entry for it. The selected run therefore cannot execute the changed fixture. Map accepted inventory values to runnable lanes or widen selections with no matching group, and add a discriminating regression case for an accepted category absent from the matrix.

${{ categoryGroup }}:
CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - File-wide guard detection fixes Issue32989.cs alone, but not a PR that also edits shared Tests/ButtonUITests.cs. The selector unions all four platforms with Button and Essentials; this loop and the MacCatalyst equivalent then emit standalone Essentials jobs on Apple. The only three Essentials fixtures are WINDOWS-only Issue32989 and ANDROID-only Issue36523/Issue37638, so those jobs execute zero tests and fail requireExecutedTests. Preserve platform/category associations or widen unsupported pairs, and add a multi-file selector-to-matrix regression test.

@kubaflo

This comment has been minimized.

@github-actions

This comment has been minimized.

@MauiBot

This comment has been minimized.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 15, 2026
@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 6 findings

Reviewed commit e1bf4fd.

See inline comments for details.

Previously reported findings were not duplicated.

Comment thread eng/helix_xharness.proj
<AndroidPackageName>%(_SelectedApk.PackageName)</AndroidPackageName>
<WorkItemTimeout>02:00:00</WorkItemTimeout>
<TestTimeout>01:00:00</TestTimeout>
<CustomCommands>xharness android test --app "$app" --package-name "$package_name" --output-directory "$output_directory" --timeout "$timeout" --expected-exit-code "$expected_exit_code" --arg "TestFilter=Category=%(_SelectedApk.Category)" &amp;&amp; "$HELIX_PYTHONPATH" "$HELIX_CORRELATION_PAYLOAD/device-selection/verify-results.py" "$output_directory"</CustomCommands>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - Splitting mobile coverage into single-category exclusion filters drops tests carrying multiple categories, even when every category is selected. Controls ButtonTests has class category Button and NestedButtonHasExpectedIconPosition has method category Layout. DeviceTestSharedHelpers.GetExcludedTestCategories turns Category=Button into exclusions including Layout, and Category=Layout into exclusions including Button; MauiProgram passes these exclusions to the headless runner. Neither selected Android work item executes this test, although an unrestricted run does. Preserve inclusive category matching or widen such selections rather than treating separate exclusion-filter runs as their union.

schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber
headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha
platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device
runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], so runUi is true even without AI output. Queue-SelectedPRTests.ps1 submits the UI pipeline, but common/ui-tests.yml excludes Windows WebView and removes every stage for this combination. The UI request has no runnable stages instead of retaining only supported device/build coverage. Resolve supported platform/category pairs before deciding to dispatch, and cover this negative case in a selector-to-pipeline regression test.

CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:
${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - Editing Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests.ps1 accepts that declared inventory value, but categoryGroupsToTest contains no such category; this gate and its Apple/Windows equivalents emit no matrix entry for it. The selected run cannot execute the changed fixture. Map accepted inventory values to runnable lanes or widen selections with no matching group, and add a regression case for an accepted category absent from the matrix.

${{ categoryGroup }}:
CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - File-wide guard detection handles Issue32989.cs alone, but not a PR that also edits shared Tests/ButtonUITests.cs. Without AI output, the selector unions all four platforms with Button and Essentials; this loop and the MacCatalyst equivalent emit standalone Essentials jobs on Apple. The only three Essentials fixtures are WINDOWS-only Issue32989 and ANDROID-only Issue36523/Issue37638, so those jobs execute zero tests and fail requireExecutedTests. Preserve platform/category associations or widen unsupported pairs, and add a multi-file selector-to-matrix regression test.

selection_ready: ${{ steps.context.outputs.ready }}
selection-fallback:
needs: [pre_activation, activation, agent]
if: ${{ !cancelled() && needs.pre_activation.outputs.selection_ready == 'true' && needs.agent.result == 'failure' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - The fallback and normal dispatch paths are not mutually exclusive when the agent fails after emitting a proposal. The compiled queue_selected_tests condition accepts any non-skipped agent result with that output type, while this fallback also runs for failure and invokes the dispatcher without AgentOutputPath. An emitted proposal adding Layout to a Button floor followed by an agent/postprocessing failure therefore queues both the floor-only selection and the expanded selection. Find-SelectedRun only reuses exact parameter matches, so it does not suppress these overlapping UI/device runs. Select one dispatch path for a failed agent with collected output.

@MauiBot

This comment has been minimized.

@kubaflo

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Tests Failure Analysis

@kubaflo — test-failure analysis for commit e1bf4fd.

Scope CI failures Commit e1bf4fd


🧪 CI Analysis — click to expand
📊 maui-pr
  • 🟡 Needs human investigation — iOS ARM64 RunOniOS_BlazorDebug_CoreCLR: project failed to run; inspect launch attachments. All five target samples have truncated timelines, not equivalent passing-test evidence.

🧪 maui-pr-devicetests

🧪 maui-pr-uitests

Matched failures predate these changes on main; default shards/test code remain unchanged.


🧭 Follow-up — actions and refresh

Next action: Restore missing diagnostics/results and inspect unresolved launch, setup, and image failures before attributing them to this PR.

Maintainers: comment /review tests to refresh this report.

@kubaflo kubaflo mentioned this pull request Nov 15, 2026
1 task done
@kubaflo

kubaflo commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Selected PR Tests

Note

Preview of /review run-tests output for a live trial on #39591 (Windows-only Shell change). Selection came from the real selector; runs were queued from this branch. Posted here because the hosted workflow can only comment once it is on main.

@FullDevStudio — targeted test selection for commit bbb4ac4.

Scope targeted tests Commit bbb4ac4

🎯 Detected coverage

  • Platforms: Windows.
  • maui-pr: full build, unit and integration validation is unchanged. not queued.
  • UI tests: Lifecycle, Shell, Shell1, Shell2 on selected platforms. queued.
  • Device tests: Controls.DeviceTests / Lifecycle, Memory, Shell on selected platforms. queued.

✅ Results

Tests Passed

Pipeline Run Status Passed Failed Not run
maui-pr-uitests 1681371 ✅ succeeded 344 0 16
maui-pr-devicetests 1681372 ✅ succeeded 179 0 0

🧪 Selection notes — click to expand

Selected runs are not full-suite certification; queued does not mean passed.

Coverage notes:

  • AI proposal validated and unioned with mandatory coverage; confidence is not a coverage guarantee.

Generated by review run-tests.

@kubaflo

kubaflo commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Selected PR Tests

Note

Preview of /review run-tests output for a live trial on #39655 (iOS/MacCatalyst CarouselView change). Selection came from the real selector; runs were queued from this branch. Posted here because the hosted workflow can only comment once it is on main.

@Dhivya-SF4094 — targeted test selection for commit f60de84.

Scope targeted tests Commit f60de84

🎯 Detected coverage

  • Platforms: iOS, MacCatalyst.
  • maui-pr: full build, unit and integration validation is unchanged. not queued.
  • UI tests: CarouselView, CollectionView, Layout, Lifecycle on selected platforms. queued.
  • Device tests: Controls.DeviceTests / CarouselView, CollectionView, Layout, Lifecycle on selected platforms. queued.

❌ Results

Tests Failed

Pipeline Run Status Passed Failed Not run
maui-pr-uitests 1681373 ❌ failed 2183 28 23
maui-pr-devicetests 1681374 ✅ succeeded 384 0 30
Failed tests (28)
  • Microsoft.Maui.TestCases.Tests.Issues.Issue38730(Mac).VerifyVerticalItemSpacingPageRendering — CollectionView1_maccatalyst_ui_tests_coreclr_controls
  • Microsoft.Maui.TestCases.Tests.Issues.Issue7678_1(Mac).VerifyCarouselViewBindingAndRendering — CarouselView_maccatalyst_ui_tests_coreclr_controls
  • Microsoft.Maui.TestCases.Tests.Issues.Issue7678(Mac).VerifyCarouselViewItemsRenderAfterBinding — CarouselView_maccatalyst_ui_tests_coreclr_controls
  • Microsoft.Maui.TestCases.Tests.Issues.Issue17664(iOS).VerifyGroupedCollectionViewVisibleItemIndices — CollectionView4_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue17664(iOS).VerifyGroupedCollectionViewVisibleItemIndices — CollectionView4_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue31897(iOS).EnsureCollectionViewLayoutOnItemsSourceChange — CollectionView2_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue19609(iOS).CanTapButtonOnEmptyView — CollectionView2_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue31897(iOS).EnsureCollectionViewLayoutOnItemsSourceChange — CollectionView2_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue19609(iOS).CanTapButtonOnEmptyView — CollectionView2_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue31680(iOS).CollectionViewInsideCarouselViewShouldNotThrowIndexOutOfRangeException — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue29264(iOS).RepeatedRotationsPreserveSelectedRecipe(3,True) — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue29264(iOS).RepeatedRotationsPreserveSelectedRecipe(3,False) — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.CarouselViewPositionVisibility(iOS).Issue12848Test — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue22417(iOS).AddItemsToCarouselViewWorks — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue27418(iOS).CarouselItemsShouldRenderProperly — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.CarouselViewAdjustPeekAreaInsets(iOS).ChangePeekAreaInsetsInOnSizeAllocatedTest — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue7678(iOS).VerifyCarouselViewItemsRenderAfterBinding — CarouselView_ios_ui_tests_coreclr_controls_latest
  • Microsoft.Maui.TestCases.Tests.Issues.Issue22417(iOS).AddItemsToCarouselViewWorks — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.CarouselViewAdjustPeekAreaInsets(iOS).ChangePeekAreaInsetsInOnSizeAllocatedTest — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue7678_1(iOS).VerifyCarouselViewBindingAndRendering — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.CarouselViewPositionVisibility(iOS).Issue12848Test — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.CarouselViewSetOrientation(iOS).RotatingCarouselViewHTMLShouldNotDisappear — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue27418(iOS).CarouselItemsShouldRenderProperly — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue29264(iOS).RepeatedRotationsPreserveSelectedRecipe(3,False) — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • Microsoft.Maui.TestCases.Tests.Issues.Issue7678(iOS).VerifyCarouselViewItemsRenderAfterBinding — CarouselView_ios_ui_tests_coreclr_controls_18_5
  • …and 3 more; see the run links above.

Comment /review tests to compare these failures with recent main runs.


🧪 Selection notes — click to expand

Selected runs are not full-suite certification; queued does not mean passed.

Coverage notes:

  • AI proposal validated and unioned with mandatory coverage; confidence is not a coverage guarantee.

Generated by review run-tests.

Embed queued Azure build IDs in the /review run-tests comment and add a
scheduled updater that edits its results section with run status,
pass/fail/not-run totals and failing tests verified through per-run
result details.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 16, 2026
@kubaflo

kubaflo commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

/rebase

@github-actions

Copy link
Copy Markdown
Contributor

Unable to rebase this branch automatically. It may have conflicts with its base branch, or the PR author has not allowed maintainer edits. Please rebase manually.

@MauiBot

MauiBot commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@kubaflo — existing-test verification for commit a00e2dce.

Result SKIPPED Platform Android Commit a00e2dce


🧪 Gate analysis — click to expand

⚠ Skipped. No runnable PR tests were selected. The fix has not been verified by this gate.

No interpretable test-group summary was produced. No additional test execution is inferred.

PR head: a00e2dce.

Without-fix baseline: 14c3ea20.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 7 findings

Reviewed commit a00e2dc.

See inline comments for details.

Previously reported findings were not duplicated.

Get-Content $PSCommandPath
$buildCommand = "./build.ps1 --target=uitests-apphost --configuration=${{ parameters.configuration }} --${{ parameters.platform }} --verbosity=diagnostic --usenuget=false --runtimevariant=${{ parameters.runtimeVariant }}"
$platformArguments = "--${{ parameters.platform }}"
if ("${{ join(',', parameters.platforms) }}" -ne "") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Input and Path Correctness - The caller-supplied ci-uitests.yml selectedPlatforms object flows through common/ui-tests.yml into this double-quoted PowerShell source without element validation. A direct pipeline request containing android and an element with a PowerShell subexpression evaluates that expression here, before the Where-Object allowlist runs, instead of treating it as platform data. The hosted selector's whitelist does not cover direct pipeline requests. Pass the array as JSON through an environment variable and validate identifiers before constructing process arguments.

Comment thread eng/helix_xharness.proj
<AndroidPackageName>%(_SelectedApk.PackageName)</AndroidPackageName>
<WorkItemTimeout>02:00:00</WorkItemTimeout>
<TestTimeout>01:00:00</TestTimeout>
<CustomCommands>xharness android test --app "$app" --package-name "$package_name" --output-directory "$output_directory" --timeout "$timeout" --expected-exit-code "$expected_exit_code" --arg "TestFilter=Category=%(_SelectedApk.Category)" &amp;&amp; "$HELIX_PYTHONPATH" "$HELIX_CORRELATION_PAYLOAD/device-selection/verify-results.py" "$output_directory"</CustomCommands>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - Separate single-category mobile work items are not an inclusive union. Controls ButtonTests has class category Button, while NestedButtonHasExpectedIconPosition also has method category Layout. DeviceTestSharedHelpers.GetExcludedTestCategories converts Category=Button into exclusions including Layout, and Category=Layout into exclusions including Button; MauiProgram passes these exclusions to the headless runner. Neither selected Android work item executes this test, even when both categories are selected, although an unrestricted run does. Other tests satisfy the nonzero-results guard and hide the omission. Preserve inclusive matching or widen such selections, and add a dual-category coverage regression case.

schemaVersion = 1; repository = $context.repository; prNumber = $context.prNumber
headSha = $context.headSha; baseSha = $context.baseSha; mergeSha = $context.mergeSha
platforms = $platforms; uiCategories = $ui; deviceProjects = $projects; deviceCategories = $device
runUi = ($platforms.Count -gt 0 -and ($broad -or $ui.Count -gt 0))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - A Windows-only src/Core/src/Handlers/WebView/WebViewHandler.Windows.cs edit produces platforms=[windows] and uiCategories=[WebView], making runUi true even without AI output. Queue-SelectedPRTests submits the UI pipeline, but common/ui-tests.yml explicitly excludes Windows WebView and removes every stage for this combination. The UI dispatch has no runnable stages instead of retaining only supported device/build coverage. Resolve platform/category support before deciding to queue UI coverage.

CATEGORYGROUP: ${{ categoryGroup }}
${{ else }}:
${{ each category in split(categoryGroup, ',') }}:
${{ if and(ne(category, 'SafeAreaEdges'), or(containsValue(parameters.selectedCategories, category), and(startsWith(category, 'CollectionView'), containsValue(parameters.selectedCategories, 'CollectionView')), and(startsWith(category, 'Shell'), containsValue(parameters.selectedCategories, 'Shell')))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - Accepted inventory values do not all map to this matrix. Editing shared Tests/Issues/Issue8167.cs selects MovedFromCompatibility via UITestCategories.Compatibility. Select-PRTests accepts that declared value, but categoryGroupsToTest contains no MovedFromCompatibility entry; this gate and its Apple/Windows equivalents produce no matching selected lane. The selected run cannot execute the changed fixture. Map every accepted category to a runnable lane or widen selections with no matching group.

selection_ready: ${{ steps.context.outputs.ready }}
selection-fallback:
needs: [pre_activation, activation, agent]
if: ${{ !cancelled() && needs.pre_activation.outputs.selection_ready == 'true' && needs.agent.result == 'failure' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - The fallback and normal dispatch paths can both run when the agent fails after emitting a proposal. The compiled queue_selected_tests condition accepts any non-skipped agent result with that output type, while this fallback also runs for failure and invokes the dispatcher without AgentOutputPath. A proposal adding Layout to a Button floor followed by an agent/postprocessing failure therefore queues both the floor-only and expanded UI/device selections. Find-SelectedRun requires exact parameter equivalence, so it cannot deduplicate these overlapping runs. Make the two dispatch paths mutually exclusive.

It 'replaces only the results block with verified Azure outcomes and marks completed runs final' {
$responses = @{
'https://dev.azure.com/dnceng-public/public/_apis/build/builds/22?api-version=7.1' = [pscustomobject]@{ status = 'completed'; result = 'failed'; definition = [pscustomobject]@{ id = 313 } }
'https://vstmr.dev.azure.com/dnceng-public/public/_apis/testresults/resultsummarybybuild?buildId=22&api-version=7.1-preview.1' = [pscustomobject]@{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - The only outcome-rendering fixture combines result=failed with two failed tests, so it cannot discriminate the formatter's independent failure signals. Removing the completed-build-result check would leave every new assertion satisfied, but a provisioning-failed build with zero published tests would receive a green Tests-Passed badge. Add separate fixtures for a failed build with zero test rows and a successful build with no failed tests, asserting both the failure-before-tests text and the non-failing badge case.

@kubaflo

kubaflo commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #40305, which reworks /review run-tests from main: AI-selected UI tests queued as a pull-request build so they show up in the PR's checks, plus Skipped checks for everything not selected.

@kubaflo kubaflo closed this Nov 16, 2026
kubaflo added a commit that referenced this pull request Nov 20, 2026
…#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces #40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from #38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| #40442 | #40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| #40443 | #40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| #40444 | #40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| #40445 | #40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| #40446 | #40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| #40447 | #40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| #40448 | #40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| #40449 | #39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
HarishKumarSF4517 pushed a commit to HarishKumarSF4517/maui that referenced this pull request Nov 20, 2026
…dotnet#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces dotnet#40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from dotnet#38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| dotnet#40442 | dotnet#40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| dotnet#40443 | dotnet#40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| dotnet#40444 | dotnet#40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| dotnet#40445 | dotnet#40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| dotnet#40446 | dotnet#40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| dotnet#40447 | dotnet#40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| dotnet#40448 | dotnet#40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| dotnet#40449 | dotnet#39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants