Skip to content

[run-tests trial 40291] [NET12] [Enhancement]Add optional initial-value preselection to DisplayPromptAsync - #40446

Closed
kubaflo wants to merge 16 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40291
Closed

kubaflo wants to merge 16 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40291

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Do not merge. Do not review. Temporary trial PR for /review run-tests (#40305). It will be closed once the trial finishes.

This branch is #40305 plus the change from #40291, so maui-pr-uitests here uses the new selection parameters. GPT-6.1 Sol and Claude Opus 5.5 each chose UI tests for the same diff; only the union of their choices runs.

Copilot CI added 16 commits November 16, 2026 21:39
The agent picks platforms and UI test categories from the PR diff. A trusted
safe-output job validates the choice against the pipeline inventory, queues
maui-pr-uitests as a pull-request build so Azure Pipelines reports native
checks on the PR head, publishes a Skipped check for every platform and
category that was not selected, and posts a collapsible summary comment that
is updated when the run finishes.
The agent reads the collected context and uses GitHub read tools; it never
needs PR code on disk.
The previous expression also enabled the zero-test guard in the default
full run, which failed every job whose test step did not produce results.
The dispatcher always sets selectedRun for targeted runs.
Untyped template parameters receive the boolean as the string 'False',
which is truthy in a bare ${{ if }}.
…nly selection

- Hand the AI proposal to review-run-tests-dispatch.yml. Jobs in the
  copilot-pat-pool environment get an OIDC subject the Azure DevOps
  federated credential does not trust; the dispatch workflow runs on main
  without an environment, like review-trigger.yml.
- Queue only after GitHub's test-merge commit is confirmed to merge the
  selected head, and cancel any build whose source version differs.
- Evaluate #if guards per platform to know which categories have enabled
  tests (including the PR's own changed tests). Drop categories and
  platforms that would run zero tests, and pass platform-qualified
  categories (ios:Window) when a category only runs on some platforms.
- Pass the sample-build platforms to PowerShell as template-evaluated
  booleans instead of interpolating parameter text.
- Teach the rebalance-ui-test-categories skill the typed parameter form.
- Add regression tests for the full-run guard, the qualified matrix
  filter, the build-script inputs and the skill.
An agent can emit its proposal and then exit non-zero; dispatching both
the proposal and the fallback would queue two runs for one command.
The run-tests comment now lists the failing tests with the platforms they
failed on, read from the public Azure DevOps test results API. When the
maui-pr-uitests check completes again (failed jobs re-run), finished comments
for that commit are refreshed too, and the list notes that it may include
failures from earlier attempts.
The workflow posts run-tests comments as its bot; -CommentAuthor lets a maintainer refresh comments from a manual Invoke-RunTests trial with the same script.
- `/review run-tests --all` (and the workflow_dispatch `all` input) skips the
  AI and queues the full suite.
- PRs into release/* and candidate/* always get the full suite.
- maui-pr-uitests only applies a selection to pull-request builds. Branch,
  scheduled and manual builds of main, release/*, candidate/* and the other
  CI branches always run everything, whatever they were queued with.
- Comments for a required full suite say so instead of calling it an AI
  selection, and skipped sections point at `--all`.
/azp run maui-pr-uitests already runs the full suite, so the flag only duplicated it. PRs into release/* and candidate/* still get the full suite without an AI selection.
…at they pick

- copilot-review-run-tests (GPT-6.1 Sol) collects the context and selects, then
  hands over to the new copilot-review-run-tests-opus (Claude Opus 5.5), which
  selects from the same context with the same shared instructions and
  dispatches review-run-tests-dispatch.yml with both runs.
- The dispatcher validates each proposal on its own and runs the union per
  platform. A test runs only if at least one model selected it.
- No full-suite fallbacks: release/candidate PRs go through the models like any
  other PR, a missing proposal from both models queues nothing, and unknown
  categories run nothing instead of everything.
- Nothing is added for the models: the Material3 lane only runs when Material3
  is selected (in the pipeline too), and Android is never added for it.
- Only the declared proposal fields are read from agent output, and each run may
  only provide its own model's proposal. A proposal that exists but cannot be
  downloaded fails the dispatch instead of being dropped.
- The comment shows what each model chose per platform and both rationales.
…ue preselection to DisplayPromptAsync

Copy of dotnet#40291 for a /review run-tests trial. Do not merge.
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 18:50 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40446

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40446"

@azure-pipelines

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: 53e2ae2.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

❌ Skill Validation Results — 53e2ae2 · [run-tests trial 40291] [NET12] [Enhancement]Add optional initial-value preselection to DisplayPromptAsync · 2026-11-19T18:51:15Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@kubaflo

kubaflo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Selected UI Tests

@kubaflo — AI-selected UI tests (GPT-6.1 Sol + Claude Opus 5.5) for commit 53e2ae20.

Result FAILED Platforms Android, iOS, MacCatalyst, Windows Categories 3 of 77 Commit 53e2ae20


🧪 Test selection — click to expand

❌ maui-pr-uitests #1690671 failed. See the failing checks below or the run for details.

GPT-6.1 Sol: Run prompt tests on all four platforms, plus alert and action-sheet coverage for the shared Apple presentation change.
Claude Opus 5.5: Adds preselectInitialValue option to DisplayPromptAsync across all platforms; runs DisplayPrompt, DisplayAlert and ActionSheet UI tests on all platforms.

🧪 Ran — 3 categories on Android, iOS, MacCatalyst, Windows

Platform Categories
Android ActionSheet, DisplayAlert, DisplayPrompt
iOS ActionSheet, DisplayAlert, DisplayPrompt
MacCatalyst ActionSheet, DisplayAlert, DisplayPrompt
Windows ActionSheet, DisplayAlert, DisplayPrompt
📊 Results — failed: 13 passed, 1 failed of 14 jobs
Job Result
Android UITests CoreClr Controls (API 30) ActionSheet ✅ succeeded
Android UITests CoreClr Controls (API 30) DisplayAlert ✅ succeeded
Android UITests CoreClr Controls (API 30) DisplayPrompt ✅ succeeded
Build UITests CoreCLR Sample App Build Sample App ✅ succeeded
Build UITests Windows Sample App Build Sample App (Windows) ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) ActionSheet ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) DisplayAlert ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) DisplayPrompt ✅ succeeded
MacCatalyst UITests CoreCLR Controls ActionSheet ✅ succeeded
MacCatalyst UITests CoreCLR Controls DisplayAlert ✅ succeeded
MacCatalyst UITests CoreCLR Controls DisplayPrompt ❌ failed
WinUI UITests Controls ActionSheet ✅ succeeded
WinUI UITests Controls DisplayAlert ✅ succeeded
WinUI UITests Controls DisplayPrompt ✅ succeeded

Failed tests (1)

Test Platforms
Issue4981.PromptInitialValueIsNotSelected MacCatalyst

Some jobs were re-run, so this list can include failures from earlier attempts that later passed.

⏭️ Skipped — 74 categories, 0 platforms

Categories not run: Accessibility, ActivityIndicator, Animation, AppLinks, Border, BoxView, Brush, Button, CarouselView, CollectionView1, CollectionView2, CollectionView3, CollectionView4, CollectionView5, CollectionView6, CollectionView7, Cells, CheckBox, ContextActions, CustomRenderers, DatePicker, Dispatcher, DragAndDrop, Entry, Editor, Effects, Essentials, FlyoutPage, Focus, Fonts, Frame, Gestures, GraphicsView, Image, ImageButton, IndicatorView, InputTransparent, IsEnabled, IsVisible, Label, Layout, Lifecycle, ManualReview, Maps, Navigation, Page, Performance, Picker, ProgressBar, RadioButton, RefreshView, SafeAreaEdges, Shadow, ScrollView, SearchBar, Shape, Slider, SoftInput, Stepper, Switch, SwipeView, Shell1, Shell2, TabbedPage, TableView, TimePicker, TitleView, ToolbarItem, Triggers, ViewBaseTests, VisualStateManager, Window, WebView, Material3

Each skipped platform and category is listed as a Skipped maui-pr-uitests (skipped) … check on this commit. Use /azp run maui-pr-uitests for the full suite.

🤖 Why these tests — AI rationale (GPT-6.1 Sol + Claude Opus 5.5)

Each model chose independently; every test that runs was selected by at least one of them.

Platform GPT-6.1 Sol Claude Opus 5.5
Android DisplayPrompt, DisplayAlert, ActionSheet DisplayPrompt, DisplayAlert, ActionSheet
iOS DisplayPrompt, DisplayAlert, ActionSheet DisplayPrompt, DisplayAlert, ActionSheet
MacCatalyst DisplayPrompt, DisplayAlert, ActionSheet DisplayPrompt, DisplayAlert, ActionSheet
Windows DisplayPrompt, DisplayAlert, ActionSheet DisplayPrompt, DisplayAlert, ActionSheet

GPT-6.1 Sol — Run prompt tests on all four platforms, plus alert and action-sheet coverage for the shared Apple presentation change.

  • Page/PromptArguments and native AlertManagers change prompts on all four platforms; Issue4981 adds runnable DisplayPrompt tests.
  • iOS PresentPopUp is shared with DisplayAlert/ActionSheet, including Catalyst.
  • No base infrastructure or control rendering changes: omit full run/Material3.

Claude Opus 5.5 — Adds preselectInitialValue option to DisplayPromptAsync across all platforms; runs DisplayPrompt, DisplayAlert and ActionSheet UI tests on all platforms.

  • New Issue4981 shared test is DisplayPrompt category → all 4 platforms.
  • Page.DisplayPromptAsync/PromptArguments (shared) plus AlertManager Android/Windows/iOS(+Catalyst) prompt changes → DisplayPrompt and DisplayAlert (existing prompt/alert dialog tests).
  • iOS/Catalyst PresentPopUp (shared by alerts and action sheets) gained an onPresented callback → ActionSheet on ios/maccatalyst (included on all for simplicity, cheap).
  • PublicAPI txt files don't affect UI behavior; no full run needed (no base VisualElement/handler changes).

Generated by /review run-tests. A targeted run is a fast signal, not full-suite certification.

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 18:55 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 18:56 — with GitHub Actions Inactive
@github-actions github-actions Bot added area-ai-agents Copilot CLI agents, agent skills, AI-assisted development platform/android platform/ios platform/macos macOS / Mac Catalyst platform/windows labels Nov 19, 2026
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 18:57 — with GitHub Actions Inactive
@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 19, 2026
@MauiBot

MauiBot commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@kubaflo — existing-test verification; Setup did not capture a tested commit.

Result INCONCLUSIVE Platform iOS


🧪 Gate analysis — click to expand

⚠ Inconclusive. Setup, build, or execution did not produce a conclusive result. This does not prove the fix is incorrect.

No interpretable test-group summary was produced. No additional test execution is inferred.

Setup did not capture a verified PR head; no commit is claimed as tested.

No immutable without-fix baseline was prepared.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 19, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 7 findings

Reviewed commit 53e2ae2.

See inline comments for details.

artifactName: 'uitest-snapshot-results-android-material3-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)'

# SafeAreaEdges tests on Android API 36 with Pixel 3 XL
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), containsValue(parameters.selectedCategories, 'SafeAreaEdges'))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Logic and Correctness Verification - Qualified selections lose dedicated Android coverage. For example, merging Android Button + SafeAreaEdges with iOS Button emits ["Button","android:SafeAreaEdges"]. This gate accepts only bare SafeAreaEdges, while the ordinary Android matrix explicitly excludes that category. Button can pass the executed-test guard without any selected notch-device tests running. The Shell-renderer gate and shard conditions similarly ignore android:Shell1/android:Shell2. Accept the qualified selectors in these dedicated lanes and add regression cases that exercise them; the existing YAML-text assertion only checks the ordinary matrices.

void ApplyInitialSelection()
{
var end = uiTextField.EndOfDocument;
var start = arguments.PreselectInitialValue ? uiTextField.BeginningOfDocument : end;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Backward Compatibility and Migration - The legacy DisplayPromptAsync overload now forwards false, so this new handler collapses MacCatalyst's native initial selection even when an existing application never opts into the new overload. For DisplayPromptAsync("Edit", "Value", initialValue: "1234"), typing a replacement now appends instead of replacing the selected value; with maxLength: 4 it is rejected altogether. Preserve the legacy overload's native behavior separately from an explicit preselectInitialValue: false request. Issue4981 exercises only the explicit true/false overload.

} | Select-Object -First 40)
foreach ($file in $testFiles) {
$null = $file.path -match '^src/Controls/tests/TestCases\.(\w+)\.Tests/'
$raw = & gh api -H 'Accept: application/vnd.github.raw' "repos/$Repository/contents/$($file.path)?ref=$($pr.head.sha)" 2>$null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Input and Path Correctness - The files API's filename reaches this contents URL without escaping its path segments. A changed test named src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue#4981.cs passes the filter, but # starts a URL fragment, truncating the requested path and removing the pinned ref from the HTTP request. The following continue silently drops the failed read from changedTestRunnable. If this test newly enables a category on a platform, Resolve-RunTestsSelection can then discard that requested coverage. URI-escape the filename segments before constructing the endpoint.


try
{
VerifyScreenshot(tolerance: 0.5, retryTimeout: TimeSpan.FromSeconds(2));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - Both new tests execute this screenshot assertion on MacCatalyst, but the snapshot contains neither mac/PromptInitialValueIsSelected.png nor mac/PromptInitialValueIsNotSelected.png. UITest selects the mac environment and VisualRegressionTester explicitly fails when its baseline is absent, so these tests cannot pass on the platform with the new selection-change workaround. The CoreCLR iOS 18.5 lane likewise resolves snapshots/ios, whereas only ios-26 baselines were added. Supply baselines for the enabled environments.

/// <param name="keyboard">The keyboard type to use for the user response.</param>
/// <param name="initialValue">A pre-defined response that will be displayed, and which can be edited by the user.</param>
/// <returns>A <see cref="Task"/> that displays a prompt display and returns the string value as entered by the user.</returns>
public Task<string> DisplayPromptAsync(string title, string message, bool preselectInitialValue, string accept = "OK", string cancel = "Cancel", string placeholder = null, int maxLength = -1, Keyboard keyboard = default(Keyboard), string initialValue = "")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Public API Surface Design - The bool alternative at parameter three makes the previously valid call page.DisplayPromptAsync("Title", "Message", default) ambiguous: the default literal converts to both the existing string accept parameter and the new bool parameter, with neither overload preferred. The new PromptArguments constructor has the same source-compatibility break for new PromptArguments("Title", "Message", default). Existing binaries retain their signatures, but these consumers cannot rebuild unchanged. Choose an overload shape that preserves resolution of existing calls; the named-argument tests do not exercise this case.

runtimeVariant: "CoreCLR"
useWindowsCV1: true
skipProvisioning: ${{ parameters.skipProvisioning }}
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), ne(replace(replace(join('', parameters.selectedCategories), 'Material3', ''), 'SafeAreaEdges', ''), ''))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Build and MSBuild - This gate examines categories selected anywhere rather than ordinary categories selected for Android. Merging an Android Material3-only proposal with an iOS Label-only proposal produces ["Material3","ios:Label"]. The gate enables android_ui_tests_coreclr, but its matrix has no matching Android category: Material3 is outside categoryGroupsToTest and ios:Label cannot match android:Label. This leaves an empty matrix with no CATEGORYGROUP for the test invocation, instead of running only the valid Material3 and iOS lanes. Gate the ordinary stage on a nonempty Android-specific ordinary selection and cover mixed dedicated/platform selections.

${{ if startsWith(categoryGroup, 'CollectionView') }}:
${{ categoryGroup }}:
CATEGORYGROUP: ${{ categoryGroup }}
${{ if or(eq(length(parameters.selectedCategories), 0), containsValue(parameters.selectedCategories, 'CollectionView'), containsValue(parameters.selectedCategories, categoryGroup)) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Build and MSBuild - The Windows CV1 matrix rejects platform-qualified shards emitted by Merge-RunTestsSelections. Selecting CollectionView1 on Windows and Label on iOS produces ["windows:CollectionView1","ios:Label"]. The CV1 stage gates accept the CollectionView substring, but this condition rejects every shard, leaving an empty matrix and no CATEGORYGROUP rather than executing the requested CV1 coverage. Accept Windows-qualified shards here and use the same Windows-specific selection in the stage gates, including when CollectionView is selected only for another platform.

@kubaflo kubaflo closed this Nov 19, 2026
@kubaflo
kubaflo deleted the run-tests-real-40291 branch November 19, 2026 20:42
kubaflo added a commit that referenced this pull request Nov 20, 2026
…#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces #40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from #38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| #40442 | #40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| #40443 | #40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| #40444 | #40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| #40445 | #40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| #40446 | #40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| #40447 | #40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| #40448 | #40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| #40449 | #39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>
HarishKumarSF4517 pushed a commit to HarishKumarSF4517/maui that referenced this pull request Nov 20, 2026
…dotnet#40305)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI and workflow infrastructure, so it targets `main`. The slash command
activates only once the workflow is on the default branch.

### Description of Change

Adds `/review run-tests`. Two AI models, **GPT-6.1 Sol** and **Claude
Opus 5.5**, each read the PR diff independently and pick the UI test
categories and platforms it can affect. `maui-pr-uitests` then runs
**only the union of their picks**. A CollectionView-only Android fix
runs the CollectionView shards on Android, not all 77 categories on 4
platforms.

Replaces dotnet#40253.

**When what runs**

| Trigger | What runs |
| --- | --- |
| `/review run-tests` on any PR (write access) | Only what the models
selected: the per-platform union of both picks |
| Both models choose the full suite (e.g. core `Element` changes) | Full
suite |
| Neither model produces a pick | Nothing is queued; the comment says so
|
| `/azp run maui-pr-uitests` | Full suite, unchanged |
| Branch, scheduled and manual builds (`main`, `release/*`,
`candidate/*`, …) | Full suite, unchanged. `ci-uitests.yml` only passes
a selection when `Build.SourceBranch` is `refs/pull/*`. |

Nothing is ever added that a model did not choose: no full-suite
fallback, unknown category names run nothing, Android and the Material3
lane run only when selected (also enforced in the pipeline template),
and only the five declared proposal fields are read.

**What the requester sees**

1. **Native checks in the PR's checks list.** `maui-pr-uitests` is
queued through the Build API as a *pull-request* build
(`reason=pullRequest` + `triggerInfo.pr.sourceSha`), so Azure Pipelines
reports the run and every job as checks on the PR head, exactly like
`/azp run`.
2. **Skipped checks for everything not selected.** One `maui-pr-uitests
(skipped) <Category>` check per category and one `maui-pr-uitests
(skipped) <Platform> UITests` check per platform, plus a
`maui-pr-uitests (test selection)` summary check.
3. **A summary comment in the PR Test Gate layout:** title, requester
line and Result / Platforms / Categories / Commit badges, with
everything else in one collapsed **Test selection** section: build
status, each model's summary, what ran, results with failed tests, the
skipped list, and **Why these tests** with a per-platform table of each
model's pick and both rationales. `review-run-tests-results.yml` updates
it in place when the `maui-pr-uitests` check completes, with a
half-hourly sweep as backup.

**How it works**

| Piece | Role |
| --- | --- |
| `copilot-review-run-tests.md` (gh-aw, GPT-6.1 Sol) | Pre-activation
authorizes an exact `/review run-tests` from a write+ user and collects
the diff, the category inventory from
`eng/pipelines/common/ui-tests.yml`, which categories have enabled tests
per platform, and which the PR's own tests enable. The read-only agent
(`checkout: false`) proposes platforms, categories, `full_run`, a
summary and a rationale. The safe-output job saves the proposal and
starts the Opus workflow. |
| `copilot-review-run-tests-opus.md` (gh-aw, Claude Opus 5.5, new) |
`workflow_dispatch` only. Verifies the source run (run-tests lock
workflow, default branch, same repo, under 3 hours old), reuses its
context artifact, and makes an independent proposal with the same shared
prompt (`shared/run-tests-selection.md`). Then it starts the dispatcher
with both run ids. If either agent fails, the chain still hands over. |
| `review-run-tests-dispatch.yml` | `workflow_dispatch` on `main`, with
**no environment**, so its OIDC subject is
`repo:dotnet/maui:ref:refs/heads/main` like `review-trigger.yml`.
Verifies both runs, downloads each proposal (a listed artifact that
fails to download fails the job), checks each proposal's model and
source run, and runs `Invoke-RunTests.ps1`. |
| `Invoke-RunTests.ps1` | Trusted. Validates each proposal against the
inventory, prunes each platform to categories with enabled tests there,
and merges the two per platform. When every selected platform runs the
same categories they are sent as plain names; otherwise every entry is
platform-qualified (`android:Button`, `ios:Label`), never a mix.
Re-checks the requester's permission, the PR head and its target branch,
waits for a verified test-merge commit, queues the pull-request build
and cancels it if Azure didn't apply the selection. Then publishes the
Skipped and selection checks and the comment. |
| `review-run-tests-results.yml` | Updates the comment when the
`azure-pipelines` `maui-pr-uitests` check completes; sweeps every 30
minutes as backup. |
| `eng/pipelines/**/ui-tests*.yml`, `ci-uitests.yml` | Optional
`selectedRun`, `selectedPlatforms`, `selectedCategories` parameters
filter stages and matrix entries at compile time. `CollectionView` and
`Shell` expand to their shards; `<platform>:<Category>` limits a
category to one platform, and every stage guard, including the dedicated
Shell renderer, SafeAreaEdges, Material3 and Windows CV1 lanes, handles
both the plain and the qualified form. The sample build gets
template-evaluated booleans, never parameter text. Defaults expand to
exactly `main`'s jobs. |
| `review-trigger.yml`, `ReviewCommandHelpers.ps1`, rebalance skill |
`/review run-tests` is excluded from `/review` and from recovery.
`rebalance-ui-test-categories` understands the typed
`categoryGroupsToTest` parameter. |

`/azp run`, `/review`, `/review gate` and `/review tests` are unchanged.
Device tests are out of scope.

**Model policy exception:** the repo's GPT-only automation policy
(`CopilotPipeline.Tests.ps1`, from dotnet#38040) gets one narrow exception:
exactly `claude-opus-5.5`, only in the run-tests selection files (the
Opus workflow and its lock file, the dispatcher, `RunTests.Common.ps1`
and the tests). Any other Claude/Anthropic model, or that model anywhere
else, still fails the test. Opus only proposes a selection; trusted code
validates it, and it can only add tests, never remove the other model's
picks.

**Security review notes:** the new Opus workflow uses only existing
secrets (the `COPILOT_PAT_0..9` pool via `copilot-pat-pool`); the
dispatcher uses the existing `AZDO_TRIGGER_TENANT_ID` /
`AZDO_TRIGGER_CLIENT_ID`. No new third-party actions. All actions are
pinned by SHA.

### Validation

**End to end in a fork (kubaflo/maui)** through the real GPT agent →
Opus agent → dispatcher path, on 8 current PRs. All 24 workflow runs
succeeded; each PR got the comment with both models' picks and the
Skipped checks. Azure is skipped there because the fork has no federated
identity.

**Real builds on dotnet/maui.** Other PRs' merge refs don't have the new
pipeline parameters until this merges, so each trial was a temporary
draft PR: this branch plus one current PR's change.
`Invoke-RunTests.ps1` ran locally with both models' proposals from the
fork run on the same diff, queued `maui-pr-uitests`, and posted the
comment. Every build got exactly the selected platforms and categories.

| Trial | Copy of | GPT-6.1 Sol | Claude Opus 5.5 | Ran | Result |
| --- | --- | --- | --- | --- | --- |
| dotnet#40442 | dotnet#40369 Windows title bar | `Window` | + `TitleView`,
`Lifecycle` | Windows, 3 categories | Passed in 39 min |
| dotnet#40443 | dotnet#40366 Shell modal stack | `Shell1/2` on iOS, Mac | same |
iOS + MacCatalyst | Passed |
| dotnet#40444 | dotnet#40363 Android EmptyView | CollectionView + CarouselView |
same | Android, 8 categories | Passed in 41 min |
| dotnet#40445 | dotnet#40344 Shell tab reselect | `Shell1/2`, `Navigation` | +
`TabbedPage` | All platforms, 4 categories | Running at time of writing
|
| dotnet#40446 | dotnet#40291 `DisplayPromptAsync` | Prompt, Alert, ActionSheet |
same | All platforms, 3 categories | 1 test failed on MacCatalyst |
| dotnet#40447 | dotnet#40234 `Element` / VSM threading | Full suite | Full suite |
Full suite | Running at time of writing |
| dotnet#40448 | dotnet#40385 CodeQL CI-only | Nothing | Nothing | Nothing (no
build) | Comment only |
| dotnet#40449 | dotnet#39487 Button corner radius | `Button`, `ImageButton` on iOS,
Mac | same | iOS + MacCatalyst | 1 Button test failed |

The failures are the same tests that failed in an earlier round on the
same copied changes, so they come from those PRs, not the selection. The
comments were refreshed from the real timelines and test results with
`Update-RunTestsResults.ps1`.

**Azure previews of this PR's merge ref:**
- Defaults: identical to `main` (13 jobs, 157 matrix entries).
- Android `Button`: 2 jobs, no Material3 lane. `Button` + `Material3`:
the Material3 lane is included.
- Qualified selections: `android:Material3` + `ios:Label` runs only the
Material3 lane on Android; `android:Shell1` also runs the Shell renderer
lane; `android:SafeAreaEdges` runs its dedicated lane;
`windows:CollectionView1` runs the CV1 lane; an Android-only
CollectionView pick never starts the Windows CV1 lane.
- `selectedPlatforms` containing `"x; Remove-Item"`: only template
booleans reach the build step.

**Tests:** `RunTests.Tests.ps1` has 39 Pester tests covering the
inventory and `#if` evaluator, pruning and qualification, the two-model
union (one model empty, neither, full only when chosen), AI-only rules,
the queue request and dropped-selection guard, the check plan, comment
rendering and updates, and static template checks. The `ReviewTrigger`,
`Recover-MissedReviewCommands` and `CopilotPipeline` suites pass. `gh aw
compile --strict` succeeds.

**Not verified before merge:**
- The slash command and the dispatcher's OIDC exchange on `dotnet/maui`
itself. GitHub only runs them from the default branch. After merge,
comment `/review run-tests` on a PR to confirm.
- The `check_run`-triggered comment update on `dotnet/maui` (the script
itself is verified; the half-hourly sweep is the backup).
- The `copilot-pat-pool` token selection in the new Opus workflow (the
fork used a single secret).

---------

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — 53e2ae20 Deployed Nov 19, 2026 by kubaflo via conclusion #3144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development platform/android platform/ios platform/macos macOS / Mac Catalyst platform/windows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants