Repository navigation
Conversation
The agent picks platforms and UI test categories from the PR diff. A trusted safe-output job validates the choice against the pipeline inventory, queues maui-pr-uitests as a pull-request build so Azure Pipelines reports native checks on the PR head, publishes a Skipped check for every platform and category that was not selected, and posts a collapsible summary comment that is updated when the run finishes.
The agent reads the collected context and uses GitHub read tools; it never needs PR code on disk.
The previous expression also enabled the zero-test guard in the default full run, which failed every job whose test step did not produce results. The dispatcher always sets selectedRun for targeted runs.
Untyped template parameters receive the boolean as the string 'False',
which is truthy in a bare ${{ if }}.
…nly selection - Hand the AI proposal to review-run-tests-dispatch.yml. Jobs in the copilot-pat-pool environment get an OIDC subject the Azure DevOps federated credential does not trust; the dispatch workflow runs on main without an environment, like review-trigger.yml. - Queue only after GitHub's test-merge commit is confirmed to merge the selected head, and cancel any build whose source version differs. - Evaluate #if guards per platform to know which categories have enabled tests (including the PR's own changed tests). Drop categories and platforms that would run zero tests, and pass platform-qualified categories (ios:Window) when a category only runs on some platforms. - Pass the sample-build platforms to PowerShell as template-evaluated booleans instead of interpolating parameter text. - Teach the rebalance-ui-test-categories skill the typed parameter form. - Add regression tests for the full-run guard, the qualified matrix filter, the build-script inputs and the skill.
An agent can emit its proposal and then exit non-zero; dispatching both the proposal and the fallback would queue two runs for one command.
The run-tests comment now lists the failing tests with the platforms they failed on, read from the public Azure DevOps test results API. When the maui-pr-uitests check completes again (failed jobs re-run), finished comments for that commit are refreshed too, and the list notes that it may include failures from earlier attempts.
Copy of dotnet#40366 for a /review run-tests trial. Do not merge.
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40424Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40424" |
This comment has been minimized.
This comment has been minimized.
Skill Validation Results
❌ Skill Validation Results —
|
Selected UI TestsShell cross-tab modal navigation fix; runs both Shell UI test shards on iOS and MacCatalyst. ✅ maui-pr-uitests #1689925 passed. ✅ Running — 2 categories on iOS, MacCatalyst
⏭️ Skipped — 75 categories, 2 platformsPlatforms not run: Android, Windows Categories not run: Each skipped platform and category is listed as a Skipped 📊 Results — succeeded: 5 passed, 0 failed of 5 jobs
🤖 Why these tests — AI rationale
Generated by /review run-tests. A targeted run is a fast signal, not full-suite certification. |
This comment has been minimized.
This comment has been minimized.
PR Test Gate |
| runtimeVariant: "CoreCLR" | ||
| useWindowsCV1: true | ||
| skipProvisioning: ${{ parameters.skipProvisioning }} | ||
| - ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), ne(replace(replace(join('', parameters.selectedCategories), 'Material3', ''), 'SafeAreaEdges', ''), ''))) }}: |
There was a problem hiding this comment.
[major] Logic and Correctness Verification - Selecting Android and iOS with Material3 and Window produces selectedCategories: [Material3, 'ios:Window'], because Window has no Android tests. This guard still includes the ordinary Android CoreCLR stage: ios:Window survives the replacements, although neither value matches its matrix. That leaves an extra job without CATEGORYGROUP, which ui-tests-steps.yml interpolates into its test invocation, causing a failure instead of running only the intended Material3 and iOS Window lanes. Gate each ordinary stage on categories that actually match that platform's matrix.
| artifactName: 'uitest-snapshot-results-android-material3-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)' | ||
|
|
||
| # SafeAreaEdges tests on Android API 36 with Pixel 3 XL | ||
| - ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), containsValue(parameters.selectedCategories, 'SafeAreaEdges'))) }}: |
There was a problem hiding this comment.
[moderate] Build & MSBuild - The documented platform-qualified category format is not recognized by the dedicated lane selectors. For selectedPlatforms: [android] and selectedCategories: ['android:SafeAreaEdges'], this condition omits the SafeAreaEdges stage, while the ordinary Android matrix explicitly excludes SafeAreaEdges, so the requested tests never execute. The Shell renderer selector and Windows CV1 matrix likewise only recognize bare categories. Apply the same platform-qualified matching used by the ordinary matrices to these dedicated lanes.
| # in that case, so only fall back when no proposal was dispatched. | ||
| run-tests-fallback: | ||
| needs: [pre_activation, activation, agent, run_selected_ui_tests] | ||
| if: ${{ !cancelled() && needs.pre_activation.outputs.context_ready == 'true' && needs.agent.result == 'failure' && needs.run_selected_ui_tests.result != 'success' }} |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - A successful agent job without a run_selected_ui_tests item never reaches the missing-proposal fallback. For example, emitting only the exposed missing_data or noop tool skips run_selected_ui_tests, and this condition also skips the fallback because the agent exited successfully. Nothing is dispatched to fulfill the request, and Invoke-RunTests.ps1's full-suite fallback is unreachable. Cover completed agent runs with no successful handover, while retaining the existing protection against dispatching both a proposal and a fallback.
| BeforeAll { | ||
| $script:pipelines = Join-Path $PSScriptRoot '../../eng/pipelines/common' | ||
| } | ||
|
|
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - These assertions check only template expressions and parameter values; none exercises the new TRX execution-count guard. In particular, there is no valid namespaced TRX fixture with executed="1" that must be accepted, contrasted with zero/all-skipped or missing results that must be rejected. A guard that rejects every selected run would still satisfy these assertions. Add discriminatory behavioral coverage of the actual guard so successful targeted runs cannot regress into unconditional failures unnoticed.
Warning
Do not merge. Do not review. Temporary trial PR for
/review run-tests(#40305). It will be closed once the trial finishes.This branch is #40305 plus the change from #40366, so
maui-pr-uitestshere uses the new selection parameters. The UI test selection comes from the AI agent's run on the same diff, and the trusted dispatcher queues only the selected categories and platforms. Second round, after the results-comment fixes.