Skip to content

[run-tests trial 40366] [iOS] Fix Shell cross-tab modal navigation - #40424

Closed
kubaflo wants to merge 11 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40366
Closed

kubaflo wants to merge 11 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40366

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Do not merge. Do not review. Temporary trial PR for /review run-tests (#40305). It will be closed once the trial finishes.

This branch is #40305 plus the change from #40366, so maui-pr-uitests here uses the new selection parameters. The UI test selection comes from the AI agent's run on the same diff, and the trusted dispatcher queues only the selected categories and platforms. Second round, after the results-comment fixes.

Copilot CI added 11 commits November 16, 2026 21:39
The agent picks platforms and UI test categories from the PR diff. A trusted
safe-output job validates the choice against the pipeline inventory, queues
maui-pr-uitests as a pull-request build so Azure Pipelines reports native
checks on the PR head, publishes a Skipped check for every platform and
category that was not selected, and posts a collapsible summary comment that
is updated when the run finishes.
The agent reads the collected context and uses GitHub read tools; it never
needs PR code on disk.
The previous expression also enabled the zero-test guard in the default
full run, which failed every job whose test step did not produce results.
The dispatcher always sets selectedRun for targeted runs.
Untyped template parameters receive the boolean as the string 'False',
which is truthy in a bare ${{ if }}.
…nly selection

- Hand the AI proposal to review-run-tests-dispatch.yml. Jobs in the
  copilot-pat-pool environment get an OIDC subject the Azure DevOps
  federated credential does not trust; the dispatch workflow runs on main
  without an environment, like review-trigger.yml.
- Queue only after GitHub's test-merge commit is confirmed to merge the
  selected head, and cancel any build whose source version differs.
- Evaluate #if guards per platform to know which categories have enabled
  tests (including the PR's own changed tests). Drop categories and
  platforms that would run zero tests, and pass platform-qualified
  categories (ios:Window) when a category only runs on some platforms.
- Pass the sample-build platforms to PowerShell as template-evaluated
  booleans instead of interpolating parameter text.
- Teach the rebalance-ui-test-categories skill the typed parameter form.
- Add regression tests for the full-run guard, the qualified matrix
  filter, the build-script inputs and the skill.
An agent can emit its proposal and then exit non-zero; dispatching both
the proposal and the fallback would queue two runs for one command.
The run-tests comment now lists the failing tests with the platforms they
failed on, read from the public Azure DevOps test results API. When the
maui-pr-uitests check completes again (failed jobs re-run), finished comments
for that commit are refreshed too, and the list notes that it may include
failures from earlier attempts.
Copy of dotnet#40366 for a /review run-tests trial. Do not merge.
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 11:28 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40424

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40424"

@azure-pipelines

This comment has been minimized.

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 11:29 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: 21068a8.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

❌ Skill Validation Results — 21068a8 · [run-tests trial 40366] [iOS] Fix Shell cross-tab modal navigation · 2026-11-19T11:29:09Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 19, 2026 11:30 — with GitHub Actions Inactive
@kubaflo

kubaflo commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Selected UI Tests

@kubaflo — AI-selected UI tests for commit 21068a85.

Result PASSED Scope Targeted Platforms iOS, MacCatalyst Categories 2 of 77 Commit 21068a85

Shell cross-tab modal navigation fix; runs both Shell UI test shards on iOS and MacCatalyst.

✅ maui-pr-uitests #1689925 passed.


✅ Running — 2 categories on iOS, MacCatalyst
Platform Categories
iOS Shell1, Shell2
MacCatalyst Shell1, Shell2
⏭️ Skipped — 75 categories, 2 platforms

Platforms not run: Android, Windows

Categories not run: Accessibility, ActionSheet, ActivityIndicator, Animation, AppLinks, Border, BoxView, Brush, Button, CarouselView, CollectionView1, CollectionView2, CollectionView3, CollectionView4, CollectionView5, CollectionView6, CollectionView7, Cells, CheckBox, ContextActions, CustomRenderers, DatePicker, Dispatcher, DisplayAlert, DisplayPrompt, DragAndDrop, Entry, Editor, Effects, Essentials, FlyoutPage, Focus, Fonts, Frame, Gestures, GraphicsView, Image, ImageButton, IndicatorView, InputTransparent, IsEnabled, IsVisible, Label, Layout, Lifecycle, ManualReview, Maps, Navigation, Page, Performance, Picker, ProgressBar, RadioButton, RefreshView, SafeAreaEdges, Shadow, ScrollView, SearchBar, Shape, Slider, SoftInput, Stepper, Switch, SwipeView, TabbedPage, TableView, TimePicker, TitleView, ToolbarItem, Triggers, ViewBaseTests, VisualStateManager, Window, WebView, Material3

Each skipped platform and category is listed as a Skipped maui-pr-uitests (skipped) … check on this commit. Use /azp run maui-pr-uitests for the full suite.

📊 Results — succeeded: 5 passed, 0 failed of 5 jobs
Job Result
Build UITests CoreCLR Sample App Build Sample App ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) Shell1 ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) Shell2 ✅ succeeded
MacCatalyst UITests CoreCLR Controls Shell1 ✅ succeeded
MacCatalyst UITests CoreCLR Controls Shell2 ✅ succeeded
🤖 Why these tests — AI rationale
  • ShellSection.cs changes modal stack cleanup only under IOS || MACCATALYST; exclude Android/Windows.
  • Shell covers both runnable shards; added Shell device test matches this scope.
  • No NavigationPage, TabbedPage or base infrastructure changes; no full run.

Generated by /review run-tests. A targeted run is a fast signal, not full-suite certification.

@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 19, 2026
@MauiBot

MauiBot commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@kubaflo — existing-test verification for commit 21068a85.

Result SKIPPED Platform Android Commit 21068a85


🧪 Gate analysis — click to expand

⚠ Skipped. No runnable PR tests were selected. The fix has not been verified by this gate.

No interpretable test-group summary was produced. No additional test execution is inferred.

PR head: 21068a85.

Without-fix baseline: 5c7a422e.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 19, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 4 findings

Reviewed commit 21068a8.

See inline comments for details.

runtimeVariant: "CoreCLR"
useWindowsCV1: true
skipProvisioning: ${{ parameters.skipProvisioning }}
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), ne(replace(replace(join('', parameters.selectedCategories), 'Material3', ''), 'SafeAreaEdges', ''), ''))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Logic and Correctness Verification - Selecting Android and iOS with Material3 and Window produces selectedCategories: [Material3, 'ios:Window'], because Window has no Android tests. This guard still includes the ordinary Android CoreCLR stage: ios:Window survives the replacements, although neither value matches its matrix. That leaves an extra job without CATEGORYGROUP, which ui-tests-steps.yml interpolates into its test invocation, causing a failure instead of running only the intended Material3 and iOS Window lanes. Gate each ordinary stage on categories that actually match that platform's matrix.

artifactName: 'uitest-snapshot-results-android-material3-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)'

# SafeAreaEdges tests on Android API 36 with Pixel 3 XL
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), containsValue(parameters.selectedCategories, 'SafeAreaEdges'))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Build & MSBuild - The documented platform-qualified category format is not recognized by the dedicated lane selectors. For selectedPlatforms: [android] and selectedCategories: ['android:SafeAreaEdges'], this condition omits the SafeAreaEdges stage, while the ordinary Android matrix explicitly excludes SafeAreaEdges, so the requested tests never execute. The Shell renderer selector and Windows CV1 matrix likewise only recognize bare categories. Apply the same platform-qualified matching used by the ordinary matrices to these dedicated lanes.

# in that case, so only fall back when no proposal was dispatched.
run-tests-fallback:
needs: [pre_activation, activation, agent, run_selected_ui_tests]
if: ${{ !cancelled() && needs.pre_activation.outputs.context_ready == 'true' && needs.agent.result == 'failure' && needs.run_selected_ui_tests.result != 'success' }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - A successful agent job without a run_selected_ui_tests item never reaches the missing-proposal fallback. For example, emitting only the exposed missing_data or noop tool skips run_selected_ui_tests, and this condition also skips the fallback because the agent exited successfully. Nothing is dispatched to fulfill the request, and Invoke-RunTests.ps1's full-suite fallback is unreachable. Cover completed agent runs with no successful handover, while retaining the existing protection against dispatching both a proposal and a fallback.

BeforeAll {
$script:pipelines = Join-Path $PSScriptRoot '../../eng/pipelines/common'
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - These assertions check only template expressions and parameter values; none exercises the new TRX execution-count guard. In particular, there is no valid namespaced TRX fixture with executed="1" that must be accepted, contrasted with zero/all-skipped or missing results that must be rejected. A guard that rejects every selected run would still satisfy these assertions. Add discriminatory behavioral coverage of the actual guard so successful targeted runs cannot regress into unconditional failures unnoticed.

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — 21068a85 Deployed Nov 19, 2026 by kubaflo via conclusion #3122
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants