Skip to content

[run-tests trial 40366] [WIP] [iOS] Fix Shell cross-tab modal navigation - #40397

Closed
kubaflo wants to merge 10 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40366
Closed

kubaflo wants to merge 10 commits into
dotnet:mainfrom
kubaflo:run-tests-real-40366

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Do not merge. Do not review. Temporary trial PR for /review run-tests (#40305). It will be closed once the trial finishes.

This branch is #40305 plus the change from #40366, so maui-pr-uitests here uses the new selection parameters. The UI test selection comes from the AI agent's run on the same diff, and the trusted dispatcher queues only the selected categories and platforms.

Copilot CI added 9 commits November 16, 2026 21:39
The agent picks platforms and UI test categories from the PR diff. A trusted
safe-output job validates the choice against the pipeline inventory, queues
maui-pr-uitests as a pull-request build so Azure Pipelines reports native
checks on the PR head, publishes a Skipped check for every platform and
category that was not selected, and posts a collapsible summary comment that
is updated when the run finishes.
The agent reads the collected context and uses GitHub read tools; it never
needs PR code on disk.
The previous expression also enabled the zero-test guard in the default
full run, which failed every job whose test step did not produce results.
The dispatcher always sets selectedRun for targeted runs.
Untyped template parameters receive the boolean as the string 'False',
which is truthy in a bare ${{ if }}.
…nly selection

- Hand the AI proposal to review-run-tests-dispatch.yml. Jobs in the
  copilot-pat-pool environment get an OIDC subject the Azure DevOps
  federated credential does not trust; the dispatch workflow runs on main
  without an environment, like review-trigger.yml.
- Queue only after GitHub's test-merge commit is confirmed to merge the
  selected head, and cancel any build whose source version differs.
- Evaluate #if guards per platform to know which categories have enabled
  tests (including the PR's own changed tests). Drop categories and
  platforms that would run zero tests, and pass platform-qualified
  categories (ios:Window) when a category only runs on some platforms.
- Pass the sample-build platforms to PowerShell as template-evaluated
  booleans instead of interpolating parameter text.
- Teach the rebalance-ui-test-categories skill the typed parameter form.
- Add regression tests for the full-run guard, the qualified matrix
  filter, the build-script inputs and the skill.
An agent can emit its proposal and then exit non-zero; dispatching both
the proposal and the fallback would queue two runs for one command.
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 18, 2026 21:28 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40397

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40397"

@azure-pipelines

This comment has been minimized.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: 698d851.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

❌ Skill Validation Results — 698d851 · [run-tests trial 40366] [WIP] [iOS] Fix Shell cross-tab modal navigation · 2026-11-18T21:33:18Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

Copy of dotnet#40366 for a /review run-tests trial. Do not merge.
@kubaflo
kubaflo force-pushed the run-tests-real-40366 branch from 17134c6 to 698d851 Compare November 18, 2026 21:32
@github-actions github-actions Bot added area-ai-agents Copilot CLI agents, agent skills, AI-assisted development platform/ios platform/macos macOS / Mac Catalyst labels Nov 18, 2026
@kubaflo

kubaflo commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Selected UI Tests

@kubaflo — AI-selected UI tests for commit 698d851e.

Result FAILED Scope Targeted Platforms iOS, MacCatalyst Categories 2 of 77 Commit 698d851e

Shell cross-tab modal navigation fix; runs both Shell UI test shards on iOS and MacCatalyst.

❌ maui-pr-uitests #1688769 failed. See the failing checks below or the run for details.


✅ Running — 2 categories on iOS, MacCatalyst
Platform Categories
iOS Shell1, Shell2
MacCatalyst Shell1, Shell2
⏭️ Skipped — 75 categories, 2 platforms

Platforms not run: Android, Windows

Categories not run: Accessibility, ActionSheet, ActivityIndicator, Animation, AppLinks, Border, BoxView, Brush, Button, CarouselView, CollectionView1, CollectionView2, CollectionView3, CollectionView4, CollectionView5, CollectionView6, CollectionView7, Cells, CheckBox, ContextActions, CustomRenderers, DatePicker, Dispatcher, DisplayAlert, DisplayPrompt, DragAndDrop, Entry, Editor, Effects, Essentials, FlyoutPage, Focus, Fonts, Frame, Gestures, GraphicsView, Image, ImageButton, IndicatorView, InputTransparent, IsEnabled, IsVisible, Label, Layout, Lifecycle, ManualReview, Maps, Navigation, Page, Performance, Picker, ProgressBar, RadioButton, RefreshView, SafeAreaEdges, Shadow, ScrollView, SearchBar, Shape, Slider, SoftInput, Stepper, Switch, SwipeView, TabbedPage, TableView, TimePicker, TitleView, ToolbarItem, Triggers, ViewBaseTests, VisualStateManager, Window, WebView, Material3

In this manual trial the Skipped maui-pr-uitests (skipped) … checks are not published; GitHub only lets the /review run-tests workflow create them. Use /azp run maui-pr-uitests for the full suite.

📊 Results — failed: 4 passed, 1 failed of 5 jobs
Job Result
Build UITests CoreCLR Sample App Build Sample App ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) Shell1 ✅ succeeded
iOS UITests CoreCLR Controls (vlatest) Shell2 ✅ succeeded
MacCatalyst UITests CoreCLR Controls Shell1 ❌ failed
MacCatalyst UITests CoreCLR Controls Shell2 ✅ succeeded

Failed tests (1)

Test Platforms
Issue35764.ClearPlaceholderIconShouldHideWhenDisabled MacCatalyst
🤖 Why these tests — AI rationale
  • ShellSection.cs changes modal stack cleanup only under IOS || MACCATALYST; exclude Android/Windows.
  • Shell covers both runnable shards; added Shell device test matches this scope.
  • No NavigationPage, TabbedPage or base infrastructure changes; no full run.

Generated by /review run-tests. A targeted run is a fast signal, not full-suite certification.

@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 18, 2026
@MauiBot

MauiBot commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@kubaflo — existing-test verification; Setup did not capture a tested commit.

Result INCONCLUSIVE Platform iOS


🧪 Gate analysis — click to expand

⚠ Inconclusive. Setup, build, or execution did not produce a conclusive result. This does not prove the fix is incorrect.

No interpretable test-group summary was produced. No additional test execution is inferred.

Setup did not capture a verified PR head; no commit is claimed as tested.

No immutable without-fix baseline was prepared.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 18, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 3 findings

Reviewed commit 698d851.

See inline comments for details.

env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.issue.number || inputs.pr_number }}
run: pwsh -NoProfile -File .github/scripts/Get-RunTestsContext.ps1 -PrNumber "$PR_NUMBER" -OutputDirectory "$RUNNER_TEMP/run-tests"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Build & MSBuild - The collector invokes pwsh, but the generated pre_activation job runs on ubuntu-slim, which does not include PowerShell, and no preceding step installs it. An authorized /review run-tests fails before producing context; activation is skipped and the fallback cannot run because context_ready was never set. Use runs-on-slim: ubuntu-latest, as the existing issue-trace-regression workflow does for its PowerShell collector, and regenerate the lock.

runtimeVariant: "CoreCLR"
useWindowsCV1: true
skipProvisioning: ${{ parameters.skipProvisioning }}
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), ne(replace(replace(join('', parameters.selectedCategories), 'Material3', ''), 'SafeAreaEdges', ''), ''))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Build & MSBuild - This stage gate examines categories for every platform rather than entries runnable in the Android matrix. Selecting Android and Windows with Material3 and Window is normalized to Material3 plus windows:Window: Window tests are compiled out on Android. Removing Material3 still leaves windows:Window, so the ordinary Android stage is emitted, but neither value creates a matrix entry. Its test step consequently has no CATEGORYGROUP, despite the separate Material3 and Windows lanes being valid. Gate ordinary stages on their own matching matrix entries, excluding qualified categories belonging elsewhere.

artifactName: 'uitest-snapshot-results-android-material3-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)'

# SafeAreaEdges tests on Android API 36 with Pixel 3 XL
- ${{ if and(containsValue(parameters.selectedPlatforms, 'android'), or(eq(length(parameters.selectedCategories), 0), containsValue(parameters.selectedCategories, 'SafeAreaEdges'))) }}:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - The documented platform-qualified category contract is handled by ordinary matrices but not the dedicated lanes. With selectedPlatforms=[android] and selectedCategories=[android:SafeAreaEdges], the ordinary matrix excludes SafeAreaEdges and this gate also omits its dedicated stage, so the requested tests never run. Similarly, android:Shell1 omits ShellRenderer coverage, while windows:CollectionView1 enables the CV1 stage without creating a matrix entry. Recognize qualified identifiers in the dedicated gates and matrices, or normalize them consistently before evaluating either.

@kubaflo kubaflo closed this Nov 19, 2026
@kubaflo
kubaflo deleted the run-tests-real-40366 branch November 19, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development platform/ios platform/macos macOS / Mac Catalyst

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants