Skip to content

[CI] /review run-tests: wait for GitHub to refresh a stale test-merge commit - #40483

Merged
kubaflo merged 1 commit into
dotnet:mainfrom
kubaflo:run-tests-stale-merge-ref
Nov 20, 2026
Merged

kubaflo merged 1 commit into
dotnet:mainfrom
kubaflo:run-tests-stale-merge-ref

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Target Branch

CI workflow infrastructure, so it targets main.

Description of Change

Follow-up to #40468, found while testing /review run-tests on open PRs after it merged.

maui-pr-uitests builds GitHub's test-merge commit (refs/pull/N/merge). GitHub does not refresh it when main moves; it waits for activity on the PR. Right after #40468 merged, every open PR's merge ref still merged an older main (21 merged #40305's main, 20 merged an even older one). Some had been stale for hours. The pipeline in those commits does not declare the new uitest_* parameters, so the dispatcher stopped straight away with "merges an older main".

Changes:

  • Wait instead of failing. Get-VerifiedPullRequest now treats a merge ref whose pipeline cannot take the selection as "not ready yet". It keeps polling for up to 10 minutes, which covers a refresh GitHub is already computing. A missing merge of a new head still gives up after about 3 minutes, as before. The dispatcher job timeout goes from 15 to 25 minutes to fit.
  • Check every parameter that will be sent. Test-RunTestsPipelineSupportsSelection also requires each uitest_<Category> / uitestPlatform* name in the queue request. Otherwise a category added to main later would reach Azure as an "Unexpected parameter" 400 on PRs whose merge ref predates it.
  • If GitHub still hasn't refreshed after 10 minutes, the comment says so and suggests commenting again later or updating the branch.

Validation

…run-tests

GitHub refreshes refs/pull/N/merge lazily after the target branch moves, so right
after a pipeline change most open PRs still merge an older main whose
maui-pr-uitests cannot take the selection. Keep waiting (up to 10 minutes) for a
refreshed merge instead of failing at once, and require the built pipeline to
declare every parameter the request sends, so a category added on main later
cannot turn into an Azure 400.
Copilot AI balanced review requested due to automatic review settings November 20, 2026 17:36
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 20, 2026 17:36 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40483

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40483"

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Stale-pipeline polls count toward the missing-merge retry cap, so a temporarily missing merge SHA can stop polling prematurely despite time remaining before the ten-minute deadline.

1 open issue
What changed in this PR

This follow-up to #40468 improves MAUI’s /review run-tests dispatcher by waiting for a compatible GitHub test-merge commit instead of immediately rejecting a stale pipeline.

Changes:

  • Polls stale pipelines for up to ten minutes and provides clearer retry guidance.
  • Validates every parameter sent in the selected-test queue request.
  • Extends the dispatcher timeout to 25 minutes and adds parameter-validation tests.
File Description
.github/​workflows/​review-run-tests-dispatch.yml Increases the job timeout to accommodate polling.
.github/​scripts/​RunTests.Tests.ps1 Tests required parameter names and updates polling assertions.
.github/​scripts/​RunTests.Common.ps1 Checks all requested parameter names with regex escaping.
.github/​scripts/​Invoke-RunTests.ps1 Polls for pipeline compatibility and improves timeout guidance.

Review effort: Balanced

} else {
Write-Host "Waiting for GitHub to compute the test-merge commit for $($context.headSha.Substring(0, 8)) (attempt $attempt)..."
# A missing or outdated merge of the head itself normally settles within a minute or two.
if ($attempt -ge 12) { return @{ pr = $current; merge = $false; pipelineReady = $true } }
@kubaflo

kubaflo commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

@dalexsoto stamp

@kubaflo

This comment has been minimized.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 20, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 3 findings

Reviewed commit 609f42f.

See inline comments for details.

} else {
Write-Host "Waiting for GitHub to compute the test-merge commit for $($context.headSha.Substring(0, 8)) (attempt $attempt)..."
# A missing or outdated merge of the head itself normally settles within a minute or two.
if ($attempt -ge 12) { return @{ pr = $current; merge = $false; pipelineReady = $true } }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - The missing-merge cap counts incompatible-pipeline polls too. After 12 stale-pipeline responses for the unchanged selected head, a temporarily null merge_commit_sha during refresh immediately returns merge=false, even with most of the ten-minute window remaining. A compatible merge on the following poll is never observed, so no tests are queued. Preserve the longer retry window once an incompatible pipeline has been observed, or count missing-merge retries separately, and cover the stale-pipeline -> missing-merge -> compatible-merge transition.


# Parameters the queue request will send; the pipeline in the built commit must declare all of them.
$requiredParameters = @((Get-RunTestsTemplateParameters $selection).Keys)
$checkPipeline = $selection.mode -eq 'selected' -and -not $DryRun

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - The expanded compatibility gate also runs with SkipQueue, although that mode only publishes the selection and sends no Azure request. For a fork with a verified merge whose pipeline has the existing sentinel declarations but lacks a selected category parameter, the previous dispatcher published a selection-only result; this version waits ten minutes and reports failure instead, suppressing the skipped-check plan. The workflow explicitly sets SkipQueue for forks. Exclude SkipQueue from the queue-parameter gate so a parameter that will never be sent cannot reject selection-only publication.

$invoke | Should -Match 'Test-RunTestsPipelineSupportsSelection \(Get-RunTestsPipelineAt \$pr\.merge_commit_sha\)'
$invoke | Should -Match 'Test-RunTestsPipelineSupportsSelection \(Get-RunTestsPipelineAt \$current\.merge_commit_sha\) -Parameters \$requiredParameters'
# GitHub refreshes a test merge of an older target branch lazily: wait for it, don't fail at once.
$invoke | Should -Match 'AddMinutes\(10\)'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Regression Prevention and Test Coverage - The stale-base refresh fix has no behavioral regression test: this test reads Invoke-RunTests.ps1 as text and checks literals, while the new executable cases exercise only parameter-name validation. Returning immediately on the first incompatible pipeline would still satisfy these assertions and restore the immediate rejection reported after #40468. Add mocked polling coverage with an unchanged PR head and an incompatible merge followed by a compatible merge, asserting that the refreshed SHA reaches the queue request. Also cover persistent incompatibility through deadline expiry, asserting that no build is queued.

@MauiBot

MauiBot commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

PR Test Gate

@kubaflo — existing-test verification for commit 609f42fd.

Result SKIPPED Platform Android Commit 609f42fd


🧪 Gate analysis — click to expand

⚠ Skipped. No runnable PR tests were selected. The fix has not been verified by this gate.

No interpretable test-group summary was produced. No additional test execution is inferred.

PR head: 609f42fd.

Without-fix baseline: f79bb5b3.

@MauiBot MauiBot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 20, 2026

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamping as requested by @kubaflo; no code review performed.

@kubaflo
kubaflo merged commit 80c7d3f into dotnet:main Nov 20, 2026
12 of 14 checks passed
kubaflo added a commit that referenced this pull request Nov 20, 2026
…ures (#40485)

<!-- Please let the below note in for people that find this PR -->
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could [test the resulting
artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from
this PR and let us know in a comment if this change resolves your issue.
Thank you!

### Target Branch

- [x] I selected the applicable base branch using the [contribution
guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting).

CI workflow infrastructure, so it targets `main`.

### Description of Change

Follow-up to #40468 and #40483, found while testing `/review run-tests`
on more PRs.

I updated #39935's branch so its test-merge commit has the new pipeline,
then commented `/review run-tests`. Both models selected 8 categories on
iOS and MacCatalyst. Both attempts (19:45 and 19:57 UTC) ended with:

> Could not queue maui-pr-uitests: The Azure DevOps request failed; see
the workflow log.

The log only shows `400 (Bad Request)`, and no build was created.
- The same request validates fine in Azure's pipeline preview.
- At the same moment, `review-trigger.yml` runs that use the same
trigger identity failed to get a token: `AADSTS53003: Access has been
blocked by Conditional Access policies. The access policy does not allow
token issuance.`
- So the 400 came from Microsoft Entra's token endpoint, not from Azure
DevOps.
- Entra's error body has `error_description`, not `message`, so
`Get-RunTestsAzdoErrorText` returned nothing.

Changes:

- **Report Entra's reason.** `Get-RunTestsAzdoErrorText` also reads
`error_description` and drops the trace and correlation ids.
- **Retry the sign-in.** `Get-RunTestsAzdoToken` retries the token
exchange up to 4 times, waiting 15, 30 and 45 seconds between attempts.
If it still fails, it throws an error with id
`RunTestsEntraSignInFailed`.
- **Say what failed.** `Invoke-RunTests.ps1` turns that error into
"Could not sign in to Azure DevOps: Microsoft Entra refused the token
(AADSTS…). Nothing was queued; comment `/review run-tests` again later."
The comment no longer suggests that Azure DevOps rejected the selection.

The Conditional Access block itself is a dnceng tenant policy on the
shared `AZDO_TRIGGER_CLIENT_ID` identity. It also affects `/review`
through `review-trigger.yml`. It can't be fixed in this repo.

### Validation

- `RunTests.Tests.ps1` and `CopilotPipeline.Tests.ps1`: 69 passed. New
tests cover:
  - the Entra error text;
- four sign-in attempts followed by the sign-in error id, with mocked
HTTP calls;
  - the dispatcher's message for a failed sign-in.
- Other real `/review run-tests` results today on `main`:
  - #40437 queued targeted build 1692555 with 9 categories;
- #40478 (`net11`) got the "can't run on PRs that target `net11` yet"
comment;
- #40369 and #40344, whose test-merge commits are stale, waited 10
minutes and then explained why.

Co-authored-by: Copilot CI <copilot-ci@microsoft.com>

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — 609f42fd Deployed Nov 20, 2026 by kubaflo via conclusion #3177
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants