Repository navigation
[CI] /review run-tests: wait for GitHub to refresh a stale test-merge commit - #40483
Conversation
…run-tests GitHub refreshes refs/pull/N/merge lazily after the target branch moves, so right after a pipeline change most open PRs still merge an older main whose maui-pr-uitests cannot take the selection. Keep waiting (up to 10 minutes) for a refreshed merge instead of failing at once, and require the built pipeline to declare every parameter the request sends, so a category added on main later cannot turn into an Azure 400.
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40483Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40483" |
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Stale-pipeline polls count toward the missing-merge retry cap, so a temporarily missing merge SHA can stop polling prematurely despite time remaining before the ten-minute deadline.
1 open issue
What changed in this PR
This follow-up to #40468 improves MAUI’s /review run-tests dispatcher by waiting for a compatible GitHub test-merge commit instead of immediately rejecting a stale pipeline.
Changes:
- Polls stale pipelines for up to ten minutes and provides clearer retry guidance.
- Validates every parameter sent in the selected-test queue request.
- Extends the dispatcher timeout to 25 minutes and adds parameter-validation tests.
| File | Description |
|---|---|
.github/workflows/review-run-tests-dispatch.yml |
Increases the job timeout to accommodate polling. |
.github/scripts/RunTests.Tests.ps1 |
Tests required parameter names and updates polling assertions. |
.github/scripts/RunTests.Common.ps1 |
Checks all requested parameter names with regex escaping. |
.github/scripts/Invoke-RunTests.ps1 |
Polls for pipeline compatibility and improves timeout guidance. |
Review effort: Balanced
| } else { | ||
| Write-Host "Waiting for GitHub to compute the test-merge commit for $($context.headSha.Substring(0, 8)) (attempt $attempt)..." | ||
| # A missing or outdated merge of the head itself normally settles within a minute or two. | ||
| if ($attempt -ge 12) { return @{ pr = $current; merge = $false; pipelineReady = $true } } |
|
@dalexsoto stamp |
This comment has been minimized.
This comment has been minimized.
| } else { | ||
| Write-Host "Waiting for GitHub to compute the test-merge commit for $($context.headSha.Substring(0, 8)) (attempt $attempt)..." | ||
| # A missing or outdated merge of the head itself normally settles within a minute or two. | ||
| if ($attempt -ge 12) { return @{ pr = $current; merge = $false; pipelineReady = $true } } |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - The missing-merge cap counts incompatible-pipeline polls too. After 12 stale-pipeline responses for the unchanged selected head, a temporarily null merge_commit_sha during refresh immediately returns merge=false, even with most of the ten-minute window remaining. A compatible merge on the following poll is never observed, so no tests are queued. Preserve the longer retry window once an incompatible pipeline has been observed, or count missing-merge retries separately, and cover the stale-pipeline -> missing-merge -> compatible-merge transition.
|
|
||
| # Parameters the queue request will send; the pipeline in the built commit must declare all of them. | ||
| $requiredParameters = @((Get-RunTestsTemplateParameters $selection).Keys) | ||
| $checkPipeline = $selection.mode -eq 'selected' -and -not $DryRun |
There was a problem hiding this comment.
[moderate] Logic and Correctness Verification - The expanded compatibility gate also runs with SkipQueue, although that mode only publishes the selection and sends no Azure request. For a fork with a verified merge whose pipeline has the existing sentinel declarations but lacks a selected category parameter, the previous dispatcher published a selection-only result; this version waits ten minutes and reports failure instead, suppressing the skipped-check plan. The workflow explicitly sets SkipQueue for forks. Exclude SkipQueue from the queue-parameter gate so a parameter that will never be sent cannot reject selection-only publication.
| $invoke | Should -Match 'Test-RunTestsPipelineSupportsSelection \(Get-RunTestsPipelineAt \$pr\.merge_commit_sha\)' | ||
| $invoke | Should -Match 'Test-RunTestsPipelineSupportsSelection \(Get-RunTestsPipelineAt \$current\.merge_commit_sha\) -Parameters \$requiredParameters' | ||
| # GitHub refreshes a test merge of an older target branch lazily: wait for it, don't fail at once. | ||
| $invoke | Should -Match 'AddMinutes\(10\)' |
There was a problem hiding this comment.
[moderate] Regression Prevention and Test Coverage - The stale-base refresh fix has no behavioral regression test: this test reads Invoke-RunTests.ps1 as text and checks literals, while the new executable cases exercise only parameter-name validation. Returning immediately on the first incompatible pipeline would still satisfy these assertions and restore the immediate rejection reported after #40468. Add mocked polling coverage with an unchanged PR head and an incompatible merge followed by a compatible merge, asserting that the refreshed SHA reaches the queue request. Also cover persistent incompatibility through deadline expiry, asserting that no build is queued.
PR Test Gate |
…ures (#40485) <!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Target Branch - [x] I selected the applicable base branch using the [contribution guide](https://github.com/dotnet/maui/blob/main/.github/CONTRIBUTING.md#choose-your-pr-base-before-starting). CI workflow infrastructure, so it targets `main`. ### Description of Change Follow-up to #40468 and #40483, found while testing `/review run-tests` on more PRs. I updated #39935's branch so its test-merge commit has the new pipeline, then commented `/review run-tests`. Both models selected 8 categories on iOS and MacCatalyst. Both attempts (19:45 and 19:57 UTC) ended with: > Could not queue maui-pr-uitests: The Azure DevOps request failed; see the workflow log. The log only shows `400 (Bad Request)`, and no build was created. - The same request validates fine in Azure's pipeline preview. - At the same moment, `review-trigger.yml` runs that use the same trigger identity failed to get a token: `AADSTS53003: Access has been blocked by Conditional Access policies. The access policy does not allow token issuance.` - So the 400 came from Microsoft Entra's token endpoint, not from Azure DevOps. - Entra's error body has `error_description`, not `message`, so `Get-RunTestsAzdoErrorText` returned nothing. Changes: - **Report Entra's reason.** `Get-RunTestsAzdoErrorText` also reads `error_description` and drops the trace and correlation ids. - **Retry the sign-in.** `Get-RunTestsAzdoToken` retries the token exchange up to 4 times, waiting 15, 30 and 45 seconds between attempts. If it still fails, it throws an error with id `RunTestsEntraSignInFailed`. - **Say what failed.** `Invoke-RunTests.ps1` turns that error into "Could not sign in to Azure DevOps: Microsoft Entra refused the token (AADSTS…). Nothing was queued; comment `/review run-tests` again later." The comment no longer suggests that Azure DevOps rejected the selection. The Conditional Access block itself is a dnceng tenant policy on the shared `AZDO_TRIGGER_CLIENT_ID` identity. It also affects `/review` through `review-trigger.yml`. It can't be fixed in this repo. ### Validation - `RunTests.Tests.ps1` and `CopilotPipeline.Tests.ps1`: 69 passed. New tests cover: - the Entra error text; - four sign-in attempts followed by the sign-in error id, with mocked HTTP calls; - the dispatcher's message for a failed sign-in. - Other real `/review run-tests` results today on `main`: - #40437 queued targeted build 1692555 with 9 categories; - #40478 (`net11`) got the "can't run on PRs that target `net11` yet" comment; - #40369 and #40344, whose test-merge commits are stale, waited 10 minutes and then explained why. Co-authored-by: Copilot CI <copilot-ci@microsoft.com>

Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Target Branch
CI workflow infrastructure, so it targets
main.Description of Change
Follow-up to #40468, found while testing
/review run-testson open PRs after it merged.maui-pr-uitestsbuilds GitHub's test-merge commit (refs/pull/N/merge). GitHub does not refresh it whenmainmoves; it waits for activity on the PR. Right after #40468 merged, every open PR's merge ref still merged an oldermain(21 merged #40305'smain, 20 merged an even older one). Some had been stale for hours. The pipeline in those commits does not declare the newuitest_*parameters, so the dispatcher stopped straight away with "merges an older main".Changes:
Get-VerifiedPullRequestnow treats a merge ref whose pipeline cannot take the selection as "not ready yet". It keeps polling for up to 10 minutes, which covers a refresh GitHub is already computing. A missing merge of a new head still gives up after about 3 minutes, as before. The dispatcher job timeout goes from 15 to 25 minutes to fit.Test-RunTestsPipelineSupportsSelectionalso requires eachuitest_<Category>/uitestPlatform*name in the queue request. Otherwise a category added tomainlater would reach Azure as an "Unexpected parameter" 400 on PRs whose merge ref predates it.Validation
RunTests.Tests.ps1andCopilotPipeline.Tests.ps1: 66 passed. A new test covers parameters that are missing from the built pipeline, including regex metacharacters.Real
/review run-testscomments after [CI] /review run-tests: queue targeted runs without extra Azure DevOps permissions, plus rollout fixes #40468 merged, using the currentmaindispatcher (without this PR):main(head pushed after the merge)selectedRun=true, the four platform booleans and only the 9uitest_*values; only the CollectionView, CV1 and per-platform stages startedmainmainmainA
/review run-testscomment, an API read and a no-op base edit did not refresh a stale merge ref within 10+ minutes; a push to the PR head refreshes it at once. Until GitHub refreshes the older PRs, they need an update or push. This PR waits for any refresh GitHub is already computing, explains the situation in the comment, and keeps a later category addition from becoming an Azure 400.