Skip to content

[CI] /review run-tests: dispatch maui-pr-uitests through a run-tests/* branch, since Entra blocks the Azure DevOps sign-in - #40504

Open
kubaflo wants to merge 1 commit into
mainfrom
run-tests-dispatch-branch
Open

kubaflo wants to merge 1 commit into
mainfrom
run-tests-dispatch-branch

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Target Branch

CI workflow infrastructure follow-up to #40468 / #40485 / #40492, so it targets main.

Why /review run-tests still queues nothing

Since about 21:00 UTC on 2026-11-20, every /review run-tests dispatch fails at the same step: Microsoft Entra refuses the GitHub OIDC → Azure DevOps token exchange with AADSTS53003: Access has been blocked by Conditional Access policies. Before that, the same code queued build 1692555. review-trigger.yml (/review → maui-copilot) fails the same way whenever it gets as far as the token step; its other runs only look green because they stop before it. #40492 added an AZDO_TRIGGER_PAT fallback, but no such secret exists, and using a personal PAT to get around a Conditional Access policy is not a fix we should ship.

The repo has no other way to queue a build. I checked each option:

Option Result
Federated identity (OIDC) AADSTS53003 on every attempt since 2026-11-20
/azp run comment from github-actions[bot] Azure Pipelines answers "Commenter does not have sufficient privileges" (probe #40500)
Queue from inside Azure DevOps with System.AccessToken The job identity (public Build Service) has no Queue builds permission on maui-pr-uitests: permission API false, queue request 403 (probe #40502)

Description of Change

/review run-tests now starts maui-pr-uitests through its YAML CI trigger, so no Azure DevOps credentials are involved:

  1. Invoke-RunTests.ps1 validates the selection as before, then uses the Git Data API to push a dispatch commit to run-tests/pr<N>-<run id>. Its tree is GitHub's test-merge tree of the PR, with eng/pipelines and .github replaced by the copies at the target branch tip, plus a generated eng/pipelines/common/run-tests-request.yml. Its parent is that target-branch commit, so the push changes no workflow. GITHUB_TOKEN is not allowed to change workflows, and its pushes don't start Actions runs.
  2. ci-uitests.yml now builds run-tests/* branches. For those branches it reads the request and passes the same selectedPlatforms / selectedCategories to common/ui-tests.yml that the uitest_* parameters produced before. If the mode isn't full, the run is narrowed, so an unfilled request runs nothing rather than the full suite. Dispatch builds get the pull-request lanes (iOS latest only).
  3. Secrets: variables.yml doesn't link the MAUI variable group (maui-cert-password) for run-tests/* builds. PR code then runs without secrets, as it does in a PR build from a fork. The pipeline YAML always comes from the target branch, never from the PR.
  4. Azure Pipelines reports its checks on the dispatch commit, so a maui-pr-uitests (selected run) check on the PR head follows the build. The skipped and selection checks are unchanged. Update-RunTestsResults.ps1 matches comments by the dispatch commit too, completes that check, and its 30-minute sweep deletes dispatch branches older than 3 days.
  5. Removed: the Entra token exchange, the PAT fallback, id-token: write, and the template-parameter queue request. The dispatch workflow now needs contents: write, and the results workflow needs checks: write and contents: write.

Validation

  • Ran Invoke-RunTests.ps1 locally with this branch as -PipelineSource, on [CollectionView] Add ItemTapped event and ItemTappedCommand #40437 at 85ec75d5, with the real GPT-6.1 Sol and Claude Opus 5.5 proposals from run 44944036321. It pushed run-tests/pr40437-1795314808 (dispatch commit 81026002), and maui-pr-uitests build 1693858 started from the CI trigger about 15 s later. That build has 8 stages instead of 12, and only maui-pr-uitests built the branch. The comment was posted, and Update-RunTestsResults.ps1 finds it by the dispatch commit.
  • Azure DevOps pipeline preview of the dispatch YAML:
    • selected: only the selected stages; each matrix holds only the selected categories, and the zero-tests guard is on.
    • full: all 12 stages.
    • the checked-in default (none): rejected with "no stages", never the full suite.
    • None of the three links the MAUI group.
  • RunTests.Tests.ps1 and CopilotPipeline.Tests.ps1: 68 passed. The new tests cover:
    • the request encoding, and that it rejects any value that would inject YAML;
    • the branch naming;
    • the pipeline guards, including the MAUI group exclusion and that no other pipeline builds run-tests/*;
    • that the dispatch path uses no Azure DevOps credentials.

For reviewers

  • For PRs from forks, the dispatch build runs the PR's code with the branch-build job token (public Build Service, which also has Manage build queue) instead of a fork PR build's read-only token. It still gets no variable-group secrets and none of the PR's pipeline YAML. If that's not acceptable, I can limit dispatch to same-repo PRs.
  • Restoring the federated identity (a Conditional Access exclusion for the workload identity) would also fix /review, which this PR does not touch.
  • After merge, run-tests/* branches appear in dotnet/maui. No ruleset, Actions workflow or other pipeline reacts to them.

…* branch instead of Azure DevOps credentials
Copilot AI balanced review requested due to automatic review settings November 22, 2026 02:37
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 22, 2026 02:37 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40504

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40504"

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 22, 2026 02:37 — with GitHub Actions Inactive
@azure-pipelines

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active issues can leave PR checks stuck in progress after partial update failures and prevent workflow retries because reruns reuse an existing dispatch branch name.

2 open issues
What changed in this PR

This PR moves /review run-tests from Azure DevOps token-based queueing to run-tests/* branch dispatch, addressing the sign-in failures left unresolved by the earlier CI fixes.

Changes:

  • Creates dispatch commits with PR code, target-branch pipelines and workflows, and validated test selections.
  • Adds dispatch-branch pipeline support while excluding the MAUI secret variable group.
  • Tracks build results through a PR-head check and comment, and removes expired dispatch branches.
File Description
eng/​pipelines/​common/​variables.yml Excludes MAUI variable-group secrets from dispatch builds.
eng/​pipelines/​common/​run-tests-request.yml Defines the dispatch request format with a run-nothing default.
eng/​pipelines/​ci-uitests.yml Triggers dispatch builds and applies selected tests and PR lanes.
.github/​workflows/​review-run-tests-results.yml Grants permissions for check updates and branch cleanup.
.github/​workflows/​review-run-tests-dispatch.yml Replaces Azure authentication with branch-creation permissions.
.github/​scripts/​Update-RunTestsResults.ps1 Updates dispatch checks and deletes expired branches.
.github/​scripts/​RunTests.Tests.ps1 Adds request, branch, check-update, and pipeline-guard coverage.
.github/​scripts/​RunTests.Common.ps1 Adds dispatch helpers and removes token-based queueing helpers.
.github/​scripts/​Invoke-RunTests.ps1 Creates dispatch commits, waits for builds, and publishes tracking checks.
.github/​scripts/​Get-RunTestsContext.ps1 Checks target-branch dispatch support before test selection.

Review effort: Balanced

Comment on lines +210 to +211
$branch = Get-RunTestsDispatchBranch -PrNumber $PrNumber -RunId $DispatchRunId
Invoke-GhJson @('-X', 'POST', "repos/$Repository/git/refs") -InputJson (@{ ref = "refs/heads/$branch"; sha = $commit.sha } | ConvertTo-Json) | Out-Null
Comment on lines +78 to +81
if ($comment.body -match '<!-- review-run-tests-check: (\d{1,20}) -->') {
$check = New-RunTestsRunCheckUpdate -Build $build -Results $results
$check | ConvertTo-Json -Depth 5 | & gh api -X PATCH "repos/$Repository/check-runs/$($Matches[1])" --input - | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Warning "Could not update check run $($Matches[1])." }
@kubaflo

kubaflo commented Nov 22, 2026

Copy link
Copy Markdown
Contributor Author

/review expert-review-with-gate

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Warning

Your /review gate couldn't start due to a transient error triggering the pipeline. The command was left unacknowledged so scheduled recovery can retry it automatically; you can also comment /review gate again in a minute.

@github-actions github-actions Bot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 22, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 3 findings

Reviewed commit ba3da0f.

Source assessment incomplete (6/7 dimensions); finding count is partial.

See inline comments for details.

$payload = @{ body = $updated } | ConvertTo-Json
$payload | & gh api -X PATCH "repos/$Repository/issues/comments/$($comment.id)" --input - | Out-Null
if ($LASTEXITCODE -ne 0) { Write-Warning "Could not update comment $($comment.id)." }
if ($comment.body -match '<!-- review-run-tests-check: (\d{1,20}) -->') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Logic and Correctness Verification - If the completed-comment PATCH succeeds but the check PATCH fails, the comment is already marked done while the PR's selected-run check stays in_progress. Scheduled sweeps exclude done comments, and a retry with -HeadSha exits at the unchanged-comment guard before reaching this update. Synchronize the check independently of comment changes and retain retryable state until both publications succeed. Add a mocked regression test for successful comment publication followed by failed check publication and retry; the new tests exercise only the check payload.

[string]$AzdoToken = $env:RUN_TESTS_AZDO_TOKEN,
[int]$DefinitionId = 313,
# Names the dispatch branch; unique per request.
[string]$DispatchRunId = $(if ($env:GITHUB_RUN_ID) { $env:GITHUB_RUN_ID } else { [string][DateTimeOffset]::UtcNow.ToUnixTimeSeconds() }),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Logic and Correctness Verification - GitHub workflow reruns preserve GITHUB_RUN_ID, so retrying a dispatch job after it creates the branch but fails during check/comment publication derives the same branch name. The unconditional create-ref POST then fails because that ref already exists, preventing recovery through Re-run failed jobs even with unchanged PR inputs. Include the run attempt in the branch identity (and update validation/cleanup), or verify and reuse the existing dispatch. Add a mocked regression test for retry with the prior ref already present; the branch-name test covers only a first attempt.

param([Parameter(Mandatory)]$PullRequest, [Parameter(Mandatory)]$MergeCommit, [Parameter(Mandatory)]$Selection)
# The trusted half: pipeline and workflows of the target branch (or the trial's pipeline source).
$sourceSha = if ($PipelineSource) { $PipelineSource } else {
(Invoke-GhJson @("repos/$Repository/git/ref/heads/$($PullRequest.base.ref)")).object.sha

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Input and Path Correctness - The PR API's base.ref is interpolated into the gh api endpoint without URI encoding. For a valid target branch such as main#test, #test becomes a URL fragment, so this resolves heads/main instead. The returned SHA passes the subsequent validation and supplies the dispatch commit's pipelines, workflows and parent from the wrong branch (or the request fails if the prefix branch is absent). Encode the ref before constructing the endpoint; the existing encoded preflight request does not protect this separate lookup.

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — ba3da0fb Deployed Nov 22, 2026 by kubaflo via conclusion #3194
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants