Skip to content

[CI] /review run-tests: recover from a rejected AZDO_TRIGGER_PAT and give working guidance while Entra blocks sign-in - #40501

Open
kubaflo wants to merge 1 commit into
mainfrom
kubaflo-fix-review-run-test-ui-trigger
Open

kubaflo wants to merge 1 commit into
mainfrom
kubaflo-fix-review-run-test-ui-trigger

Conversation

@kubaflo

@kubaflo kubaflo commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Target Branch

This PR is for Initial base
New features, API changes, or .NET 12 work (including .NET 12 fixes) main
.NET 11 servicing fixes net11, for review
Older servicing, release integration, infrastructure, or documentation The branch designated for that work

CI workflow infrastructure follow-up to #40492 and #40485. The workflow runs only from main, so this targets main.

For .NET 11 fixes, only a maintainer retargets the approved PR to the active
inflight lane and merges it after CI validates that final base. Do not merge
individual fixes directly into net11; agents must not retarget or merge.

Description of Change

Why /review run-tests still doesn't queue UI tests after #40492

I checked the raw logs of every review-run-tests-dispatch run:

When (UTC) Credential Result
11-20 10:40 to 17:56 Federated identity (OIDC) Builds queued (1691859, 1691861, 1691867, 1691911, 1692548, 1692555)
11-20 19:57 OIDC Signed in; the queue request returned 400 (token accepted)
11-20 21:05 to 11-21 20:11 OIDC AADSTS53003: Access has been blocked by Conditional Access policies on every attempt
11-21 14:07 (run 44912312706) AZDO_TRIGGER_PAT (#40492) 401 (Unauthorized) from Azure DevOps. The secret was removed afterwards, so later runs used OIDC again and got AADSTS53003.
  • The token request didn't change. The sign-in body is identical between 80c7d3f1 (worked) and 33339ed5 (blocked). [CI] /review run-tests: report and retry Microsoft Entra sign-in failures #40485 only added retries and reporting.
  • The block is not limited to run-tests. review-trigger.yml (the /review gate) uses the same identity and failed at 2026-11-22T02:04Z with the same AADSTS53003 (run 44974360023).
  • The PAT was refused, not malformed. A bogus PAT gets 203 and a sign-in page from Azure DevOps. The real secret got 401, which fits a token Azure DevOps recognizes but refuses under the same tenant policy.
  • A bot can't post /azp run. github-actions[bot] has none permission on dotnet/maui, so the Azure Pipelines app would reject the comment. That app can't take the selected-category parameters anyway.

Conclusion: No repository change can queue the targeted run while the tenant policy blocks every Azure DevOps credential from GitHub-hosted runners. A tenant admin has to exempt the trigger identity (AZDO_TRIGGER_CLIENT_ID) from the Conditional Access policy, or approve the PAT. Once that's done, the existing flow works unchanged.

What this PR fixes

  • Trimmed PAT: AZDO_TRIGGER_PAT is trimmed, so a pasted newline or space no longer breaks Basic auth.
  • Rejected PAT falls back: Azure DevOps can answer 401, or 203 with its HTML sign-in page. Before, a 203 was read as a queued build. Both now count as a rejected PAT, and the run falls back to the federated identity. Timeouts and other errors still fail without a retry, so a build is never queued twice.
  • Useful comment on sign-in failure: The PR comment used to say "comment /review run-tests again later", which can't work while the policy is in place. It now says to use /azp run maui-pr-uitests for the full suite, and that targeted runs return once sign-in works again. A plain 401 from the queue call gets the same message.
  • Docs: .github/docs/trigger-azdo-pipeline-setup.md covers AADSTS53003 and the PAT failure modes.

Validation

  • Real Azure DevOps replay: I replayed the failed request for [CollectionView] Add ItemTapped event and ItemTappedCommand #40437 (context and GPT-6.1 Sol proposal from run 44944036321) through the changed script. I used a bogus PAT with surrounding whitespace and a fake OIDC endpoint, and stubbed the gh writes so nothing was posted. Results:
    • The trimmed PAT was masked, and Azure DevOps answered 203.
    • The script logged "Azure DevOps rejected AZDO_TRIGGER_PAT (HTTP 203) … Trying the federated identity."
    • Entra sign-in was retried 4 times, then the new /azp run maui-pr-uitests guidance was written to the comment and warning.
  • Pester: Invoke-Pester ./.github/scripts/RunTests.Tests.ps1 passed 58, failed 0.
  • Parse check: The PowerShell parser reports 0 errors.

Issues Fixed

Follow-up to #40492 (the PAT fallback didn't work) and #40485.

…point requesters at /azp run while sign-in is blocked

- Trim the PAT secret; treat Azure DevOps' 401 or 203 sign-in page as a rejected PAT
  and fall back to the federated identity instead of failing (a 203 was previously
  misread as a queued build).
- On any sign-in failure, the PR comment now says to use /azp run maui-pr-uitests
  for the full suite instead of retrying /review run-tests, which cannot succeed
  while Conditional Access blocks the identity (AADSTS53003).
- Document AADSTS53003 and the PAT failure modes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 22, 2026 02:13
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool November 22, 2026 02:13 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 40501

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 40501"

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Credential fallback can repeat a non-idempotent queue POST after a successful response without a build ID, risking duplicate builds.

3 open issues
What changed in this PR

This CI follow-up to #40492 and #40485 improves credential recovery and failure guidance for /review run-tests, which queues targeted MAUI UI tests.

Changes:

  • Trims the optional PAT and adds federated-identity fallback when Azure DevOps rejects it.
  • Detects invalid queue responses and directs users to /azp run maui-pr-uitests when sign-in fails.
  • Documents Conditional Access errors and PAT troubleshooting.
File Description
.github/​scripts/​Invoke-RunTests.ps1 Adds PAT recovery, response validation, and sign-in failure guidance.
.github/​docs/​trigger-azdo-pipeline-setup.md Adds troubleshooting guidance for blocked sign-in and rejected PATs.

Review effort: Balanced

$build = Invoke-RestMethod -Method Post -Uri "$($base)?api-version=7.1" -Headers $headers `
-ContentType 'application/json' -Body $body -TimeoutSec 60 -StatusCodeVariable patStatus
# Azure DevOps answers an unusable PAT with 203 and its HTML sign-in page.
if ($patStatus -eq 203 -or -not $build.id) { throw [System.UnauthorizedAccessException]::new("HTTP $patStatus") }
| `403` from AzDO REST API | Missing permissions | Ensure the identity has "Queue builds" on the specific pipeline AND Basic access level. |
| `OIDC environment variables not available` | Missing `id-token: write` permission | Add `permissions: { id-token: write }` at the **job** level (not workflow level). |
| `Failed to get Azure AD token` | Wrong client_id/tenant_id or federated credential mismatch | Verify secrets match the MI's Client ID and Tenant ID. Check federated credential subject matches the actual OIDC claim. |
| `AADSTS53003: Access has been blocked by Conditional Access policies` | A tenant Conditional Access policy refuses token issuance to the identity from GitHub-hosted runners (seen from 2026-11-20 for `/review`, `/review gate` and `/review run-tests`). Retrying or changing the request does not help. | A tenant admin must exempt the identity (`AZDO_TRIGGER_CLIENT_ID`) from the policy. Meanwhile `/azp run maui-pr-uitests` (Azure Pipelines app) runs the full UI suite. |
if ($patStatus -eq 203 -or -not $build.id) { throw [System.UnauthorizedAccessException]::new("HTTP $patStatus") }
} catch {
$status = [int]$_.Exception.Response?.StatusCode
if ($_.Exception -isnot [System.UnauthorizedAccessException] -and $status -ne 401) { throw }
@kubaflo

kubaflo commented Nov 22, 2026

Copy link
Copy Markdown
Contributor Author

/review expert-review-with-gate

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Nov 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Warning

Your /review gate couldn't start due to a transient error triggering the pipeline. The command was left unacknowledged so scheduled recovery can retry it automatically; you can also comment /review gate again in a minute.

@github-actions github-actions Bot removed the s/agent-review-in-progress AI review is currently running for this PR label Nov 22, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Review — 2 findings

Reviewed commit aee0254.

See inline comments for details.

$build = Invoke-RestMethod -Method Post -Uri "$($base)?api-version=7.1" -Headers $headers `
-ContentType 'application/json' -Body $body -TimeoutSec 60 -StatusCodeVariable patStatus
# Azure DevOps answers an unusable PAT with 203 and its HTML sign-in page.
if ($patStatus -eq 203 -or -not $build.id) { throw [System.UnauthorizedAccessException]::new("HTTP $patStatus") }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Logic and Correctness Verification - A successful PAT queue response with an empty body or missing id is treated as an authentication rejection: the catch clears $build and the federated path repeats the same non-idempotent POST. If the first request created a build but its response omitted the ID, a second build is queued and the first remains untracked, outside selection/SHA verification and cancellation. Restrict fallback to confirmed authentication rejection (401/203); a successful response without an ID must report an unknown queue outcome without another POST. Add mocked dispatcher regression tests asserting queue-call counts for trimmed PAT success, 401/203 fallback, and timeout/500 or successful responses missing an ID. RunTests.Tests.ps1 currently tests the token helper and dispatcher source text, not this queue path.

$queueError = "Could not sign in to Azure DevOps ($(Format-RunTestsText $_.Exception.Message 250)). $signInHelp"
} elseif ([int]$_.Exception.Response?.StatusCode -eq 401) {
Write-Warning "Azure DevOps refused the credentials: $($_.Exception.Message)"
$queueError = "Azure DevOps refused the trigger credentials (HTTP 401). $signInHelp"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[moderate] Logic and Correctness Verification - This 401 handler also catches the cancellation PATCH after a successful queue POST. If the returned selection or sourceVersion is wrong and cancellation returns 401, the unexpected build has already been queued and was not cancelled. The new guidance nevertheless asserts 'Nothing was queued', and the outer catch discards the build ID. Distinguish cancellation failure after confirmed queueing from authentication failure before queueing; retain the build link and report that the unexpected run may still be active.

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — aee0254b Deployed Nov 22, 2026 by kubaflo via conclusion #3192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants