Skip to content

fix: stagger nightly e2e schedules to reduce API throttling#8141

Merged
awesomenix merged 1 commit into
mainfrom
aganeshkumar/stagger-nightly-e2e-schedules
Mar 20, 2026
Merged

fix: stagger nightly e2e schedules to reduce API throttling#8141
awesomenix merged 1 commit into
mainfrom
aganeshkumar/stagger-nightly-e2e-schedules

Conversation

@ganeshkumarashok

Copy link
Copy Markdown
Contributor

Summary

  • Staggers the nightly e2e pipeline schedules by 1 hour each to avoid concurrent Azure Compute Gallery API calls that cause ResourceCollectionRequestsThrottled errors
  • Linux e2e stays at 0 8 * * * (midnight PST, unchanged)
  • GPU e2e moves to 0 9 * * * (1am PST)
  • Windows e2e moves to 0 10 * * * (2am PST)

Test plan

  • Verify each pipeline picks up the new cron schedule after merge
  • Monitor next nightly runs for reduced ResourceCollectionRequestsThrottled failures in GPU e2e (pipeline 443878)

All three e2e pipelines (linux, gpu, windows) were scheduled at the
same cron time (0 8 * * * UTC), causing them to hit the Azure Compute
Gallery API simultaneously and trigger ResourceCollectionRequestsThrottled
errors. Stagger by 1 hour each:
- Linux e2e:   0 8 * * * (midnight PST, unchanged)
- GPU e2e:     0 9 * * * (1am PST)
- Windows e2e: 0 10 * * * (2am PST)
@awesomenix awesomenix enabled auto-merge (squash) March 20, 2026 00:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR staggers the nightly Azure Pipelines E2E schedules to reduce concurrent Azure Compute Gallery API calls that have been triggering ResourceCollectionRequestsThrottled failures.

Changes:

  • Move GPU E2E schedule from 0 8 * * * to 0 9 * * * (1am PST).
  • Move Windows E2E schedule from 0 8 * * * to 0 10 * * * (2am PST).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
.pipelines/e2e-gpu.yaml Shifts nightly GPU E2E cron to 1 hour after the main Linux E2E run.
.pipelines/e2e-windows.yaml Shifts nightly Windows E2E cron to 2 hours after the main Linux E2E run.

@awesomenix awesomenix merged commit fb4ab18 into main Mar 20, 2026
24 checks passed
@awesomenix awesomenix deleted the aganeshkumar/stagger-nightly-e2e-schedules branch March 20, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants