GitHub Community
[2025-07-23] Incident Thread #167276
-
|
❗ An incident has been declared:Incident with Actions Hosted Runners Subscribe to this Discussion for updates on this incident. Please upvote or emoji react instead of commenting +1 on the Discussion to avoid overwhelming the thread. Any account guidance specific to this incident will be shared in thread and on the Incident Status Page. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 1 reply
-
UpdateWe're investigating delays provisioning Actions Hosted Runners. Customers may see delays over 5 minutes for jobs starting. |
Beta Was this translation helpful? Give feedback.
-
UpdateWe are applying mitigations to increase Actions Hosted Runners capacity, and are starting to see recovery. We’re monitoring to ensure continued stability. |
Beta Was this translation helpful? Give feedback.
-
Incident ResolvedThis incident has been resolved. |
Beta Was this translation helpful? Give feedback.
-
|
We are seeing all our GHA's stuck in queue still. When can we expect our GHAs to start triggering? |
Beta Was this translation helpful? Give feedback.
-
|
My issue is resolved: https://github.com/orgs/community/discussions/167300 |
Beta Was this translation helpful? Give feedback.
-
Incident SummaryOn July 23rd, 2025, from approximately 14:30 to 16:30 UTC, GitHub Actions experienced delayed job starts for workflows in private repos using Ubuntu-24 standard hosted runners. This was due to resource provisioning failures in one of our datacenter regions. During this period, approximately 2% of Ubuntu-24 hosted runner jobs on private repos were delayed. Other hosted runners, self-hosted runners, and public repo workflows were unaffected. To mitigate the issue, additional worker capacity was added from a different datacenter region at 15:35 UTC and further increased at 16:00 UTC. By 16:30 UTC, job queues were healthy and service was operating normally. Since the incident, we have deployed changes to improve how regional health is accounted for when allocating new runners, and we are investigating further improvements to our automated capacity scaling logic and manual overrides to prevent a recurrence. |
Beta Was this translation helpful? Give feedback.
Incident Summary
On July 23rd, 2025, from approximately 14:30 to 16:30 UTC, GitHub Actions experienced delayed job starts for workflows in private repos using Ubuntu-24 standard hosted runners. This was due to resource provisioning failures in one of our datacenter regions. During this period, approximately 2% of Ubuntu-24 hosted runner jobs on private repos were delayed. Other hosted runners, self-hosted runners, and public repo workflows were unaffected.
To mitigate the issue, additional worker capacity was added from a different datacenter region at 15:35 UTC and further increased at 16:00 UTC. By 16:30 UTC, job queues were healthy and service was operating normally. Since the incident…