feat: Add support to enable backoff when rate limited by the Github API #6644

andrewhibbert · 2025-03-24T16:39:51Z

Adds support to backoff when a rate limit error occurs as per handle rate limit errors appropriately:

If the retry-after response header is present, you should not retry your request until after that many seconds has elapsed.
If the x-ratelimit-remaining header is 0, you should not make another request until after the time specified by the x-ratelimit-reset header. The x-ratelimit-reset header is in UTC epoch seconds.

[N/A]When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
[N/A] A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #6643

Relates to kedacore/keda-docs#1557

Signed-off-by: andrewhibbert <[email protected]>

zroubalik

This is great addition, I wonder whether we should try to add this capabilities to all scalers that use HTTP client to optain metrics?

WDYT @kedacore/keda-maintainers

zroubalik · 2025-03-26T13:27:17Z

/run-e2e github
Update: You can check the progress here

andrewhibbert · 2025-05-23T17:15:28Z

Hey, just wandering if there is any update on this?

JorTurFer · 2025-05-27T20:41:19Z

I think that this is nice but I'm worried about if the waiting time is so long, as it'll look as an issue in KEDA (nothing will happen) when it's a desired behaviour

JorTurFer · 2025-05-27T20:42:27Z

pkg/scalers/github_runner_scaler.go

 func (s *githubRunnerScaler) GetMetricsAndActivity(ctx context.Context, metricName string) ([]external_metrics.ExternalMetricValue, bool, error) {
-	queueLen, err := s.GetWorkflowQueueLength(ctx)
+	if s.metadata.enableBackoff {
+		wait, waitDuration := s.shouldWaitForRateLimit()


Let's introduce some debug logs here, announcing that we are waiting. If someone is debugging why KEDA doesn't scale, it can be discovered from logs easily

There are debug logs in shouldWaitForRateLimit() currently. Are these sufficient or are you looking for something different?

That's totally true, mb

I've been thinking about this and I'm not fully sure if we should wait when 429. What will it happen if a user burns the hour quota in just a few minutes? In that scenario, the process will stuck here waiting and wairting. From operator metrics pov, the loop will be blocked, but from metrics server pov, we will open more and more routines which will stuck here.
Maybe we can include a max waiting time of IDK, 15 seconds. Other option that comes to my mind is to use the current implementation you have done to skip requests meanwhile the rate limit has been exceded to reduce the load.
@kedacore/keda-core-contributors ?

would it make sense that scaler caches the last metric value and returns that while it's waiting to stop being rate-limitted? That way KEDA stack won't be growing with sleeping goroutines.

Maybe we can include a max waiting time of IDK, 15 seconds. Other option that comes to my mind is to use the current implementation you have done to skip requests meanwhile the rate limit has been exceded to reduce the load.

I like this approach.

Hi, I have made a change so that it returns the previous queue length rather than rechecking whilst it is rate limited. Following the best practices (https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api?apiVersion=2022-11-28#handle-rate-limit-errors-appropriately) which says not to try again until after it has reset. Let me know however if you do want a maximum waiting time, I think however this should be much longer or configurable to reduce the likelihood of a ban from the API

JorTurFer

Nice improvement! Just a minor nit inline

rickbrouwer · 2025-07-23T09:10:05Z

Hi @andrewhibbert, I see there is still a merge conflict.

Signed-off-by: andrewhibbert <[email protected]>

…t_backoff

andrewhibbert · 2025-08-01T16:14:05Z

Fixed merge conflicts

rickbrouwer · 2025-08-01T16:45:14Z

/run-e2e github
Update: You can check the progress here

Signed-off-by: andrewhibbert <[email protected]>

Copilot

Pull Request Overview

This PR adds support for backing off when rate limited by the GitHub API, implementing GitHub's recommended best practices for handling rate limits. The implementation respects both Retry-After headers and X-RateLimit-Reset headers to determine appropriate backoff periods.

Adds enableBackoff configuration option to GitHub runner scaler
Implements rate limit detection and backoff logic using previous queue length during rate limit periods
Updates rate limit error handling to use structured RateLimit type instead of inline parsing

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
pkg/scalers/github_runner_scaler.go	Core implementation of rate limit backoff logic and RateLimit struct
pkg/scalers/github_runner_scaler_test.go	Test coverage for rate limit backoff functionality
tests/scalers/github_runner/github_runner_test.go	Integration test configuration update to enable backoff
CHANGELOG.md	Documentation of the new feature

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-18T09:50:05Z

pkg/scalers/github_runner_scaler.go

+	remaining, _ := strconv.Atoi(header.Get("X-RateLimit-Remaining"))
+	reset, _ := strconv.ParseInt(header.Get("X-RateLimit-Reset"), 10, 64)
+	resetTime := time.Unix(reset, 0)
+
+	if retryAfterStr := header.Get("Retry-After"); retryAfterStr != "" {
+		if retrySeconds, err := strconv.Atoi(retryAfterStr); err == nil {
+			retryAfterTime = time.Now().Add(time.Duration(retrySeconds) * time.Second)


Error handling is ignored when parsing rate limit headers. If these headers contain invalid values, the rate limit logic could behave unexpectedly. Consider handling parsing errors or adding validation.

Suggested change

remaining, _ := strconv.Atoi(header.Get("X-RateLimit-Remaining"))

reset, _ := strconv.ParseInt(header.Get("X-RateLimit-Reset"), 10, 64)

resetTime := time.Unix(reset, 0)

if retryAfterStr := header.Get("Retry-After"); retryAfterStr != "" {

if retrySeconds, err := strconv.Atoi(retryAfterStr); err == nil {

retryAfterTime = time.Now().Add(time.Duration(retrySeconds) * time.Second)

remainingStr := header.Get("X-RateLimit-Remaining")

remaining, err := strconv.Atoi(remainingStr)

if err != nil {

s.logger.V(1).Info(fmt.Sprintf("Invalid X-RateLimit-Remaining header value: %q, error: %v. Defaulting to 0.", remainingStr, err))

remaining = 0

}

resetStr := header.Get("X-RateLimit-Reset")

reset, err := strconv.ParseInt(resetStr, 10, 64)

if err != nil {

s.logger.V(1).Info(fmt.Sprintf("Invalid X-RateLimit-Reset header value: %q, error: %v. Defaulting to 0.", resetStr, err))

reset = 0

}

resetTime := time.Unix(reset, 0)

if retryAfterStr := header.Get("Retry-After"); retryAfterStr != "" {

if retrySeconds, err := strconv.Atoi(retryAfterStr); err == nil {

retryAfterTime = time.Now().Add(time.Duration(retrySeconds) * time.Second)

} else {

s.logger.V(1).Info(fmt.Sprintf("Invalid Retry-After header value: %q, error: %v. Ignoring.", retryAfterStr, err))

Copilot · 2025-09-18T09:50:06Z

pkg/scalers/github_runner_scaler_test.go

+	if queueLen, err := scaler.GetWorkflowQueueLength(context.Background()); err != nil {
+		fmt.Println(err)
+		t.Fail()
+	} else if queueLen != 1 {
+		fmt.Printf("Expected queue length of 1 got %d\n", queueLen)
+		t.Fail()
+	}


Use t.Errorf() or t.Fatalf() instead of fmt.Println() and t.Fail() for better test error reporting. This provides clearer test failure messages and follows Go testing conventions.

Copilot · 2025-09-18T09:50:06Z

pkg/scalers/github_runner_scaler_test.go

+	if queueLen, err := scaler.GetWorkflowQueueLength(context.Background()); err != nil {
+		fmt.Println(err)
+		t.Fail()
+	} else if queueLen != 1 {
+		fmt.Printf("Expected queue length of 1 after rate limit backoff got %d\n", queueLen)
+		t.Fail()
+	}


Use t.Errorf() or t.Fatalf() instead of fmt.Println() and t.Fail() for better test error reporting. This provides clearer test failure messages and follows Go testing conventions.

andrewhibbert added 8 commits March 11, 2025 18:49

feat: Add support to enable backoff when rate limited by the Github API

6c2bc2a

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

dbf4583

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

6f37932

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

f5fc28a

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

b25f1fe

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

70ae11e

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

2210309

Signed-off-by: andrewhibbert <[email protected]>

feat: Add support to enable backoff when rate limited by the Github API

c61bdf1

Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert requested a review from a team as a code owner March 24, 2025 16:39

andrewhibbert changed the title ~~Implement backoff when receive rate limit errors~~ feat: Add support to enable backoff when rate limited by the Github API Mar 24, 2025

Merge branch 'main' into feat_backoff

12e1779

zroubalik reviewed Mar 26, 2025

View reviewed changes

JorTurFer reviewed May 27, 2025

View reviewed changes

JorTurFer approved these changes Jun 1, 2025

View reviewed changes

andrewhibbert requested a review from zroubalik June 10, 2025 11:49

andrewhibbert added 2 commits August 1, 2025 17:09

fix: Stop github scaler from scaling if there are no job labels

0872d8c

Signed-off-by: andrewhibbert <[email protected]>

Merge branch 'feat_backoff' of github.com:andrewhibbert/keda into fea…

ea0fff7

…t_backoff

andrewhibbert added 2 commits August 29, 2025 12:01

Merge branch 'main' into feat_backoff

1cf0105

feat: Add support to enable backoff when rate limited by the Github API

22112c2

Signed-off-by: andrewhibbert <[email protected]>

andrewhibbert requested a review from a team as a code owner August 29, 2025 14:06

keda-automation requested a review from a team August 29, 2025 14:06

feat: Add support to enable backoff when rate limited by the Github API

32b33e0

Signed-off-by: andrewhibbert <[email protected]>

zroubalik requested a review from Copilot September 18, 2025 09:47

Copilot AI reviewed Sep 18, 2025

View reviewed changes

andrewhibbert closed this by deleting the head repository Oct 15, 2025

-	remaining, _ := strconv.Atoi(header.Get("X-RateLimit-Remaining"))
-	reset, _ := strconv.ParseInt(header.Get("X-RateLimit-Reset"), 10, 64)
-	resetTime := time.Unix(reset, 0)
-	if retryAfterStr := header.Get("Retry-After"); retryAfterStr != "" {
-		if retrySeconds, err := strconv.Atoi(retryAfterStr); err == nil {
-			retryAfterTime = time.Now().Add(time.Duration(retrySeconds) * time.Second)
+	remainingStr := header.Get("X-RateLimit-Remaining")
+	remaining, err := strconv.Atoi(remainingStr)
+	if err != nil {
+		s.logger.V(1).Info(fmt.Sprintf("Invalid X-RateLimit-Remaining header value: %q, error: %v. Defaulting to 0.", remainingStr, err))
+		remaining = 0
+	}
+	resetStr := header.Get("X-RateLimit-Reset")
+	reset, err := strconv.ParseInt(resetStr, 10, 64)
+	if err != nil {
+		s.logger.V(1).Info(fmt.Sprintf("Invalid X-RateLimit-Reset header value: %q, error: %v. Defaulting to 0.", resetStr, err))
+		reset = 0
+	}
+	resetTime := time.Unix(reset, 0)
+	if retryAfterStr := header.Get("Retry-After"); retryAfterStr != "" {
+		if retrySeconds, err := strconv.Atoi(retryAfterStr); err == nil {
+			retryAfterTime = time.Now().Add(time.Duration(retrySeconds) * time.Second)
+		} else {
+			s.logger.V(1).Info(fmt.Sprintf("Invalid Retry-After header value: %q, error: %v. Ignoring.", retryAfterStr, err))

feat: Add support to enable backoff when rate limited by the Github API #6644

feat: Add support to enable backoff when rate limited by the Github API #6644

Uh oh!

Conversation

andrewhibbert commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zroubalik left a comment

Choose a reason for hiding this comment

Uh oh!

zroubalik commented Mar 26, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewhibbert commented May 23, 2025

Uh oh!

JorTurFer commented May 27, 2025

Uh oh!

JorTurFer May 27, 2025

Choose a reason for hiding this comment

Uh oh!

andrewhibbert May 30, 2025

Choose a reason for hiding this comment

Uh oh!

JorTurFer Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

JorTurFer Aug 17, 2025

Choose a reason for hiding this comment

Uh oh!

wozniakjan Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zroubalik Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

andrewhibbert Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JorTurFer left a comment

Choose a reason for hiding this comment

Uh oh!

rickbrouwer commented Jul 23, 2025

Uh oh!

andrewhibbert commented Aug 1, 2025

Uh oh!

rickbrouwer commented Aug 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andrewhibbert commented Mar 24, 2025 •

edited

Loading

zroubalik commented Mar 26, 2025 •

edited by github-actions bot

Loading

wozniakjan Aug 18, 2025 •

edited

Loading

andrewhibbert Aug 29, 2025 •

edited

Loading

rickbrouwer commented Aug 1, 2025 •

edited by github-actions bot

Loading