API Rate Exceeded

[rjosborne]

Hi,

Could somebody please help me explain how to avoid the github API rate limit exceeded errors I'm seeing?

I saw them initially when trying to first install aggregator-cli but they went away after logging into github via chrome (coincidence?).

After trying to setup a rule today (meaning various commands such as add / update / map rule etc, I am now getting this error again.

Am I genuinely exceeding the limit for authenticated requests now, or is there something I can / need to do to get the full authenticated allowance of API calls?

Steps to reproduce

• aggregator-cli v0.9.10
• step taken: attempting to update an existing rule

Expected behavior

• the rule should update

Actual behavior

• the update is failing with an API rate limit exceeded error

Diagnostic logs

[2020-06-12 13:08:45Z] aggregator-cli v0.9.10 (build: 0.9.10.0 Release) (c) Copyright © TFS Aggregator Team
[2020-06-12 13:08:45Z] Authenticating to Azure...
[2020-06-12 13:08:46Z] Connected to subscription xxxxxxxxxxxxxxxxxxxxxx
[2020-06-12 13:08:46Z] Checking runtime package versions in GitHub
[2020-06-12 13:08:48Z] API rate limit exceeded for xx.xxx.xxx.xxx. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)

Environment

Please share additional details about your environment.
Version

[giuliov]

Weird. The code does only one calls to GitHub API to get the list of Releases. The call is not authenticated, so the limit is

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests.

A solution might be to ask for credentials and go authenticated, creating more friction.

The other option is to implement the OAuth Flow: it is a lot of non-trivial work, both in terms of code and to handle the secrets. I would be more than happy if you submit a PR with this solution.

[rjosborne]

Thank you for this explanation.

How would you feel about a change to add an additional (optional) "--noupgrade" flag to some of the commands?

I can see a use case where if somebody is running a lot of commands sequentially (say they are wanting to add a bunch of rules, or they're trying to test a rule and having to debug and fix some issues which means multiple update.rule calls).

In this case, while it makes sense to go through the update version process the first time to ensure you have the correct version, doing it several times in quick succession (and therefore burning through the 60 API calls allocated) seems slightly less useful as it's highly unlikely to have changed and therefore just burns through your allocation - potentially to the point of being unable to make any further changes until your allocation gets reset.

[giuliov]

Oh, I understand your scenario and I think an even simpler solution would be to cache for some time the result of that call. We do not release so often to be worth checking more than once a day or so.