Twitter rate limiting forever #8864
Description
After running a long overnight job, I'm seeing a scenario where Twitter will get stuck in a rate limit forever until getting a KeyboardInterrupt:
[twitter][info] Waiting until 01:04:52 (rate limit)
[twitter][info] Waiting until 01:19:53 (rate limit)
[twitter][info] Waiting until 01:34:54 (rate limit)
[twitter][info] Waiting until 01:49:55 (rate limit)
[twitter][info] Waiting until 02:04:56 (rate limit)
[twitter][info] Waiting until 02:19:57 (rate limit)
[twitter][info] Waiting until 02:34:59 (rate limit)
[twitter][info] Waiting until 02:50:00 (rate limit)
[twitter][info] Waiting until 03:05:01 (rate limit)
[twitter][info] Waiting until 03:20:03 (rate limit)
[twitter][info] Waiting until 03:35:04 (rate limit)
[twitter][info] Waiting until 03:50:05 (rate limit)
[twitter][info] Waiting until 04:05:06 (rate limit)
[twitter][info] Waiting until 04:20:08 (rate limit)
[twitter][info] Waiting until 04:35:09 (rate limit)
[twitter][info] Waiting until 04:50:10 (rate limit)
[twitter][info] Waiting until 05:05:11 (rate limit)
[twitter][info] Waiting until 05:20:12 (rate limit)
[twitter][info] Waiting until 05:35:13 (rate limit)
[twitter][info] Waiting until 05:50:14 (rate limit)
[twitter][info] Waiting until 06:05:15 (rate limit)
[twitter][info] Waiting until 06:20:16 (rate limit)
[twitter][info] Waiting until 06:35:17 (rate limit)
[twitter][info] Waiting until 06:50:18 (rate limit)
[twitter][info] Waiting until 07:05:19 (rate limit)
[twitter][info] Waiting until 23:14:22 (rate limit)
[twitter][info] Waiting until 23:29:25 (rate limit)
[twitter][info] Waiting until 23:44:29 (rate limit)
[twitter][info] Waiting until 23:59:32 (rate limit)
[twitter][info] Waiting until 00:14:35 (rate limit)
[twitter][info] Waiting until 00:29:38 (rate limit)
[twitter][info] Waiting until 00:44:41 (rate limit)
[twitter][info] Waiting until 00:59:45 (rate limit)
[twitter][info] Waiting until 01:14:48 (rate limit)
This continues without ever getting a valid request in from here on.
I'm trying to track if gallery-dl isn't waiting long enough when it detects a rate limit, or if additional steps need to be taken based on data in the request.
Twitter is sending rate limit headers in the response:
reply: 'HTTP/1.1 429 Too Many Requests\r\n'
header: Date: Sat, 10 Jan 2026 22:03:18 GMT
header: Content-Type: application/json; charset=utf-8
header: Content-Length: 82
header: Connection: keep-alive
header: perf: 7402827104
header: vary: Origin
header: vary: accept-encoding
header: pragma: no-cache
header: Server: cloudflare envoy
header: expires: Tue, 31 Mar 1981 05:00:00 GMT
header: Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
header: last-modified: Sat, 10 Jan 2026 22:03:18 GMT
header: x-frame-options: SAMEORIGIN
header: Content-Encoding: gzip
header: x-transaction-id: 855b70515ca15f4d
header: x-xss-protection: 0
header: x-rate-limit-limit: 500
header: x-rate-limit-reset: 1768083433
header: content-disposition: attachment; filename=json.json
header: x-tfe-preserve-body: true
header: x-content-type-options: nosniff
header: x-rate-limit-remaining: 497
header: x-twitter-response-tags: BouncerCompliant
header: x-response-time: 15
header: origin-cf-ray: 9bbf76b35cc3e738-EWR
header: strict-transport-security: max-age=631138519; includeSubdomains
header: x-served-by: t4_a
header: cf-cache-status: DYNAMIC
but it looks like it may not be honoring the values it's giving. Twitter is a bad actor in general, so it's hard to say what it's doing here. This might need an exponential backoff, or a different kind of workaround.
It also looks like gallery-dl is ignoring --sleep-429 in favor of using Twitter's header values, making this difficult to work around in the client.