'429 Too Many Requests' when downloading from flickr with sleep 3-10

[francoisbruneau]

Running gallery-dl 1.30.8 on Mac OS Tahoe 26.0
Created custom api keys following documentation, and added them to ~/.config/gallery-dl/config.json .

Even with --sleep parameter set to 3-10, I am getting code 429 warnings after a couple of minutes. This is strange since those sleep times should keep me way below the Flickr API rate limit of 3600 hits per hours.

Here's gallery-dl verbose output:

[flickr][debug] Sleeping 7.31 seconds (download)
[urllib3.connectionpool][debug] https://live.staticflickr.com:443 "GET /65535/54563472386_47218c24e9_o_d.jpg HTTP/1.1" 429 117
[downloader.http][warning] '429 Too Many Requests' for 'https://live.staticflickr.com/65535/54563472386_47218c24e9_o_d.jpg'
[download][error] Failed to download flickr_54563472386.jpg

My config file is the following:

% cat config.json
{
    "extractor": {
        "flickr": {
            "api-key": "my-api-key",
            "api-secret": "my-api-secret",
            "access-token": "my-access-token",
            "access-token-secret": "my-access-token-secret"
        }
    },
    "downloader": {

    },
    "output": {

    },
    "postprocessor": {

    }
}

Does anyone have an idea what could be triggering these "Too Many Requests" warnings in such circumstances?

[mikf]

There was a similar issue previously (#6360), which was "fixed" by using download URLs and adding _d to each URL. (0e18fa3)

Maybe this is now what's causing the 429 errors,
or Flickr has increased its limits even more.

[francoisbruneau]

FYI The issue still occurs on latest version (1.31.6) after running for a couple of minutes.

Debug output is as follows:

[flickr][debug] Sleeping 4.58 seconds (download)
[urllib3.connectionpool][debug] Resetting dropped connection: live.staticflickr.com
[urllib3.connectionpool][debug] https://live.staticflickr.com:443 "GET /65535/54859115493_6c39986235_k_d.jpg HTTP/1.1" 429 117
[downloader.http][warning] '429 Too Many Requests' for 'https://live.staticflickr.com/65535/54859115493_6c39986235_k_d.jpg'
[download][error] Failed to download lou-o-bedlam -  [54859115493].jpg
[flickr][debug] Sleeping 5.42 seconds (download)
[urllib3.connectionpool][debug] Resetting dropped connection: live.staticflickr.com

[jdmaltz]

FYI this still seems to be an issue on (1.3.10) even with proper api-keys and cookies.

[ebdoran]

Maybe try something like this:

gallery-dl --sleep 2-4 --sleep-request 1.5 --retries 20 --sleep-429 60 "URL"

--sleep 2-4: Randomizes the wait between 2 and 4 seconds for every image.
--sleep-429 60: This is a newer flag in gallery-dl. If it hits a 429, it will stop everything for 60 seconds before trying the next file.
--sleep-request 1.5: Slows down the "talk" between the tool and the Flickr API.

[jdmaltz]

it still occurs after about 20 or 30 links, I tried to increase these numbers but no joy. This occurs with or without a VPN.

[francoisbruneau]

There was a similar issue previously (#6360), which was "fixed" by using download URLs and adding _d to each URL. (0e18fa3)

Maybe this is now what's causing the 429 errors, or Flickr has increased its limits even more.

@mikf Removing _dfrom URLs does fix the issue indeed. Been running for 30 minutes without hitting a single 429, whereas I would have one from the first minute on before the change.

[jdmaltz]

@francoisbruneau how do you remove the _d from urls? is it a commandline argument or a code change?

[francoisbruneau]

@jdmaltz Code change. Basically reverting 0e18fa3 .

[mikf]

Removing _dfrom URLs does fix the issue indeed

ae25d976

[jdmaltz]

ok so should I wait for the next release you use this or is there a way I can make my local install do this?

[mikf]

@jdmaltz
You are probably better off installing the dev version. Either from

https://github.com/gdl-org/builds/releases/tag/2026.04.16

or with
gallery-dl --update-to dev

or with
python -m pip install -U --force-reinstall --no-deps https://codeberg.org/mikf/gallery-dl/archive/master.tar.gz

[jdmaltz]

Thank you very much @mikf and @francoisbruneau that fixed it!