Add configurable time interval/delay between fetching for updated images

[starcraft66]

Is your feature request related to a problem? Please describe.
I've started using the digest update strategy for a few of my images and found that it quickly runs me into docker hub pull rate limits, I'd like argo image updater to only check for new images every few hours instead of every few minutes to avoid running into this rate limit.

Describe the solution you'd like
I'd like to see a setting of some sort to configure the interval at which argocd image updater checks for image updates. Perhaps this setting could also be per-registry since not all registries have the same rate limits.

Describe alternatives you've considered
I know that I can set up a "pull-through" registry mirror but if I understand correctly, it would still generate the same amount of api calls when constantly fetching digests for tags right?

[jannfis]

Hi @starcraft66 - there are already a couple of configuration options to control that behavior:

• The --interval command line option let's you specify the duration between the checks on a global level (e.g. 24h or 30m)
• On a per registry level, you can control the rate limit for requests per seconds to the registry (by using the limit: <number> parameter in registries.conf) - this alas is an under-documented feature and defaults to 20 currently (and only considers requests per seconds currently, and therefore may not be suitable in your case).

Also, keep in mind that the Image Updater currently runs one big loop for checking all images at once. I think it would be nice to decouple this to a per application basis, and then you could have different intervals per application (or even per registry).

The Docker Hub pull limits are imho a very unfortunate design decision for us, because every fetch of a manifest (not the image blob itself) counts as a pull from the point of view of Docker Hub. But we don't have another choice to retrieve the meta data for an image that we need in case of digest or latest updater methods. My general advice would be to just another public registry (e.g. quay.io) without those strict pull limits, but I also understand that this is not always possible.

I think the digest method uses 3 manifest pulls per tag (have not verified), per check. Given the default interval of 2 minutes, that would make 90 per hour already for a single tag to be tracked. That's already near the limit of 100 per 6 hours. :(

[starcraft66]

Thanks for the detailed reply, the --interval option is exactly what I was looking for. For some reason I missed the documentation for it.

My general advice would be to just another public registry (e.g. quay.io) without those strict pull limits, but I also understand that this is not always possible.

I definitely would go with that if I were hosting my own images but unfortunately I am mostly using this project to keep random public images up to date.

I left some feedback on the Docker Hub issue tracker btw.
docker/hub-feedback#2093

[starcraft66]

A docker hub developer has left some insightful feedback on the issue I opened and linked in my last comment, it looks like we can use HTTP HEAD requests and compare the value of the Docker-Content-Digest header to perform updates with the digest strategy without completely destroying the pull rate limit.

[jannfis]

it looks like we can use HTTP HEAD requests and compare the value of the Docker-Content-Digest header [...]

That's interesting. I'm wondering if that is a standard implemented by all the registries, or just specific to Docker Hub. Anyway, thank you for digging that up, it's definitely something to consider. I also wonder whether they have a similar header for the build date of the manifest in the HTTP headers. Will investigate.

[cannonpalms]

Also, keep in mind that the Image Updater currently runs one big loop for checking all images at once. I think it would be nice to decouple this to a per application basis, and then you could have different intervals per application (or even per registry).

Pull limits aren't the limiting factor anymore if this is implemented, so I think this is a better path forward versus continuing to attempt to poll for tags on all registries and adding an additional HEAD request in front of every GET.

[jannfis]

Hey, apparently, Image Updater was misbehaving with the digest strategy and fetched metadata for all tags in a given repository from the registry, while it only should have fetched metadata for the single tag we're interested in.

This has been fixed with PR #337, which will be released with v0.11.2 shortly.

[starcraft66]

Hahaha no wonder this thing was constantly blowing through my docker hub rate limit.

I've since moved to Renovate's Kustomize update strategy since it fits my needs better by opening pull requests but I'll keep this project in mind for totally automated updates if ever I need that again, thanks!

[jannfis]

Closed by #337