tagsortmode set to "none" fails when used with Amazon ECR

[diranged]

Describe the bug
Amazon ECR's /tags/list endpoint returns back the list of tags not sorted by date (it might be lexicographical). To get around this, we're trying to set tagsortmode: none to have the image updater make subsequent API calls to figure out when each particular blob was uploaded. When we do this, we get errors back from the AWS ECR endpoint throwing 403's on the calls:

time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-6dc596c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-a13b52e"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-faf148f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-435ae68"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-8b0251a"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-0da85a5"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-5e2d313"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-9dc110b"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-c7841fe"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:d7123eb7a7121961948503f687efece8ea6775d52717ecebe13c064686b873dd"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c99ed1523e8da73f3f22aee87b053e53a8502678dd74f96bb981fec0cbcdc9e0"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:129549edceb37fc64fe9d75f41340b36596ed65d93ea77a9a5d2c08e802caf5e"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:d7123eb7a7121961948503f687efece8ea6775d52717ecebe13c064686b873dd"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c2145f571d8940d08a65bb525ec5570c393277e7064d29edcf6fd75f55b2b042"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:04616d20fc3485406963893785af497862b3df5ad5aac9f93f218cc2234f1a1c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:2f1ca1c0340c449942bac1cf40f79a0cf330ecb6cc2ee623e4e6f5b0dc4334f0"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8e8d999649ab7a60f514787083d340ca6cd8b76a563bde5e7629d6e188fbc401"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-cca0372: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-9bfbeef (operation 22/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-de66320"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:0eb8a8b378010e73b2a7e81daab79d9a6fcad7ebe6fc5ca9f5cbe19a10c1452f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:e02ccbfcfd0e1a65116d7f196ce00b073d2bf1256436b2df10e2f0e21e4c610f"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:0115d417b3566bafcd95a9d130e336bd7320069f11fb0a04d45d8457ec621415"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:54a4ee3284c2c588bd2ad6bda905ee536bfde4e3dbcdb54c59fd595beaade704"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8c97f70618eccbb73831496e0142c528b27996964013695f883cd0c3cc711ba1"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8d53a0eafb8613965ce1e3e085d0d19c53d6fab33ea310699b7e0a543a5c0c9c"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:81ec223defc03ce6c2b24165a7fbd520fc0b6ac168ffb7c81a82ad1a5676bd33"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-9f971e8: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:c99ed1523e8da73f3f22aee87b053e53a8502678dd74f96bb981fec0cbcdc9e0\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-39c6706 (operation 23/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-9bfbeef"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-9dc110b: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:129549edceb37fc64fe9d75f41340b36596ed65d93ea77a9a5d2c08e802caf5e\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"
time="2021-06-15T22:30:43Z" level=trace msg="Getting manifest for image myorg/myapp:main-7f2db5e (operation 24/24)"
time="2021-06-15T22:30:43Z" level=trace msg="https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/manifests/main-39c6706"
time="2021-06-15T22:30:43Z" level=error msg="error fetching metadata for myorg/myapp:main-8b0251a: could not get metadata: Head \"https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:8e8d999649ab7a60f514787083d340ca6cd8b76a563bde5e7629d6e188fbc401\": http: non-successful response (status=403 body=\"\")"
time="2021-06-15T22:30:43Z" level=trace msg="released semaphore and terminated waitgroup"
time="2021-06-15T22:30:43Z" level=trace msg="acquired metadata semaphore"

However, when we try CURL'ing the endpoint's reported in the URLs above, we can get the response just fine:

bash$ TOKEN=$(aws --region us-west-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')
$ curl --head -i -H "Authorization: Basic $TOKEN" https://1234.dkr.ecr.us-west-2.amazonaws.com/v2/myorg/myapp/blobs/sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b
HTTP/1.1 200 OK
Content-Length: 7394
Content-Type: application/vnd.docker.container.image.v1+json
Docker-Content-Digest: sha256:252116da9bb49eba91dbcd7874c20a36eccc432439eaad356c3026d6166a081b
Docker-Distribution-Api-Version: registry/2.0
Date: Tue, 15 Jun 2021 22:36:36 GMT

To Reproduce
Follow the instructions I posted originally at #112 (comment)... then switch tagsortmode to none

Expected behavior
I expect the API calls to work because I can execute them myself via Curl..

Version
Latest master build as of 1d19e6a

[jannfis]

Unfortunately, I do not have an AWS account to test ECR functionality so I am not able to set up a reproduction scenario here.

I have a few questions, tho:

1. Do you receive a 403 for every meta data request, or only for some?

2. Do you use the same AWS account when using curl and the request succeeds? HTTP 403 is usually emitted if the request was authenticated successfully, but does not have permissions to access the requested resource.

3. I'm not sure how ECR handles request limits, but do you have any quota set to your registry maybe for max pulls/second? Other registries will usually emit HTTP 429 if the limit is exceeded, but I'm unsure how this is with ECR. Keep in mind that the Image Updater uses concurrency extensively, so low limits may be reached quickly.

[diranged]

@jannfis I just figured the issue out - and perhaps its time to start a document somewhere in the image updater docs talking about ECR. We were missing the ecr:BatchCheckLayerAvailability permission on our IAM Permissions Boundary. For anyone in the future finding this issue, here's the permissions that are now working with tagsortmode: none:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:ListTagsForResource"
            ],
            "Resource": [
                "*"
            ],
            "Sid": "AmazonECRReadAccess"
        }
    ]
}

[AndresJulia]

hey @diranged how did you manage to get the awscli inside argocd-image-updater?