Invalid not-modified ETag

[zevisert]

This is essentially #905 again. I looked at #2420 and #1298. But I think what I'm seeing is most like #905. Reading through that, it seems like #1159 should have fixed this because I can't tell what's different in my case.

The file I'm trying to add is hosted by Vercel, and I'd be surprised if they aren't using ETag correctly.

Inspecting with curl, Vercel seems to be using the ETag header correctly

Original request

✅ We get an ETag to use for this resource

$ curl --http1.1 --silent --location \
    --dump-header  /dev/stderr --output /dev/null \
    https://install.python-poetry.org

HTTP/1.1 200 OK
Date: Wed, 27 Apr 2022 18:53:12 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 26678
Connection: keep-alive
cache-control: public, max-age=0, must-revalidate
etag: "d040e95ab7ab5295e98d5f5e12a2f879"
access-control-allow-origin: *
content-disposition: inline; filename="install-poetry.py"
Age: 599187
X-Vercel-Cache: HIT
server: Vercel
x-vercel-id: pdx1:pdx1::dzr7p-1651085592481-1e07247457a4
strict-transport-security: max-age=63072000
Accept-Ranges: bytes

With unambiguous If-None-Match

✅ We don't get an Etag back, because it's not ambiguous which one was matched

$ curl --http1.1 --silent --location \
    --dump-header  /dev/stderr --output /dev/null \
    --header 'If-None-Match: "d040e95ab7ab5295e98d5f5e12a2f879"' \
    https://install.python-poetry.org

HTTP/1.1 304 Not Modified
Date: Wed, 27 Apr 2022 18:55:43 GMT
Connection: keep-alive
x-vercel-cache: HIT
server: Vercel
x-vercel-id: pdx1::mp7vq-1651085743092-b8af595ac812
cache-control: public, max-age=0, must-revalidate

With ambiguous If-None-Match

✅ We get an Etag back to differentiate which tag was matched

$ curl --http1.1 --silent --location \
    --dump-header  /dev/stderr --output /dev/null \
    --header 'If-None-Match: "d040e95ab7ab5295e98d5f5e12a2f879","other-etag-123"' \
    https://install.python-poetry.org

HTTP/1.1 304 Not Modified
Date: Wed, 27 Apr 2022 18:58:36 GMT
Connection: keep-alive
cache-control: public, max-age=0, must-revalidate
etag: "d040e95ab7ab5295e98d5f5e12a2f879"
X-Vercel-Cache: HIT
server: Vercel
x-vercel-id: pdx1:pdx1::8j9tz-1651085916877-389908ccce11

Repro

# syntax=docker/dockerfile:1.3
# Filename: Dockerfile.etag 

ARG POETRY_SCRIPT="https://install.python-poetry.org"
ARG POETRY_HOME="/opt/poetry"

# Stage goal: install pyphon poetry
FROM python:3.10-slim as poetry
ARG POETRY_HOME
ARG POETRY_SCRIPT

ADD ${POETRY_SCRIPT} ${POETRY_HOME}/poetry-install.py
RUN python ${POETRY_HOME}/poetry-install.py --yes

Build output: docker --log-level=debug build --file=Dockerfile.etag --progress=plain .

DEBU[0000] serving grpc connection                      
#1 [internal] load build definition from Dockerfile.etag
#1 sha256:8e2f0894ec1007cc98e89f152cfad6ea805de0c09a296addaf024d1f362c1450
#1 transferring dockerfile: 43B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 sha256:4fcc1877f062c46ec29df6e319bbdae35648a8490e2c7058076660002871e192
#2 transferring context: 2B done
#2 DONE 0.1s

#3 resolve image config for docker.io/docker/dockerfile:1.3
#3 sha256:4fcd28d33487ad029eab28c03869fd56295f3902c713674c129a438f7a780653
#3 ...

#4 [auth] docker/dockerfile:pull token for registry-1.docker.io
#4 sha256:216d0f0164c37d73fe8ef61d35e74a6b150578dfb536c11610262d78c1f88050
#4 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1.3
#3 sha256:4fcd28d33487ad029eab28c03869fd56295f3902c713674c129a438f7a780653
#3 DONE 6.7s

#5 docker-image://docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2
#5 sha256:7862c1373501a4a9cd96ccd04641bb1d96c86d034546e74fe74585e3dd12f952
#5 CACHED

#7 [internal] load .dockerignore
#7 sha256:902dcd56192bdc4532fb533e0880b09a035c103874b67a2aaef4c76d466a46ff
#7 DONE 0.0s

#6 [internal] load build definition from Dockerfile.etag
#6 sha256:1260236a4a3f9cdedb89bdb87be4e0aef2ded0319fc50a926e74931ba5c3dcd1
#6 DONE 0.0s

#8 [internal] load metadata for docker.io/library/python:3.10-slim
#8 sha256:b8cce7183299ea8471dad4044ce489ad9730ef208466a01382bd3a85a0492215
#8 ...

#9 [auth] library/python:pull token for registry-1.docker.io
#9 sha256:a39031416f712a973884166c09d90520548e6fe3d95d86c70205bd350de75929
#9 DONE 0.0s

#8 [internal] load metadata for docker.io/library/python:3.10-slim
#8 sha256:b8cce7183299ea8471dad4044ce489ad9730ef208466a01382bd3a85a0492215
#8 DONE 1.1s

#10 [1/3] FROM docker.io/library/python:3.10-slim@sha256:ad719a882ac0594bdc92abccff22e5b6713bbf9893320264fd9e8affd8a6f470
#10 sha256:deee634555ec41d495cfd1d5572a9a8c14ff3635ab130fe14a614310eec5ce5c
#10 DONE 0.0s

#11 https://install.python-poetry.org
#11 sha256:7e80bb8fe6911ec39be75cf653b92b1e03a777b3f757e83a4b53814d5ec43573
#11 ERROR: invalid not-modified ETag: "d040e95ab7ab5295e98d5f5e12a2f879"
------
 > https://install.python-poetry.org:
------
failed to load cache key: invalid not-modified ETag: "d040e95ab7ab5295e98d5f5e12a2f879"

OS Info:

$ wsl.exe --status

Default Distribution: Ubuntu
Default Version: 2

Windows Subsystem for Linux was last updated on 2022-03-27
WSL automatic updates are on.

Kernel version: 5.10.102.1

$ docker version

Client: Docker Engine - Community
 Cloud integration: v1.0.23
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:21 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Let me know if I can provide more details

[tonistiigi]

I tested the Dockerfile you posted with v0.10.1 and seemed to work fine.

[steve36871]

I am having the same issue today.

[zevisert]

@tonistiigi Is there a way to tell what buildkit version is included with docker-desktop? We use what is essentially the k8s rootless-statefulset example here, those daemons are all v0.10.1 and seem to be working fine (consecutive back to back builds work fine). It's just the builds with docker build ... that I can get to consistently fail right now.

buildctl --addr 'kube-pod://buildkitd-0?namespace=buildkit' \
    build \
    --frontend dockerfile.v0 \
    --local context=. \
    --local dockerfile=.

I did have this issue last week, but it our daemons were on v0.9.3 at the time. It may as well be fixed, and maybe we can close this if we can prove that docker desktop for windows hasn't shipped buildkit v0.10.1+ yet.

[zevisert]

Actually, I tracked that down myself, my local docker engine is 20.10.14 and it's release notes point to using docker/buildx@v0.8.1, which lists buildkit@v0.10.0-rc2 in it's vendored modules.

It seems that buildx shipped a release candidate, but even then both docker build and buildctl I think are running against buildkit v0.10+. Both should have #2629 included, which seems like the last change to the Etag code. I can't tell why one works and the other doesn't.

Even with that said, the ETag in question here isn't weak, so even v0.9.3 should have been correct in this case, but I still saw the error then.

[gh640]

I met the same issue running ADD https://install.python-poetry.org .... I also could reproduce the error with the Dockerfile zevisert posted:

failed to load cache key: invalid not-modified ETag: "d040e95ab7ab5295e98d5f5e12a2f879"

My environment:

❯ docker version
Client:
 Cloud integration: v1.0.23
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:49:20 2022
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.7.1 (77678)
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:14 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I used the script https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py GitHub hosts instead of https://install.python-poetry.org as a temporary workaround (It worked).

ADD https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py /tmp/install-poetry.py

[tonistiigi]

@zevisert Dockerd 20.10.14 uses buildkit 0.8.* https://github.com/moby/moby/blob/v20.10.14/vendor.conf#L36 . If you use docker buildx you can always switch to container/k8s driver and run any version of buildkit directly.

[zevisert]

We already run buildkitd on k8s, but the problem is only reproducible now with our local development environment, as we use docker compose for that. I guess that our ticket for switching to skaffold gets higher priority. 👍

we can close this if we can prove that docker desktop for windows hasn't shipped buildkit v0.10.1+ yet.

Thanks for the help!

[owenallenaz]

@tonistiigi You stated "If you use docker buildx you can always switch to container/k8s driver and run any version of buildkit directly." How would one do that? I'm hitting this error because the version of docker supported in CircleCI is only up to 20.10.18 and it has a pre v0.10.1 version. I've tried using # syntax=docker/dockerfile:1.4 but that hasn't resolved the issue.

[tonistiigi]

@owenallenaz https://docs.docker.com/build/drivers/docker-container/