Build cannot export to registry on localhost

[sudo-bmitch]

While setting up a local demo, I found that buildx is unable to access a registry server running on localhost. I can see that my registry server is listening on port 5000 with a curl command. This is with the docker-container driver, so I suspect that container is not using the host namespace and therefore cannot see any services running on localhost.

$ docker buildx build -f Dockerfile.buildx --target debug --platform linux/amd64,linux/arm64 -t localhost:5000/bmitch-public/golang-hello:buildx1 --output type=registry .
[+] Building 3.1s (24/24) FINISHED
 => [internal] load build definition from Dockerfile.buildx                                                                       0.4s
 => => transferring dockerfile: 39B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                 0.5s
 => => transferring context: 34B                                                                                                  0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/debian:latest                                                      0.7s
 => [linux/amd64 internal] load metadata for docker.io/tonistiigi/xx:golang                                                       0.7s
 => [linux/amd64 internal] load metadata for docker.io/library/golang:1.12-alpine                                                 0.8s
 => [internal] load build context                                                                                                 0.2s
 => => transferring context: 105B                                                                                                 0.0s
 => [linux/amd64 debug 1/2] FROM docker.io/library/debian@sha256:118cf8f3557e1ea766c02f36f05f6ac3e63628427ea8965fb861be904ec35a6  0.0s
 => => resolve docker.io/library/debian@sha256:118cf8f3557e1ea766c02f36f05f6ac3e63628427ea8965fb861be904ec35a6f                   0.0s
 => [linux/amd64 xgo 1/1] FROM docker.io/tonistiigi/xx:golang@sha256:4703827f56e3964eda6ca07be85046d1dd533eb0ed464e549266c10a4cd  0.0s
 => => resolve docker.io/tonistiigi/xx:golang@sha256:4703827f56e3964eda6ca07be85046d1dd533eb0ed464e549266c10a4cd8a29f             0.0s
 => [linux/amd64 dev 1/6] FROM docker.io/library/golang:1.12-alpine@sha256:cee6f4b901543e8e3f20da3a4f7caac6ea643fd5a46201c3c2387  0.0s
 => => resolve docker.io/library/golang:1.12-alpine@sha256:cee6f4b901543e8e3f20da3a4f7caac6ea643fd5a46201c3c2387183a332d989       0.0s
 => CACHED [linux/amd64 dev 2/6] COPY --from=xgo / /                                                                              0.0s
 => CACHED [linux/amd64 dev 3/6] RUN apk add --no-cache git ca-certificates                                                       0.0s
 => CACHED [linux/amd64 dev 4/6] RUN adduser -D appuser                                                                           0.0s
 => CACHED [linux/amd64 dev 5/6] WORKDIR /src                                                                                     0.0s
 => CACHED [linux/amd64 dev 6/6] COPY . /src/                                                                                     0.0s
 => CACHED [linux/amd64 build 1/1] RUN CGO_ENABLED=0 go build -ldflags '-w -extldflags -static' -o app .                          0.0s
 => CACHED [linux/amd64 debug 2/2] COPY --from=build /src/app /app                                                                0.0s
 => CACHED [linux/amd64 dev 2/6] COPY --from=xgo / /                                                                              0.0s
 => CACHED [linux/amd64 dev 3/6] RUN apk add --no-cache git ca-certificates                                                       0.0s
 => CACHED [linux/amd64 dev 4/6] RUN adduser -D appuser                                                                           0.0s
 => CACHED [linux/amd64 dev 5/6] WORKDIR /src                                                                                     0.0s
 => CACHED [linux/amd64 dev 6/6] COPY . /src/                                                                                     0.0s
 => CACHED [linux/amd64 build 1/1] RUN CGO_ENABLED=0 go build -ldflags '-w -extldflags -static' -o app .                          0.0s
 => CACHED [linux/amd64 debug 2/2] COPY --from=build /src/app /app                                                                0.0s
 => ERROR exporting to image                                                                                                      1.4s
 => => exporting layers                                                                                                           0.1s
 => => exporting manifest sha256:fb7fb1aacd96dcd6c9a6d2654fb2a9cf7692c3ebfd4d15bd1dd397d38713a589                                 0.2s
 => => exporting config sha256:8c443cd193baf5e58914a1ad50d8311e25f7d9ac86772a6ab2df99ed7f4ef6f3                                   0.2s
 => => exporting manifest sha256:d63ec5c6531662c1185b1cc90755573a1bbc1b4754998181847598433fe30e5e                                 0.2s
 => => exporting config sha256:3838e43619611f78eedbc6604fedc3ab134f2beb4225d45d10bb37698603189e                                   0.2s
 => => exporting manifest list sha256:4c8694f90dda751d32ccbd9e48bdeba1042467f07bd0193378e254141e7464ec                            0.2s
 => => pushing layers                                                                                                             0.0s
------
 > exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request: Head http://localhost:5000/v2/bmitch-public/golang-hello/blobs/sha256:8c443cd193baf5e58914a1ad50d8311e25f7d9ac86772a6ab2df99ed7f4ef6f3: dial tcp 127.0.0.1:5000: connect: connection refused

$ curl -sSLk https://localhost:5000/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

$ docker buildx ls
NAME/NODE     DRIVER/ENDPOINT             STATUS  PLATFORMS
new *         docker-container
  new0        unix:///var/run/docker.sock running linux/amd64
default       docker
  default     default                     running linux/amd64

Note this is a low priority issue for me, I'd much rather see #80 solved.

[tonistiigi]

I guess we can just switch the container to host netns. It is privileged anyway so shouldn't be a problem but we probably want to allow configuring rootless in this driver as well in the future.

@tiborvass

[sudo-bmitch]

This is a rare enough situation for me that I'd be more than happy with an option somewhere, and the default is to have more isolation inside a container. Perhaps the endpoint/build context could include some added options, e.g.

unix:///var/run/docker.sock,net=host,rootless=false

I have the same issue I guess.
I am running a registry on one of my own servers.
All my builds are pushed to this registry for my homelab which setup is meant to be permanent.
Unfortunately I am facing the mentioned issue.

[dt-rush]

Just got hit by this. Is there any possible workaround? I don't want to have to publish to dockerhub to be able to experiment with multi-arch builds. --load doesn't work either for multi-arch (aka "manifest") builds atm, as of: #59

[tonistiigi]

Use create --driver-opt network=host

[dt-rush]

cc @tonistiigi

For anyone reaching this thread in the future:

The version bundled with docker 19.03.1 (the one i'm running on my dev machine) is:

$ docker buildx version
github.com/docker/buildx v0.2.2-10-g3f18b65-tp-docker 3f18b659a09804c738226dbf6bacbcae54afd7c6

The above version doesn't have --driver-opt.

My best bet seems to be to just use the "releases" page of this repo to download the buildx binary, since the bundling in Docker CLI is somewhat funky anyhow, in light of:

I actually couldn't get docker buildx working on a fresh debian install VM which had docker v19.03.3 on it, despite having docker version reporting Experimental: true (I got "docker buildx is not a docker command").

[dt-rush]

@tonistiigi any idea how to pass the --insecure flag for communicating with the local registry? I now have buildx build --push 127.0.0.1:5000/myimage:tag failing with "http: server gave HTTP response to HTTPS client"

[dt-rush]

(I have "insecure-registries": ["127.0.0.1:5000"] in /etc/docker/daemon.json)

[dt-rush]

(If this doesn't have a solution I'm happy to open a separate issue for it)

[tonistiigi]

@dt-rush Use --config https://github.com/docker/buildx#--config-file with http: true set for the insecure registry.

[dt-rush]

@tonistiigi should we not be using the insecure-registries info from docker's config? or have a comparable flag?

[dt-rush]

Seems a bit cumbersome to have to maintain a config file in a globally-accessible location, for each build no matter where we run buildx from, rather than using the existing system associated with docker push

I could try to PR this?

[tonistiigi]

This flag is only for buildx create not for every build. If you use docker driver (without create) it is already using the docker config.

[dt-rush]

@tonistiigi Ah, I see, you use it once during create for the builder. That's nice. But my last question, I hope is: what do you mean by "if you use docker driver"? The default builder for docker, without buildx --platform capabilities? In that case, of course that uses the docker config, but doesn't build multi-arch images, hahaha

[tonistiigi]

@dt-rush https://github.com/docker/buildx#building-with-buildx explains drivers and you can see it with buildx inspect. docker driver is similar to DOCKER_BUILDKIT=1 docker build, so for example, it does understand --platform but will not allow you to specify multiple atm.