Pushing cache to "registry" cache with multi-node builder only uploads cache from one node

[potiuk]

Originally reported at moby/buildkit#2758

It's been confirmed by @tonistiigi that this is a problem with buildx multi-node builder.

When you are building a multi-platform image with multiple builders (to avoid emulation) and use --cache-to type=registry, the resulting registry cache only contains cache for the platform that that was build last.

I tried to utilize buildkit to build Apache Airflow (https://github.com/apache/airflow) multi-platform images. I am using latest buildkit and latest docker:.

Hosts used for the multi-platform builds

I have two builder hosts:

1. AMD builder (Linux Mint 20.3) with buildx plugin installed
   github.com/docker/buildx v0.7.1 0584689 - docker builder created there

2. ARM Builder (Mac Pro M1 late 2021) with DockerDesktop 4.6.0 (with buildx pre-install installed) - with new Virtualization framework enabled.

Builder configuration

I configured my buildx builds to use both builders. I connected the MacOS builder to the Linux Host via forwarded docker socket and I am running all my multi-platform builds from the Linux Host.

This is the builders I see with docker buildx ls:

airflow_cache       docker-container                     
  airflow_cache0    unix:///var/run/docker.sock running  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
  airflow_cache1    tcp://127.0.0.1:2375        running  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Build command

I want to build a multi-platform image for both ARM and AMD and I want to do it in a single buildx command. Additionally I want to store cache for both platfiorms in the same image but with :cache tag.

My image is multi-staging, so I want to push cache for all stages (hence mode=max)

The (simpliified) command to run the build is:

docker buildx build --progress=default --pull --platform linux/amd64,linux/arm64 \
    --cache-from=ghcr.io/potiuk/airflow/main/ci/python3.7:cache \
    --cache to=type=registry,ref=ghcr.io/potiuk/airflow/main/ci/python3.7:cache,mode=max  \
    --push 
    -t ghcr.io/potiuk/airflow/main/ci/python3.7:latest --target main . -f Dockerfile.ci

While the ghcr.io/potiuk/airflow/main/ci/python3.7:latest image is perfectly fine (nice, multiplatform image), the ghcr.io/potiuk/airflow/main/ci/python3.7:cache image only contains cache by the "LAST" build image - i.e if the AMD image was faster to build and push cache, the cache from the ARM builder pushed later seems to override the AMD cache stored there.
I could not find any way to somehow merge those two caches (especially that I cannot specifiy two different cache destination for each of the platforms). This renders the --cache-to,type=registrry essentially useless for multiplatform builds.

I reverted to "inline" mode and it seems to work, but I would really love to keep the latest cache in a separate tag of the image.

[Nick-0314]

The cache of arm64 overwrites the cache of amD64, causing only one of the two platforms to be available. Using inline mode was a bit expensive for us to manage local storage, there were frequent disk space runs out, and the cache would disappear after the BuildKit container was restarted

[Nick-0314]

@tonistiigi Can this problem be circumvented by adding the default schema suffix to cache-to-Registry? For example: "repo/ubuntu:cache-linux-arm64". Is this easy to develop? It is currently possible to define multiple cache-from, but once cache-to can be suffixed, I can cache-from multiple schemas

[potiuk]

This is what I am planning to do - but then such multiplatform image cannot be prepared with single buildx command because you can specify only one --cache-to when you run single multi-platform build even with remote builders

Which renders buildx feature of preparing multi-platform image with remote builders in a single command pretty useless.

[potiuk]

What I actually plan to do is do it in two steps (until it is fixed):

1. Build a single multiplatform image and push it without cache
2. Run separate two steps to AGAIN build and push (only cache) in two separate commands for two platforms separately

This is quite an overhead though the build cacje in builders will be reused so the overhead for running 3 commands instead of one should be bearable.