Docker overlay native diff disabled resulting image build failure

[hligit]

Description

Docker storage native overlay diff is disabled by default

Impact

Some Docker image build failed for error layer doesn't exist.

Environment and steps to reproduce

1. Set-up:

Launch an EC2 instance from Flatcar-stable-2765.2.5.

2. Task:

Build a Docker image with below Dockerfile:

# echo hello > 1
# cat <<EOF > Dockerfile
FROM scratch AS base
COPY 1 /1
FROM base
# "useless" copy works ...
COPY --from=base /1 /1
# ... and the COPY statement after it fails
COPY 1 /2
EOF

# docker build -t mytest .
Sending build context to Docker daemon  3.072kB
Step 1/5 : FROM scratch AS base
 --->
Step 2/5 : COPY 1 /1
 ---> 6fe1b1fd9cab
Step 3/5 : FROM base
 ---> 6fe1b1fd9cab
Step 4/5 : COPY --from=base /1 /1
 ---> 12bd28054ffd
Step 5/5 : COPY 1 /2
failed to export image: failed to create image: failed to get layer sha256:ec22588dcfceca289a1df0a426789512fd73560b296eb6a64b60b14b316be019: layer does not exist

Notice overlay module features redirect_dir and metacopy are enabled and docker reports native overlay diff is disabled.

# cat /sys/module/overlay/parameters/redirect_dir
Y
# cat /sys/module/overlay/parameters/metacopy
Y
# docker info
......
Server:
......
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false

Expected behavior

Expect the Docker image be built successfully

Additional information

The same Docker image can be built without issue by turning off overlay module features redirect_dir and metacopy.

# echo N > /sys/module/overlay/parameters/redirect_dir
# echo N > /sys/module/overlay/parameters/metacopy
# systemctl restart docker
# docker build -t mytest .
Sending build context to Docker daemon  3.072kB
Step 1/5 : FROM scratch AS base
 --->
Step 2/5 : COPY 1 /1
 ---> Using cache
 ---> 6fe1b1fd9cab
Step 3/5 : FROM base
 ---> 6fe1b1fd9cab
Step 4/5 : COPY --from=base /1 /1
 ---> Using cache
 ---> 12bd28054ffd
Step 5/5 : COPY 1 /2
 ---> bf943f2dd295
Successfully built bf943f2dd295
Successfully tagged mytest:latest

Notice now Docker reports that native overlay diff is enabled

# docker info
......
Server:
......
 Server Version: 19.03.15
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
......

Please add any information here that does not fit the above format.

The change that enabled the two overlay features is flatcar-archive/coreos-overlay#691. Related issue #170

[sayanchowdhury]

This looks like an upstream issue to me.

Related:
issue: moby/moby#37965
PR: moby/moby#34342

[tormath1]

I can't reproduce on recent version of Flatcar:

core@localhost ~ $ cat Dockerfile
FROM scratch AS base
COPY 1 /1
FROM base
# "useless" copy works ...
COPY --from=base /1 /1
# ... and the COPY statement after it fails
COPY 1 /2
core@localhost ~ $ docker build -t mytest .
[+] Building 1.1s (7/7) FINISHED                                                                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 243B                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                       0.0s
 => => transferring context: 93B                                                                                                                                                        0.0s
 => [base 1/1] COPY 1 /1                                                                                                                                                                0.0s
 => [stage-1 1/2] COPY --from=base /1 /1                                                                                                                                                0.0s
 => [stage-1 2/2] COPY 1 /2                                                                                                                                                             0.0s
 => exporting to image                                                                                                                                                                  1.0s
 => => exporting layers                                                                                                                                                                 0.9s
 => => writing image sha256:01d1aad5537244302db7b6358daa5e71bad32f3ef587d26b9095b59bd9694d6a                                                                                            0.0s
 => => naming to docker.io/library/mytest                                                                                                                                               0.0s
core@localhost ~ $ cat /sys/module/overlay/parameters/redirect_dir
Y
core@localhost ~ $ cat /sys/module/overlay/parameters/metacopy
Y
core@localhost ~ $ docker info
Client:
 Version:    28.0.1
 ...
core@localhost ~ $ cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=4344.0.0
VERSION_ID=4344.0.0
BUILD_ID=2025-05-27-1621
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 4344.0.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:4344.0.0:*:*:*:*:*:*:*"

Let's close this. Thanks for reporting it!