Panic at checksum generation (probably when copying a lot of files between build steps)

[Masterrg]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

We are using Jenkins to build a Docker image of our website. This website is static only and gets generated via Svelte.

The build process of the website works without any issues. In the last step we are copying the generated files from the build step into our final image:

# Stage: Server Staging
# Creates a staging ready server
FROM nginx:1.25.4-alpine AS server-staging
#- Move the created files to the nginx html folder
COPY --from=build-staging /app/build /usr/share/nginx/html
#- Move our nginx conf to the nginx image
COPY --from=build-staging /app/conf/nginx/nginx_nopw.conf /etc/nginx/conf.d/default.conf
#- Expose port 80 for nginx
EXPOSE 80

During the first COPY the memory usage of Docker increases until it crashes with a panic.

Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: panic: runtime error: slice bounds out of range [::787386877] with length 268435455
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: goroutine 124146 [running]:
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt/internal/common.UnsafeByteSlice(...)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/internal/common/unsafe.go:26
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt/internal/common.WriteInodeToPage({0xc03bf50540?, 0x1, 0xc2799e4180?}, 0xc2def5e000)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/internal/common/inode.go:81 +0x305
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*node).write(0xc1dc292000?, 0x2eeea?)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/node.go:199 +0x79
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*node).spill(0xc2431020e0)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/node.go:334 +0x26a
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*Bucket).spill(0xc03bf50500)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/bucket.go:786 +0x365
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*Bucket).spill(0xc03bf504c0)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/bucket.go:753 +0x105
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*Bucket).spill(0xc1dc292018)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/bucket.go:753 +0x105
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*Tx).Commit(0xc1dc292000)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/tx.go:204 +0x2db
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: go.etcd.io/bbolt.(*DB).Update(0xc?, 0xc03bf50480)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/go.etcd.io/bbolt/db.go:915 +0xca
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: github.com/moby/buildkit/cache/metadata.(*StorageItem).SetExternal(0xc00191f5c0, {0x556ea28378be, 0x17}, {0xc2b0074000, 0x2eee91e6, 0x2eee91e6})
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/cache/metadata/metadata.go:314 +0xbf
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: github.com/moby/buildkit/cache.(*cacheMetadata).SetExternal(0xc0008d5f50?, {0x556ea28378be?, 0xc0008d5f90?}, {0xc2b0074000?, 0x100?, 0x0?})
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/cache/metadata.go:216 +0x25
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: github.com/moby/buildkit/cache/contenthash.cacheMetadata.SetContentHash(...)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/cache/contenthash/checksum.go:197
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: github.com/moby/buildkit/cache/contenthash.(*cacheContext).save(0xc00191f620)
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/cache/contenthash/checksum.go:293 +0x147
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]: created by github.com/moby/buildkit/cache/contenthash.(*cacheContext).lazyChecksum.func1 in goroutine 124091
Aug 26 11:50:27 testautomation-agent-002 dockerd[3491686]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/cache/contenthash/checksum.go:817 +0x59
Aug 26 11:50:29 testautomation-agent-002 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 26 11:50:29 testautomation-agent-002 systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 26 11:50:29 testautomation-agent-002 systemd[1]: docker.service: Consumed 5min 48.304s CPU time, 46.5G memory peak.
Aug 26 11:50:31 testautomation-agent-002 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 26 11:50:31 testautomation-agent-002 systemd[1]: Starting docker.service - Docker Application Container Engine...

Reproduction

As mentioned we are building a static version of our website and put everything in a Docker image. The website includes more than 600k files and folders. When we are performing "minimal builds" with just some 10k files everything is working as expected, so we are assuming, it might be related to the maximum amount of files.

As workarounds we also tested to

• gzip all files, transfer those files in the final step and extract them into the final nginx folder
• mount the build folder from the previous step and copy everything from there into the final nginx folder

Both workarounds end in the same issue.

Version information

> docker buildx version
github.com/docker/buildx v0.26.1 1a8287f

> docker buildx inspect
Name:   default
Driver: docker

Nodes:
Name:             default
Endpoint:         default
Status:           running
BuildKit version: v0.23.2
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
GC Policy rule#0:
 All:            false
 Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration:  48h0m0s
 Max Used Space: 24.06GiB
GC Policy rule#1:
 All:            false
 Keep Duration:  1440h0m0s
 Reserved Space: 174.2GiB
 Max Used Space: 22.35GiB
 Min Free Space: 87.54GiB
GC Policy rule#2:
 All:            false
 Reserved Space: 174.2GiB
 Max Used Space: 22.35GiB
 Min Free Space: 87.54GiB
GC Policy rule#3:
 All:            true
 Reserved Space: 174.2GiB
 Max Used Space: 22.35GiB
 Min Free Space: 87.54GiB

> docker version
Client: Docker Engine - Community
 Version:           28.3.3
 API version:       1.51
 Go version:        go1.24.5
 Git commit:        980b856
 Built:             Fri Jul 25 11:34:00 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.3.3
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.5
  Git commit:       bea959c
  Built:            Fri Jul 25 11:34:13 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

> docker info
Client: Docker Engine - Community
 Version:    28.3.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.39.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.3.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.41+deb13-amd64
 Operating System: Debian GNU/Linux 13 (trixie)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 125.6GiB
 Name: testautomation-agent-002
 ID: 0a3a9105-4e9f-40b0-9a1d-2ea6adf1d820
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

[crazy-max]

Hum I wonder if bbolt db has been corrupted during OOM kill that interrupts db commits. Can you post the db /var/lib/docker/buildkit/metadata_v2.db here?

Aug 26 11:50:29 testautomation-agent-002 systemd[1]: docker.service: Consumed 5min 48.304s CPU time, 46.5G memory peak.

Seems the system was under very heavy memory pressure during the build. I guess this is a Jenkins node, do you know if there are many builds running on it at the time it happened?

Also what is the size of /app/build in build-staging stage?

[Masterrg]

Hey crazy-max, thank you very much for your support!

Here's the metadata_v2.db
metadata_v2.zip

It's a Jenkins Node but the build process is the only thing running on it. During the copy process the memory usage of docker increases drastically. Also the system has 128GB of RAM, so at least in theory there should be enough memory available.

Regarding the size of /app/build:

#17 [build-staging 3/3] RUN du -hs /app/build
#17 19.55 32G	/app/build
#17 DONE 20.8s

Once again thanks for your support!

[tonistiigi]

Looks like this was likely fixed in etcd-io/bbolt#978 (already updated in master branch)

[thaJeztah]

Yes, was updated in this PR;

• vendor: go.etcd.io/bbolt v1.4.2 #6136