docker buildx debug --on=error build not invoking debug shell

[matth65]

The docker buildx debug --on=error command is not invoking the shell command for debugging. In the example below, the build error should trigger buildx to invoke the bash shell. However, buildx is not launching bash. Is there something incorrect with the way I'm using --on=error here?

Command output

$ BUILDX_EXPERIMENTAL=1 docker buildx debug --on=error --invoke=/bin/bash build --progress=plain .
#0 building with "default" instance using docker-container driver

#1 [internal] connecting to local controller
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 114B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/ubuntu:20.04
#3 DONE 0.2s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/4] FROM docker.io/library/ubuntu:20.04@sha256:8e5c4f0285ecbb4ead070431d29b576a530d3166df73ec44affc1cd27555141b
#5 resolve docker.io/library/ubuntu:20.04@sha256:8e5c4f0285ecbb4ead070431d29b576a530d3166df73ec44affc1cd27555141b done
#5 DONE 0.0s

#6 [2/4] RUN echo "hello"
#6 CACHED

#7 [3/4] RUN stop-here
#7 0.078 /bin/sh: 1: stop-here: not found
#7 ERROR: process "/bin/sh -c stop-here" did not complete successfully: exit code: 127
------
 > [3/4] RUN stop-here:
0.078 /bin/sh: 1: stop-here: not found
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:5
--------------------
   3 |     RUN echo "hello"
   4 |     
   5 | >>> RUN stop-here
   6 |     
   7 |     RUN echo "world"
--------------------
ERROR: process "/bin/sh -c stop-here" did not complete successfully: exit code: 127
Launching interactive container. Press Ctrl-a-c to switch to monitor console
Interactive container was restarted with process "n4imofhaz2jazb7sybp2vrvaa". Press Ctrl-a-c to switch to the new container
(buildx) Switched IO

(buildx) 
(buildx) exit
Dockerfile:5
--------------------
   3 |     RUN echo "hello"
   4 |     
   5 | >>> RUN stop-here
   6 |     
   7 |     RUN echo "world"
--------------------
ERROR: process "/bin/sh -c stop-here" did not complete successfully: exit code: 127

Dockerfile

FROM ubuntu:20.04 as base

RUN echo "hello"

RUN stop-here

RUN echo "world"

Docker Version

$ docker version
Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:52 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:52 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Buildx Version

$ docker buildx version
github.com/docker/buildx v0.19.3 48d6a39

OS

$ lsb_release -a
LSB Version:	core-11.1.0ubuntu2-noarch:printing-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

Other Notes

Note that --on=always seems to work when there are no errors in the Dockerfile. See the example below.

$ BUILDX_EXPERIMENTAL=1 docker buildx debug --on=always --invoke=/bin/bash build --progress=plain .
#0 building with "default" instance using docker-container driver

#1 [internal] connecting to local controller
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 99B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/ubuntu:20.04
#3 DONE 0.4s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/3] FROM docker.io/library/ubuntu:20.04@sha256:8e5c4f0285ecbb4ead070431d29b576a530d3166df73ec44affc1cd27555141b
#5 resolve docker.io/library/ubuntu:20.04@sha256:8e5c4f0285ecbb4ead070431d29b576a530d3166df73ec44affc1cd27555141b 0.0s done
#5 DONE 0.0s

#6 [2/3] RUN echo "hello"
#6 CACHED

#7 [3/3] RUN echo "world"
#7 0.075 world
#7 DONE 0.1s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Launching interactive container. Press Ctrl-a-c to switch to monitor console
Interactive container was restarted with process "jc5zuho6qe3foxrn62t2r4bry". Press Ctrl-a-c to switch to the new container
root@buildkitsandbox:/# 
root@buildkitsandbox:/# ls
bin   dev  home  lib32  libx32  mnt  proc  run   srv  tmp  var
boot  etc  lib   lib64  media   opt  root  sbin  sys  usr
root@buildkitsandbox:/# exit
exit
Switched IO
(buildx) exit

[andyneff]

This appears to be a new behavior.

• In version 0.17.1, it is working as expected. The command in invoked
• In 0.19.2 or later, it is not working as expected. The command is not invoked.

After some digging:

• This is working in 0.18.0 RC1
• This is not working in 0.18.0 RC2

[andyneff]

This appears to have broken in #2722.

• buildx-v0.18.0-rc1-17-g2bdf451b - Works
• buildx-v0.18.0-rc1-20-g202c390f - Does not work

[andyneff]

Specifically something in this commit breaks the invoke on error. It's working correctly in 1060328

[crazy-max]

@andyneff Thanks for investigating on this issue, this is indeed a regression, sorry about that. #2958 should fix this.