Support `RUN --gpus ...` or equivalent.

[gabrieldemarmiesse]

COPY --from=docker/buildx#239 ./ ./

Hi and thanks for this awesome tool!

We make heavy use of buildkit (soon buildx) in Tensorflow Addons to run our tests. It's awesome to be able to run all those tests in parralel in isolated environements, even on remote servers. With the docker cache it's awesome, no need to pull the dependencies again... etc.

Something that we're stuck with though is that we can't run our gpu tests in a dockerfile (like we do for all the other tests).

Would it be possible to support gpus for an entire stage? Or even better for a single RUN instruction?

Thanks a lot!

[tonistiigi]

cc @tiborvass

[kniec]

How about the privileged frontend in buildkit?
That should provide you with a privileged build step that has access to all devices, including the GPUs.

[sskorol]

@kniec @tonistiigi @tiborvass I want to extend the original #1800 issue. In my case, I'm trying to build libraries with CUDA in docker on Jetson Nano/Xavier boards based on their official images. By default, they provide CUDA access only in runtime. But it could be also enabled for the build process as well by adding a default nvdia runtime in docker daemon settings. As far as I understood, this option activates CUDA libs' auto-mounting into containers while building. And it perfectly works until I enable buildkit feature. When it comes to steps that require CUDA libraries, it fails to locate them. Assuming such nvidia runtime specifics, would privileged or insecure options still work, or buildkit completely overrides the default runtime?

[tonistiigi]

@sskorol Buildkit vendored into Docker will only use runc runtime always atm. If you replace the binary it may work with other runtimes as well but this behavior is not recommended as it means the same dockerfile might behave differently on separate machines. #1800 does not assume any custom runtime dependency. You can either just access/initialize devices directly in the insecure mode in runc or we build in some initialization helpers like docker/cli#1714 .

[kniec]

Can this be triggered via frontend?

[gregruthenbeck]

Found that it wants the libs from the win-side:

LD_LIBRARY_PATH=/usr/lib/wsl/lib/ python3 -c "import torch; print(torch.cuda.is_available())"

[kognat-docs]

Is there no progress on this after a month? This makes custom CUDA acceleration operators for PyTorch in WSL2 Docker impossible.

The footprint this leaves in the Deep Learning use case is large, I think users just one more reason to leave Microsoft and use Linux for Deep Learning engineering