Skip to content

contrib: check nvidia drivers and support non PCI-based GPUs for WSL #5880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 8, 2025
Merged

contrib: check nvidia drivers and support non PCI-based GPUs for WSL #5880

merged 2 commits into from
Apr 8, 2025

Conversation

@crazy-max
Copy link
Member

@crazy-max crazy-max commented Mar 26, 2025 •
edited
Loading

We currently don't support non PCI-based nvidia GPUs like it is on WSL:

$ ls -al /sys/class/drm/card0/device/
total 0
drwxr-xr-x 3 root root    0 Feb 24 14:31 .
drwxr-xr-x 8 root root    0 Feb 24 14:31 ..
-rw-r--r-- 1 root root 4096 Feb 24 14:31 driver_override
drwxr-xr-x 4 root root    0 Feb 24 14:31 drm
-r--r--r-- 1 root root 4096 Feb 24 14:31 modalias
lrwxrwxrwx 1 root root    0 Feb 24 14:41 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Feb 24 14:31 uevent

There is no vendor or device which typically appear for PCI-based GPUs. Since the device looks under bus/platform instead of bus/pci, it is likely a WSL-specific GPU mapping that doesn't expose PCI info.

WSL2 maps GPUs through DirectX, and the presence of /dev/dxg means a GPU is available but we don't know what vendor (could be Intel, Nvidia, AMD, ...).

So as best-effort we can assume that non PCI-based nvidia GPUs for WSL don't need drivers installation and can directly generate the CDI specification.

We also need to check if drivers are installed for both Linux and WSL. We are relying on libcuda install location for this. This is specially useful since buildx request gpu when creating the container builder related: docker/buildx#3063

BUILDKITD_TAGS=nvidia EXPORT_BASE=ubuntu ./hack/images local moby/buildkit
docker buildx create --name nvidia --driver-opt "image=moby/buildkit:local" --buildkitd-flags "--debug" --bootstrap

# syntax=docker.io/docker/dockerfile-upstream:1-labs
FROM ubuntu
RUN --device=nvidia.com/gpu nvidia-smi
$ docker buildx --builder nvidia build .
...
#7 preparing device nvidia.com/gpu
#7 0.000 > /usr/bin/nvidia-smi -L
#7 0.340 GPU 0: NVIDIA GeForce GTX 1050 Ti (UUID: GPU-c5f53f8f-c215-47d2-fa69-d7b4fb13701d)
#7 0.365 > apt-get update
#7 0.435 Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease
#7 0.435 Hit:2 http://archive.ubuntu.com/ubuntu noble InRelease
#7 0.440 Hit:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease
#7 0.454 Hit:4 http://archive.ubuntu.com/ubuntu noble-backports InRelease
#7 0.794 Reading package lists...
#7 2.150 > apt-get install -y gpg
#7 2.158 Reading package lists...
#7 3.510 Building dependency tree...
#7 3.737 Reading state information...
#7 4.060 The following additional packages will be installed:
#7 4.061   dirmngr gpg-agent gpgconf libksba8 libreadline8t64 libsqlite3-0
#7 4.062   pinentry-curses readline-common
#7 4.064 Suggested packages:
#7 4.064   gnupg dbus-user-session libpam-systemd pinentry-gnome3 tor keyboxd scdaemon
#7 4.064   pinentry-doc readline-doc
#7 4.112 The following NEW packages will be installed:
#7 4.115   dirmngr gpg gpg-agent gpgconf libksba8 libreadline8t64 libsqlite3-0
#7 4.116   pinentry-curses readline-common
#7 4.343 0 upgraded, 9 newly installed, 0 to remove and 17 not upgraded.
#7 4.343 Need to get 2286 kB of archives.
#7 4.343 After this operation, 5596 kB of additional disk space will be used.
#7 4.343 Get:1 http://archive.ubuntu.com/ubuntu noble/main amd64 readline-common all 8.2-4build1 [56.5 kB]
#7 4.632 Get:2 http://archive.ubuntu.com/ubuntu noble/main amd64 libreadline8t64 amd64 8.2-4build1 [153 kB]
#7 4.813 Get:3 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libsqlite3-0 amd64 3.45.1-1ubuntu2.1 [701 kB]
#7 5.065 Get:4 http://archive.ubuntu.com/ubuntu noble/main amd64 gpgconf amd64 2.4.4-2ubuntu17 [103 kB]
#7 5.080 Get:5 http://archive.ubuntu.com/ubuntu noble/main amd64 libksba8 amd64 1.6.6-1build1 [122 kB]
#7 5.095 Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 dirmngr amd64 2.4.4-2ubuntu17 [323 kB]
#7 5.129 Get:7 http://archive.ubuntu.com/ubuntu noble/main amd64 gpg amd64 2.4.4-2ubuntu17 [565 kB]
#7 5.176 Get:8 http://archive.ubuntu.com/ubuntu noble/main amd64 pinentry-curses amd64 1.2.1-3ubuntu5 [35.2 kB]
#7 5.178 Get:9 http://archive.ubuntu.com/ubuntu noble/main amd64 gpg-agent amd64 2.4.4-2ubuntu17 [227 kB]
#7 5.428 debconf: delaying package configuration, since apt-utils is not installed
#7 5.469 Fetched 2286 kB in 1s (2190 kB/s)
#7 5.545 Selecting previously unselected package readline-common.
(Reading database ... 8615 files and directories currently installed.)
#7 5.580 Preparing to unpack .../0-readline-common_8.2-4build1_all.deb ...
#7 5.583 Unpacking readline-common (8.2-4build1) ...
#7 5.622 Selecting previously unselected package libreadline8t64:amd64.
#7 5.624 Preparing to unpack .../1-libreadline8t64_8.2-4build1_amd64.deb ...
#7 5.637 Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8 to /lib/x86_64-linux-gnu/libhistory.so.8.usr-is-merged by libreadline8t64'
#7 5.646 Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8.2 to /lib/x86_64-linux-gnu/libhistory.so.8.2.usr-is-merged by libreadline8t64'
#7 5.652 Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8 to /lib/x86_64-linux-gnu/libreadline.so.8.usr-is-merged by libreadline8t64'
#7 5.658 Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8.2 to /lib/x86_64-linux-gnu/libreadline.so.8.2.usr-is-merged by libreadline8t64'
#7 5.661 Unpacking libreadline8t64:amd64 (8.2-4build1) ...
#7 5.709 Selecting previously unselected package libsqlite3-0:amd64.
#7 5.711 Preparing to unpack .../2-libsqlite3-0_3.45.1-1ubuntu2.1_amd64.deb ...
#7 5.714 Unpacking libsqlite3-0:amd64 (3.45.1-1ubuntu2.1) ...
#7 5.782 Selecting previously unselected package gpgconf.
#7 5.784 Preparing to unpack .../3-gpgconf_2.4.4-2ubuntu17_amd64.deb ...
#7 5.787 Unpacking gpgconf (2.4.4-2ubuntu17) ...
#7 5.826 Selecting previously unselected package libksba8:amd64.
#7 5.828 Preparing to unpack .../4-libksba8_1.6.6-1build1_amd64.deb ...
#7 5.831 Unpacking libksba8:amd64 (1.6.6-1build1) ...
#7 5.873 Selecting previously unselected package dirmngr.
#7 5.874 Preparing to unpack .../5-dirmngr_2.4.4-2ubuntu17_amd64.deb ...
#7 5.895 Unpacking dirmngr (2.4.4-2ubuntu17) ...
#7 5.941 Selecting previously unselected package gpg.
#7 5.942 Preparing to unpack .../6-gpg_2.4.4-2ubuntu17_amd64.deb ...
#7 5.945 Unpacking gpg (2.4.4-2ubuntu17) ...
#7 6.005 Selecting previously unselected package pinentry-curses.
#7 6.006 Preparing to unpack .../7-pinentry-curses_1.2.1-3ubuntu5_amd64.deb ...
#7 6.009 Unpacking pinentry-curses (1.2.1-3ubuntu5) ...
#7 6.042 Selecting previously unselected package gpg-agent.
#7 6.044 Preparing to unpack .../8-gpg-agent_2.4.4-2ubuntu17_amd64.deb ...
#7 6.047 Unpacking gpg-agent (2.4.4-2ubuntu17) ...
#7 6.102 Setting up libksba8:amd64 (1.6.6-1build1) ...
#7 6.111 Setting up pinentry-curses (1.2.1-3ubuntu5) ...
#7 6.128 Setting up libsqlite3-0:amd64 (3.45.1-1ubuntu2.1) ...
#7 6.137 Setting up readline-common (8.2-4build1) ...
#7 6.231 Setting up libreadline8t64:amd64 (8.2-4build1) ...
#7 6.240 Setting up gpgconf (2.4.4-2ubuntu17) ...
#7 6.250 Setting up gpg (2.4.4-2ubuntu17) ...
#7 6.260 Setting up gpg-agent (2.4.4-2ubuntu17) ...
#7 7.103 Setting up dirmngr (2.4.4-2ubuntu17) ...
#7 7.336 Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
#7 7.384 Downloading NVIDIA GPG key
#7 9.906 > apt-get update
#7 9.991 Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease
#7 10.02 Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  InRelease [1581 B]
#7 10.15 Hit:3 http://archive.ubuntu.com/ubuntu noble InRelease
#7 10.16 Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  Packages [462 kB]
#7 10.24 Hit:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease
#7 10.33 Hit:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease
#7 10.45 Fetched 464 kB in 1s (906 kB/s)
#7 10.45 Reading package lists...
#7 11.83 > apt-get install -y --no-install-recommends nvidia-container-toolkit-base
#7 11.83 Reading package lists...
#7 13.21 Building dependency tree...
#7 13.44 Reading state information...
#7 13.79 The following NEW packages will be installed:
#7 13.79   nvidia-container-toolkit-base
#7 16.67 0 upgraded, 1 newly installed, 0 to remove and 17 not upgraded.
#7 16.67 Need to get 3710 kB of archives.
#7 16.67 After this operation, 20.4 MB of additional disk space will be used.
#7 16.67 Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  nvidia-container-toolkit-base 1.17.5-1 [3710 kB]
#7 17.52 debconf: delaying package configuration, since apt-utils is not installed
#7 17.56 Fetched 3710 kB in 4s (1056 kB/s)
#7 17.69 Selecting previously unselected package nvidia-container-toolkit-base.
(Reading database ... 8734 files and directories currently installed.)
#7 17.70 Preparing to unpack .../nvidia-container-toolkit-base_1.17.5-1_amd64.deb ...
#7 17.70 Unpacking nvidia-container-toolkit-base (1.17.5-1) ...
#7 18.37 Setting up nvidia-container-toolkit-base (1.17.5-1) ...
#7 18.43 time="2025-04-01T08:09:30Z" level=info msg="Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.40.07"
#7 18.79 time="2025-04-01T08:09:30Z" level=info msg="Auto-detected mode as 'nvml'"
#7 19.14 time="2025-04-01T08:09:31Z" level=warning msg="Failed to evaluate symlink /dev/dri/by-path/pci-0000:84:00.0-card; ignoring"
#7 19.14 time="2025-04-01T08:09:31Z" level=warning msg="Failed to evaluate symlink /dev/dri/by-path/pci-0000:84:00.0-render; ignoring"
#7 19.14 dev/dri/renderD128"
#7 19.14 time="2025-04-01T08:09:31Z" level=warning msg="Failed to evaluate symlink /dev/dri/by-path/pci-0000:84:00.0-card; ignoring"
#7 19.14 time="2025-04-01T08:09:31Z" level=warning msg="Failed to evaluate symlink /dev/dri/by-path/pci-0000:84:00.0-render; ignoring"
#7 19.50 time="2025-04-01T08:09:31Z" level=info msg="Using driver version 550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /dev/nvidia-uvm as /dev/nvidia-uvm"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /dev/nvidiactl as /dev/nvidiactl"
#7 19.54 dev/nvidia-uvm-tools"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /dev/nvidia-uvm as /dev/nvidia-uvm"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /dev/nvidiactl as /dev/nvidiactl"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate libnvidia-egl-gbm.so.*.*: pattern libnvidia-egl-gbm.so.*.* not found\nlibnvidia-egl-gbm.so.*.*: not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate libnvidia-egl-wayland.so.*.*: pattern libnvidia-egl-wayland.so.*.* not found\nlibnvidia-egl-wayland.so.*.*: not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate libnvidia-vulkan-producer.so.550.40.07: pattern libnvidia-vulkan-producer.so.550.40.07 not found\nlibnvidia-vulkan-producer.so.550.40.07: not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate nvidia_drv.so: pattern nvidia_drv.so not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate libglxserver_nvidia.so.550.40.07: pattern libglxserver_nvidia.so.550.40.07 not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate glvnd/egl_vendor.d/10_nvidia.json: pattern glvnd/egl_vendor.d/10_nvidia.json not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate egl/egl_external_platform.d/10_nvidia_wayland.json: pattern egl/egl_external_platform.d/10_nvidia_wayland.json nottime="2025-04-01T08:09:31Z" level=warning msg="Could not locate egl/egl_external_platform.d/10_nvidia_wayland.json: pattern egl/egl_external_platform.d/10_nvidia_wayland.json not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate nvidia/nvoptix.bin: pattern nvidia/nvoptix.bin not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: pattern X11/xorg.conf.d/nvidia-drm-outime="2025-04-01T08:09:31Z" level=warning msg="Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: pattern X11/xorg.conf.d/nvidia-drm-outputclass.conf not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found\npattern vulkan/icd.d/nvidia_layers.json notime="2025-04-01T08:09:31Z" level=warning msg="Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found\npattern vulkan/icd.d/nvidia_layers.json not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07"
#7 19.54 \npattern vulkan/implicit_layer.d/nvidia_layers.json not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07"
#7 19.54 0.07"
#7 19.54 07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07"
#7 19.54 0.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.40.07 as /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.40.07"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate /nvidia-persistenced/socket: pattern /nvidia-persistenced/socket not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate /nvidia-fabricmanager/socket: pattern /nvidia-fabricmanager/socket not found"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /lib/firmware/nvidia/550.40.07/gsp_ga10x.bin as /lib/firmware/time="2025-04-01T08:09:31Z" level=info msg="Selecting /lib/firmware/nvidia/550.40.07/gsp_ga10x.bin as /lib/firmware/nvidia/550.40.07/gsp_ga10x.bin"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /lib/firmware/nvidia/550.40.07/gsp_tu10x.bin as /lib/firmware/nvidia/550.40.07/gsp_tu10x.bin"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/bin/nvidia-debugdump as /usr/bin/nvidiatime="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistetime="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-ctime="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control"
#7 19.54 time="2025-04-01T08:09:31Z" level=info msg="Selecting /usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server"
#7 19.54 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate nvidia-imex-ctl: pattern nvidia-imex-ctl nottime="2025-04-01T08:09:31Z" level=warning msg="Could not locate nvidia-imex-ctl: pattern nvidia-imex-ctl not found"
#7 19.55 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate nvidia_drv.so: pattern nvidia_drv.so not found"
#7 19.55 time="2025-04-01T08:09:31Z" level=warning msg="Could not locate libglxserver_nvidia.so.550.40.07: pattern libglxserver_nvidia.so.550.40.07 not found"
#7 19.55 time="2025-04-01T08:09:31Z" level=info msg="Generated CDI spec with version 0.8.0"
#7 DONE 19.6s

#8 [2/2] RUN --device=nvidia.com/gpu nvidia-smi
#8 0.521 Tue Apr  1 08:09:32 2025       
#8 0.521 +-----------------------------------------------------------------------------------------+
#8 0.521 | NVIDIA-SMI 550.40.07              Driver Version: 550.40.07      CUDA Version: 12.4     |
#8 0.521 |-----------------------------------------+------------------------+----------------------+
#8 0.521 | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
#8 0.521 | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
#8 0.521 |                                         |                        |               MIG M. |
#8 0.521 |=========================================+========================+======================|
#8 0.522 |   0  NVIDIA GeForce GTX 1050 Ti     Off |   00000000:84:00.0 Off |                  N/A |
#8 0.522 | 50%   44C    P0             N/A /   75W |       0MiB /   4096MiB |      0%      Default |
#8 0.522 |                                         |                        |                  N/A |
#8 0.522 +-----------------------------------------+------------------------+----------------------+
#8 0.522                                                                                          
#8 0.522 +-----------------------------------------------------------------------------------------+
#8 0.522 | Processes:                                                                              |
#8 0.522 |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
#8 0.522 |        ID   ID                                                               Usage      |
#8 0.522 |=========================================================================================|
#8 0.522 |  No running processes found                                                             |
#8 0.522 +-----------------------------------------------------------------------------------------+
#8 DONE 2.2s

Generated CDI specs within the container builder:

$ docker exec -it buildx_buildkit_nvidia0 cat /etc/cdi/nvidia.yaml

---
cdiVersion: 0.5.0
containerEdits:
  deviceNodes:
  - path: /dev/nvidia-modeset
  - path: /dev/nvidia-uvm
  - path: /dev/nvidia-uvm-tools
  - path: /dev/nvidiactl
  env:
  - NVIDIA_VISIBLE_DEVICES=void
  hooks:
  - args:
    - nvidia-cdi-hook
    - create-symlinks
    - --link
    - ../libnvidia-allocator.so.1::/usr/lib/x86_64-linux-gnu/gbm/nvidia-drm_gbm.so
    hookName: createContainer
    path: /usr/bin/nvidia-cdi-hook
  - args:
    - nvidia-cdi-hook
    - create-symlinks
    - --link
    - libcuda.so.1::/usr/lib/x86_64-linux-gnu/libcuda.so
    hookName: createContainer
    path: /usr/bin/nvidia-cdi-hook
  - args:
    - nvidia-cdi-hook
    - enable-cuda-compat
    - --host-driver-version=550.40.07
    hookName: createContainer
    path: /usr/bin/nvidia-cdi-hook
  - args:
    - nvidia-cdi-hook
    - update-ldcache
    - --folder
    - /usr/lib/x86_64-linux-gnu
    hookName: createContainer
    path: /usr/bin/nvidia-cdi-hook
  mounts:
  - containerPath: /usr/bin/nvidia-cuda-mps-control
    hostPath: /usr/bin/nvidia-cuda-mps-control
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/bin/nvidia-cuda-mps-server
    hostPath: /usr/bin/nvidia-cuda-mps-server
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/bin/nvidia-debugdump
    hostPath: /usr/bin/nvidia-debugdump
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/bin/nvidia-persistenced
    hostPath: /usr/bin/nvidia-persistenced
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/bin/nvidia-smi
    hostPath: /usr/bin/nvidia-smi
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.40.07
    hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.550.40.07
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /lib/firmware/nvidia/550.40.07/gsp_ga10x.bin
    hostPath: /lib/firmware/nvidia/550.40.07/gsp_ga10x.bin
    options:
    - ro
    - nosuid
    - nodev
    - bind
  - containerPath: /lib/firmware/nvidia/550.40.07/gsp_tu10x.bin
    hostPath: /lib/firmware/nvidia/550.40.07/gsp_tu10x.bin
    options:
    - ro
    - nosuid
    - nodev
    - bind
devices:
- containerEdits:
    deviceNodes:
    - path: /dev/nvidia0
    - path: /dev/dri/card1
    - path: /dev/dri/renderD128
    hooks:
    - args:
      - nvidia-cdi-hook
      - chmod
      - --mode
      - "755"
      - --path
      - /dev/dri
      hookName: createContainer
      path: /usr/bin/nvidia-cdi-hook
  name: "0"
- containerEdits:
    deviceNodes:
    - path: /dev/nvidia0
    - path: /dev/dri/card1
    - path: /dev/dri/renderD128
    hooks:
    - args:
      - nvidia-cdi-hook
      - chmod
      - --mode
      - "755"
      - --path
      - /dev/dri
      hookName: createContainer
      path: /usr/bin/nvidia-cdi-hook
  name: GPU-c5f53f8f-c215-47d2-fa69-d7b4fb13701d
- containerEdits:
    deviceNodes:
    - path: /dev/nvidia0
    - path: /dev/dri/card1
    - path: /dev/dri/renderD128
    hooks:
    - args:
      - nvidia-cdi-hook
      - chmod
      - --mode
      - "755"
      - --path
      - /dev/dri
      hookName: createContainer
      path: /usr/bin/nvidia-cdi-hook
  name: all
kind: nvidia.com/gpu

@crazy-max crazy-max requested a review from tonistiigi March 26, 2025 10:15
tonistiigi
tonistiigi reviewed Mar 26, 2025
View reviewed changes
@crazy-max crazy-max force-pushed the contrib-nvidia-nopci branch from 36ed40a to da31bd4 Compare March 27, 2025 14:30
@crazy-max crazy-max changed the title contrib: support non PCI-based nvidia GPUs for WSL contrib: check nvidia drivers installed and support non PCI-based nvidia GPUs for WSL Mar 27, 2025
@crazy-max crazy-max changed the title contrib: check nvidia drivers installed and support non PCI-based nvidia GPUs for WSL contrib: check nvidia drivers installed and support non PCI-based GPUs for WSL Mar 27, 2025
@crazy-max crazy-max changed the title contrib: check nvidia drivers installed and support non PCI-based GPUs for WSL contrib: check nvidia drivers and support non PCI-based GPUs for WSL Mar 27, 2025
@crazy-max crazy-max self-assigned this Mar 27, 2025
@crazy-max crazy-max modified the milestone: v0.21.0 Mar 27, 2025
@crazy-max crazy-max force-pushed the contrib-nvidia-nopci branch from da31bd4 to e2ef5e5 Compare March 28, 2025 11:22
@crazy-max crazy-max requested a review from tonistiigi March 28, 2025 11:22
tonistiigi
tonistiigi reviewed Mar 28, 2025
View reviewed changes
@crazy-max crazy-max force-pushed the contrib-nvidia-nopci branch from e2ef5e5 to f85a66c Compare April 1, 2025 08:38
@crazy-max crazy-max requested a review from tonistiigi April 1, 2025 09:08
@crazy-max crazy-max added this to the v0.21.0 milestone Apr 7, 2025
@tonistiigi tonistiigi merged commit f8c1909 into moby:master Apr 8, 2025
108 checks passed
@crazy-max crazy-max deleted the contrib-nvidia-nopci branch April 8, 2025 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tonistiigi tonistiigi tonistiigi approved these changes

Assignees

@crazy-max crazy-max

Labels

area/cdi

Projects

None yet

Milestone

v0.21.0

Development

Successfully merging this pull request may close these issues.

2 participants

@crazy-max @tonistiigi