Alpaca does not detect 4060 GPU

[karypid]

I use Alpaca from the flatpak on flahub, but it seems to fail to launch a GPU-accelerated container on my system.

It has to be something with the flatpak, as when I run the below docker hub image and connect Alpaca to it, everything works fine (Nvidia 4060 is used):

❯ cat .config/containers/systemd/ollama.container
[Unit]
Description=The ollama container

[Container]
ContainerName=ollama
Image=ollama/ollama
AddDevice=nvidia.com/gpu=all
SecurityLabelDisable=true
PublishPort=11434:11434
Volume=ollama:/root/.ollama:z

# Remove if you don't want autostart
[Install]
WantedBy=default.target

When I run from the command line with debug enabled I get:

❯ OLLAMA_DEBUG=true flatpak run com.jeffser.Alpaca
INFO	[main.py | main] Alpaca version: 5.0.5
MESA-INTEL: warning: ../src/intel/vulkan/anv_formats.c:782: FINISHME: support YUV colorspace with DRM format modifiers
MESA-INTEL: warning: ../src/intel/vulkan/anv_formats.c:814: FINISHME: support more multi-planar formats with DRM modifiers
INFO	[instance_manager.py | start] Starting Alpaca's Ollama instance...
INFO	[instance_manager.py | start] Started Alpaca's Ollama instance
2025/03/01 18:09:23 routes.go:1205: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11435 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/myuser/.var/app/com.jeffser.Alpaca/data/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-03-01T18:09:23.528Z level=INFO source=images.go:432 msg="total blobs: 5"
time=2025-03-01T18:09:23.528Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
time=2025-03-01T18:09:23.528Z level=INFO source=routes.go:1256 msg="Listening on [::]:11435 (version 0.5.12)"
time=2025-03-01T18:09:23.528Z level=DEBUG source=sched.go:106 msg="starting llm scheduler"

>>> Here is the part where GPU detection starts

time=2025-03-01T18:09:23.528Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
INFO	[instance_manager.py | start] client version is 0.5.12
time=2025-03-01T18:09:23.531Z level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-03-01T18:09:23.531Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-03-01T18:09:23.532Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/lib/ollama/libcuda.so* /app/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/sdk/llvm15/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/ollama/libcuda.so* /app/plugins/AMD/lib/ollama/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"

>>> When I run ls with the path above, I get a libcuda present which seems to not be located?

time=2025-03-01T18:09:23.536Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths=[]

>>> Below it tried to locate cudart which is found:

time=2025-03-01T18:09:23.536Z level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so*
time=2025-03-01T18:09:23.536Z level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/app/lib/ollama/libcudart.so* /app/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcudart.so* /usr/lib/sdk/llvm15/lib/libcudart.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcudart.so* /usr/lib/ollama/libcudart.so* /app/plugins/AMD/lib/ollama/libcudart.so* /app/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-03-01T18:09:23.538Z level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/app/lib/ollama/cuda_v11/libcudart.so.11.3.109 /app/lib/ollama/cuda_v12/libcudart.so.12.4.127]"

>>> It seems toi me that libcudart is embedded in the flatpak from the path, but below you see initialization fail with error 100

cudaSetDevice err: 100
time=2025-03-01T18:09:23.549Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/lib/ollama/cuda_v11/libcudart.so.11.3.109: cudart init failure: 100"
cudaSetDevice err: 100
time=2025-03-01T18:09:23.556Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/lib/ollama/cuda_v12/libcudart.so.12.4.127: cudart init failure: 100"
time=2025-03-01T18:09:23.557Z level=WARN source=amd_linux.go:61 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2025-03-01T18:09:23.557Z level=DEBUG source=amd_linux.go:101 msg="evaluating amdgpu node /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-03-01T18:09:23.557Z level=DEBUG source=amd_linux.go:121 msg="detected CPU /sys/class/kfd/kfd/topology/nodes/0/properties"
time=2025-03-01T18:09:23.557Z level=INFO source=amd_linux.go:402 msg="no compatible amdgpu devices detected"
time=2025-03-01T18:09:23.557Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-03-01T18:09:23.557Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="30.9 GiB" available="16.0 GiB"

Now regarding the first detection attempt for libcuda, if I run ls -l with the glob pattern listed I get:

❯ ls -l /app/lib/ollama/libcuda.so* /app/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so* /usr/lib/sdk/llvm15/lib/libcuda.so* /usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so* /usr/lib/ollama/libcuda.so* /app/plugins/AMD/lib/ollama/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*
ls: cannot access '/app/lib/ollama/libcuda.so*': No such file or directory
ls: cannot access '/app/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/x86_64-linux-gnu/openh264/extra/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/sdk/llvm15/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/x86_64-linux-gnu/GL/default/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/ollama/libcuda.so*': No such file or directory
ls: cannot access '/app/plugins/AMD/lib/ollama/libcuda.so*': No such file or directory
ls: cannot access '/usr/local/cuda*/targets/*/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/*-linux-gnu/nvidia/current/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/*-linux-gnu/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/wsl/lib/libcuda.so*': No such file or directory
ls: cannot access '/usr/lib/wsl/drivers/*/libcuda.so*': No such file or directory
ls: cannot access '/opt/cuda/lib*/libcuda.so*': No such file or directory
ls: cannot access '/usr/local/cuda/lib*/libcuda.so*': No such file or directory
ls: cannot access '/usr/local/lib*/libcuda.so*': No such file or directory
lrwxrwxrwx. 17 root root       20 Feb  9 10:25  /usr/lib64/libcuda.so -> libcuda.so.570.86.16
lrwxrwxrwx. 17 root root       20 Feb  9 10:25  /usr/lib64/libcuda.so.1 -> libcuda.so.570.86.16
-rwxr-xr-x.  5 root root 71189464 Jan  1  1970  /usr/lib64/libcuda.so.570.86.16
lrwxrwxrwx. 17 root root       20 Feb  9 10:25  /usr/lib/libcuda.so -> libcuda.so.570.86.16
lrwxrwxrwx. 17 root root       20 Feb  9 10:25  /usr/lib/libcuda.so.1 -> libcuda.so.570.86.16
-rwxr-xr-x.  5 root root 25609636 Jan  1  1970  /usr/lib/libcuda.so.570.86.16

So I am not sure why the file is not located. Here is the complete list of permission in flatseal, I allowed "All system files" in an attempt to get around this but it did not make a difference:

Furthermore I don't know why libcudart fails with this error:

time=2025-03-01T18:09:23.549Z level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /app/lib/ollama/cuda_v11/libcudart.so.11.3.109: cudart init failure: 100"

What does this error code mean? I should note that nvidia-smi does mention the alpaca-launched ollama process:

❯ nvidia-smi 
Sat Mar  1 18:23:53 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.16              Driver Version: 570.86.16      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   50C    P8              2W /   55W |      45MiB /   8188MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3624      G   /usr/bin/gnome-shell                      2MiB |
|    0   N/A  N/A           30947      G   missioncenter                             2MiB |
|    0   N/A  N/A          332011      G   /usr/bin/python3                          2MiB |
|    0   N/A  N/A          342570      G   /usr/bin/gjs                              2MiB |
+-----------------------------------------------------------------------------------------+

~ 
❯ ps -ef --forest | egrep "332011|342570"
myuser+  342570  342569  5 18:14 ?        00:00:28  |           \_ /usr/bin/gjs /app/bin/com.github.tchx84.Flatseal
myuser+  332011  332010  0 18:09 pts/1    00:00:06  |       |           \_ /usr/bin/python3 /app/bin/alpaca
myuser+  332085  332011  0 18:09 pts/1    00:00:00  |       |               \_ ollama serve
myuser+  359389  329155  0 18:23 pts/0    00:00:00  |           \_ grep --color=auto -E --color=auto 332011|342570

[Jeffser]

I'm moving all conversation about AMD to this discussion, please take a look!