Skip to content

'torch' wheels not compatible with 'cuda-toolkit' (CUDA 12 only) #255

New issue
New issue
Closed
Closed
'torch' wheels not compatible with 'cuda-toolkit' (CUDA 12 only)#255
Assignees
jameslamb
@jameslamb

Description

@jameslamb
jameslamb
opened on Feb 25, 2026

Description

PyTorch publishes torch CUDA wheels to their own indices.

ref: https://pytorch.org/get-started/locally/#linux-pip

Those packages are very very tightly pinned to a bunch of nvidia-* wheels, and for some CUDA 12 packages those pins are not compatible with ANY cuda-toolkit packages on pypi.org.

RAPIDS depends heavily on cuda-toolkit, so that makes newer RAPIDS wheels incompatible with some of those torch CUDA 12 wheels 😬

Reproducible Example

On x86_64:

docker run \
    --rm \
    -it rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    bash

# download PyTorch
mkdir -p ./wheels-dir
pip download \
    --index-url https://download.pytorch.org/whl/cu126 \
    --isolated \
    --no-deps \
    --prefer-binary \
    -d ./wheels-dir \
    'torch==2.10.0'

pip install \
    --index-url https://pypi.org/simple \
    --isolated \
    --prefer-binary \
    'cuda-toolkit[all]==12.*' \
    ./wheels-dir/torch-*.whl

Yields a solver error like this:

ERROR: Cannot install cuda-toolkit[all]==12.0.0, cuda-toolkit[all]==12.0.1, cuda-toolkit[all]==12.1.0, cuda-toolkit[all]==12.1.1, cuda-toolkit[all]==12.2.0, cuda-toolkit[all]==12.2.1, cuda-toolkit[all]==12.2.2, cuda-toolkit[all]==12.3.0, cuda-toolkit[all]==12.3.1, cuda-toolkit[all]==12.3.2, cuda-toolkit[all]==12.4.0, cuda-toolkit[all]==12.4.1, cuda-toolkit[all]==12.5.0, cuda-toolkit[all]==12.5.1, cuda-toolkit[all]==12.6.0, cuda-toolkit[all]==12.6.1, cuda-toolkit[all]==12.6.2, cuda-toolkit[all]==12.6.3, cuda-toolkit[all]==12.8.0, cuda-toolkit[all]==12.8.1, cuda-toolkit[all]==12.9.0, cuda-toolkit[all]==12.9.1, torch and torch==2.10.0+cu126 because these package versions have conflicting dependencies.

The conflict is caused by:
    torch 2.10.0+cu126 depends on nvidia-cublas-cu12==12.6.4.1; platform_system == "Linux"
    nvidia-cudnn-cu12 9.10.2.21 depends on nvidia-cublas-cu12
    nvidia-cusolver-cu12 11.7.1.2 depends on nvidia-cublas-cu12
    cuda-toolkit[all] 12.9.1 depends on nvidia-cublas-cu12==12.9.1.4.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.9.0 depends on nvidia-cublas-cu12==12.9.0.13.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.8.1 depends on nvidia-cublas-cu12==12.8.4.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.8.0 depends on nvidia-cublas-cu12==12.8.3.14.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    torch 2.10.0+cu126 depends on nvidia-cuda-nvrtc-cu12==12.6.77; platform_system == "Linux"
    cuda-toolkit[all] 12.6.3 depends on nvidia-cuda-nvrtc-cu12==12.6.85.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.6.2 depends on nvidia-cublas-cu12==12.6.3.3.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.6.1 depends on nvidia-cublas-cu12==12.6.1.4.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.6.0 depends on nvidia-cublas-cu12==12.6.0.22.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.5.1 depends on nvidia-cublas-cu12==12.5.2.13.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.5.0 depends on nvidia-cublas-cu12==12.5.2.13.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.4.1 depends on nvidia-cublas-cu12==12.4.5.8.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.4.0 depends on nvidia-cublas-cu12==12.4.2.65.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.3.2 depends on nvidia-cublas-cu12==12.3.4.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.3.1 depends on nvidia-cublas-cu12==12.3.4.1.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.3.0 depends on nvidia-cublas-cu12==12.3.2.9.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.2 depends on nvidia-cublas-cu12==12.2.5.6.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.1 depends on nvidia-cublas-cu12==12.2.4.5.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.0 depends on nvidia-cublas-cu12==12.2.1.16.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.1.1 depends on nvidia-cublas-cu12==12.1.3.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.1.0 depends on nvidia-cublas-cu12==12.1.0.26.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.0.1 depends on nvidia-cublas-cu12==12.0.2.224.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.0.0 depends on nvidia-cublas-cu12==12.0.1.189.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"

Additionally, some packages in these conflicts have no matching distributions available for your environment:
    nvidia-cublas-cu12
    nvidia-cuda-nvrtc-cu12

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

cuda-toolkit and torch are both ==-pinning to a bunch of packages, and there's no mix where the pins overlap.

test-install.sh (click me) 
#!/bin/bash

test-install() {
    torch_req=$1
    torch_index_tag=$2
    ctk_req=$3

    PYTORCH_INDEX="https://download.pytorch.org/whl/${torch_index_tag}"

    # download PyTorch
    mkdir -p /tmp/wheels-dir
    pip download \
        --index-url "${PYTORCH_INDEX}" \
        --isolated \
        --no-deps \
        --prefer-binary \
        -d /tmp/wheels-dir \
        "${torch_req}+${torch_index_tag}"

    pip install \
        --dry-run \
        --index-url https://pypi.org/simple \
        --isolated \
        --prefer-binary \
        "${ctk_req}" \
        /tmp/wheels-dir/torch-*.whl
}

test-install $@
✅ CUDA 13, torch 2.10.0 - works (cuda-toolkit 13.0.2) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda13.1.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.10.0' 'cu130' 'cuda-toolkit[all]==13.*'

# Would install ... cuda-bindings-13.0.3 cuda-pathfinder-1.3.5 cuda-toolkit-13.0.2 ...
✅ CUDA 13, torch 2.9.1 - works (cuda-toolkit 13.0.0) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda13.1.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.9.1' 'cu130' 'cuda-toolkit[all]==13.*'

# Would install ... cuda-toolkit-13.0.0 ...
✅ CUDA 13, torch 2.9.0 - works (cuda-toolkit 13.0.0) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda13.1.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.9.0' 'cu130' 'cuda-toolkit[all]==13.*'

# Would install ... cuda-toolkit-13.0.0 ...
✅ CUDA 12.9, torch 2.10.0 - works (cuda-toolkit 12.9.1) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.10.0' 'cu129' 'cuda-toolkit[all]==12.*'

# Would install ... cuda-bindings-12.9.4 cuda-pathfinder-1.3.5 cuda-toolkit-12.9.1 ...
✅ CUDA 12.9, torch 2.9.1 - works (cuda-toolkit 12.9.1) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.9.0' 'cu129' 'cuda-toolkit[all]==12.*'

# Could not find a version that satisfies the requirement torch==2.7.0+cu129 (from versions: 2.8.0+cu129, 2.9.0+cu129, 2.9.1+cu129, 2.10.0+cu129)
❌ CUDA 12.9, torch 2.7.0 - fails (no torch packages) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.7.0' 'cu129' 'cuda-toolkit[all]==12.*'

# ERROR: Could not find a version that satisfies the requirement torch==2.7.0+cu129
# (from versions: 2.8.0+cu129, 2.9.0+cu129, 2.9.1+cu129, 2.10.0+cu129)
✅ CUDA 12.8, torch 2.7.0 - works (cuda-toolkit 12.8.0) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.7.0' 'cu128' 'cuda-toolkit[all]==12.*'

# Would install .. cuda-toolkit-12.8.0 ...
❌ CUDA 12.8, torch 2.6.0 - fails (no torch packages) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.6.0' 'cu128' 'cuda-toolkit[all]==12.*'

# ERROR: Could not find a version that satisfies the requirement torch==2.6.0+cu128
# (from versions: 2.7.0+cu128, 2.7.1+cu128, 2.8.0+cu128, 2.9.0+cu128, 2.9.1+cu128, 2.10.0+cu128)
❌ CUDA 12.6, torch 2.6.0 - fails (torch & cuda-toolkit incompatible) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.6.0' 'cu126' 'cuda-toolkit[all]==12.*'

# ERROR: Could not find a version that satisfies the requirement torch==2.6.0+cu128
# (from versions: 2.7.0+cu128, 2.7.1+cu128, 2.8.0+cu128, 2.9.0+cu128, 2.9.1+cu128, 2.10.0+cu128)

Fails like this:

The conflict is caused by:
    The user requested cuda-toolkit==12.*
    cuda-toolkit[all] 12.9.1 depends on cuda-toolkit 12.9.1 (from https://files.pythonhosted.org/packages/7c/8f/a28e7da158e96ad61f7e1035e53851fafaddf22445300d664e68ec657fdc/cuda_toolkit-12.9.1-py2.py3-none-any.whl (from https://pypi.org/simple/cuda-toolkit/))
    torch 2.6.0+cu126 depends on nvidia-cublas-cu12==12.6.4.1; platform_system == "Linux" and platform_machine == "x86_64"
    nvidia-cudnn-cu12 9.5.1.17 depends on nvidia-cublas-cu12
    nvidia-cusolver-cu12 11.7.1.2 depends on nvidia-cublas-cu12
    cuda-toolkit[all] 12.9.0 depends on nvidia-cublas-cu12==12.9.0.13.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.8.1 depends on nvidia-cublas-cu12==12.8.4.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.8.0 depends on nvidia-cublas-cu12==12.8.3.14.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    torch 2.6.0+cu126 depends on nvidia-cuda-nvrtc-cu12==12.6.77; platform_system == "Linux" and platform_machine == "x86_64"
    cuda-toolkit[all] 12.6.3 depends on nvidia-cuda-nvrtc-cu12==12.6.85.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.6.2 depends on nvidia-cublas-cu12==12.6.3.3.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.6.1 depends on nvidia-cublas-cu12==12.6.1.4.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.6.0 depends on nvidia-cublas-cu12==12.6.0.22.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.5.1 depends on nvidia-cublas-cu12==12.5.2.13.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.5.0 depends on nvidia-cublas-cu12==12.5.2.13.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.4.1 depends on nvidia-cublas-cu12==12.4.5.8.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.4.0 depends on nvidia-cublas-cu12==12.4.2.65.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.3.2 depends on nvidia-cublas-cu12==12.3.4.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.3.1 depends on nvidia-cublas-cu12==12.3.4.1.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.3.0 depends on nvidia-cublas-cu12==12.3.2.9.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.2 depends on nvidia-cublas-cu12==12.2.5.6.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.1 depends on nvidia-cublas-cu12==12.2.4.5.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.2.0 depends on nvidia-cublas-cu12==12.2.1.16.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.1.1 depends on nvidia-cublas-cu12==12.1.3.1.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.1.0 depends on nvidia-cublas-cu12==12.1.0.26.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"
    cuda-toolkit[all] 12.0.1 depends on nvidia-cublas-cu12==12.0.2.224.*; (sys_platform == "linux" or sys_platform == "win32") and extra == "all"
    cuda-toolkit[all] 12.0.0 depends on nvidia-cublas-cu12==12.0.1.189.*; (sys_platform == "win32" or sys_platform == "linux") and extra == "all"

Additionally, some packages in these conflicts have no matching distributions available for your environment:
    cuda-toolkit
    nvidia-cublas-cu12
    nvidia-cuda-nvrtc-cu1
❌ CUDA 12.6, torch 2.5.0 - fails (no torch packages) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.5.0' 'cu126' 'cuda-toolkit[all]==12.*'

# ERROR: ERROR: Could not find a version that satisfies the requirement torch==2.5.0+cu126
# (from versions: 2.6.0+cu126, 2.7.0+cu126, 2.7.1+cu126, 2.8.0+cu126, 2.9.0+cu126, 2.9.1+cu126, 2.10.0+cu126)
✅ CUDA 12.4, torch 2.5.0 - works (cuda-toolkit 12.4.1) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.5.0' 'cu124' 'cuda-toolkit[all]==12.*'

# Would install ... cuda-toolkit-12.4.1 ...
✅ CUDA 12.4, torch 2.4.0 - works (cuda-toolkit 12.4.0) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.4.0' 'cu124' 'cuda-toolkit[all]==12.*'

# Would install ... cuda-toolkit-12.4.0 ...
❌ CUDA 12.4, torch 2.3.0 - fails (no torch packages) 
docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    rapidsai/ci-wheel:26.04-cuda12.9.1-rockylinux8-py3.12 \
    ./test-install.sh 'torch==2.3.0' 'cu124' 'cuda-toolkit[all]==12.*'

# ERROR: Could not find a version that satisfies the requirement torch==2.3.0+cu124
# (from versions: 2.4.0+cu124, 2.4.1+cu124, 2.5.0+cu124, 2.5.1+cu124, 2.6.0+cu124)

Possible Solutions

Once every version of torch that RAPIDS supports uses cuda-toolkit in its wheels, this issue will be fully resolved. That was done in pytorch/pytorch#174390, though users won't see that change until PyTorch 2.11 at the earliest.

Until then, the testing approach above could be used to find compatible versions.

I found that all of these had at least 1 compatible cuda-toolkit release on pypi.org (and RAPIDS pins very broadly to cuda-toolkit==12.*, so shouldn't introduce any additional conflicts):

# --extra-index-url https://download.pytorch.org/whl/cu129
torch==2.10.0+cu129
torch==2.9.1+cu129

# --extra-index-url https://download.pytorch.org/whl/cu128
torch==2.7.0+cu128

# --extra-index-url https://download.pytorch.org/whl/cu124
torch==2.5.0+cu124
torch==2.4.0+cu124

Notes

RAPIDS started depending on cuda-toolkit wheels in 25.10: #213

torch wheels will start using cuda-toolkit in PyTorch 3.11: pytorch/pytorch#174390

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions