Compile bug: CUDA build for mmq breaks for compute capability 120

[mramendi]

Git commit

$ git rev-parse HEAD
c54bba8

Operating systems

Linux

GGML backends

CUDA

Problem description & steps to reproduce

When CUDA compute 120 is enabled, the build breaks on mmq; see commands and logs in the fields below

First Bad Commit

c8a2417

Compile command

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="80;86;90;120" -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs"

make --build build --config Release -j 6

Relevant log output

27%] Building CUDA object ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-q4_0.cu.o
ptxas /tmp/tmpxft_00007d6d_00000000-6_mmq-instance-mxfp4.compute_120.ptx, line 1025; error   : Instruction 'mma with block scale' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_00007d6d_00000000-6_mmq-instance-mxfp4.compute_120.ptx, line 1025; error   : Feature '.kind::mxf4' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_00007d6d_00000000-6_mmq-instance-mxfp4.compute_120.ptx, line 1025; error   : Feature '.block_scale' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_00007d6d_00000000-6_mmq-instance-mxfp4.compute_120.ptx, line 1025; error   : Feature '.scale_vec::2X' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_00007d6d_00000000-6_mmq-instance-mxfp4.compute_120.ptx, line 1039; error   : Instruction 'mma with block scale' not supported on .target 'sm_120'
(many, many other similar lines)

[am17an]

You need to add 120a, or use #18361

[raziel2001au]

I just updated to the latest code, I'm getting very similar compile errors for sm_121:

ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112320; error   : Feature '.block_scale' not supported on .target 'sm_121'                                                                                                       
ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112320; error   : Feature '.scale_vec::2X' not supported on .target 'sm_121'                                                                                                     
ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112334; error   : Instruction 'mma with block scale' not supported on .target 'sm_121'                                                                                           
ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112334; error   : Feature '.kind::mxf4' not supported on .target 'sm_121'                                                                                                        
ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112334; error   : Feature '.block_scale' not supported on .target 'sm_121'                                                                                                       
ptxas /tmp/tmpxft_000064a1_00000000-7_mmq-instance-mxfp4.ptx, line 112334; error   : Feature '.scale_vec::2X' not supported on .target 'sm_121'                                                                                                     
(many, many other similar lines)

I use a bash script on the DGX Spark to update and compile llama.cpp, it's always compiled the same way, so I'm confident it's something that was introduced.

[vektory79]

Sadly, for last commit (33 minutes ago) is not fix build for me:

cd llama.cpp
git pull
rm -r ./build
cmake \
    -B ./build \
    -DGGML_CUDA=ON \
    -DGGML_BLAS=OFF \
    -DGGML_NATIVE=ON \
    -DGGML_RPC=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CPU_ALL_VARIANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DGGML_CUDA_FA_ALL_QUANTS=true \
    -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc --all)

I've same error for this script :(

[am17an]

I think the problem here is that you use GGML_NATIVE and CMAKE_CUDA_ARCHITECTURES, whereas the fix only applies to either. I will fix it, but for now you can either remove CMAKE_CUDA_ARCHITECTURES or use 120a instead of 120

[vektory79]

I have:

• RTX 5090
• Ubuntu 24.04
• cmake version 3.28.3
• NVIDIA Driver Version: 590.48.01
• The CUDA compiler identification is NVIDIA 13.1.80

[vektory79]

I think the problem here is that you use GGML_NATIVE and CMAKE_CUDA_ARCHITECTURES, whereas the fix only applies to either. I will fix it, but for now you can either remove CMAKE_CUDA_ARCHITECTURES or use 120a instead of 120

Ok. Thanks a lot. I will test performance for that options combinations :)

[BahamutRU]

I use cuda 12.8, ubuntu 24.04, rtx 5060 ti with command:

cmake \
    -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLAMA_CURL=ON \
    -DGGML_CUDA=ON \
    -DGGML_VULKAN=OFF \
    -DGGML_RPC=OFF \
    -DGGML_BLAS=ON \
    -DGGML_BLAS_VENDOR=OpenBLAS \
    -DLLAMA_OPENSSL=ON \
    -DGGML_SCHED_MAX_COPIES=4 \
    -DGGML_CUDA_FA_ALL_QUANTS=ON \
    -DGGML_CCACHE=OFF

And get this error.

Then I use:
-DCMAKE_CUDA_ARCHITECTURES= …
120a
120a-real…
122f
122f-real

All of this version dropped an error.

-- Using CUDA architectures: 121f
CMake Error at /usr/share/cmake-3.28/Modules/Internal/CMakeCUDAArchitecturesValidate.cmake:9 (message):
  CMAKE_CUDA_ARCHITECTURES:

    121f

  is not one of the following:

    * a semicolon-separated list of integers, each optionally
      followed by '-real' or '-virtual'
    * a special value: all, all-major, native

Call Stack (most recent call first):
  /usr/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:116 (cmake_cuda_architectures_validate)
  ggml/src/ggml-cuda/CMakeLists.txt:42 (enable_language)

And
-DCMAKE_CUDA_ARCHITECTURES= …
90
120
120-real

And again give classic error.

ptxas /tmp/tmpxft_0000973b_00000000-6_mmq-instance-mxfp4.ptx, line 122018; error   : Instruction 'mma with block scale' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_0000973b_00000000-6_mmq-instance-mxfp4.ptx, line 122018; error   : Feature '.kind::mxf4' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_0000973b_00000000-6_mmq-instance-mxfp4.ptx, line 122018; error   : Feature '.block_scale' not supported on .target 'sm_120'
ptxas /tmp/tmpxft_0000973b_00000000-6_mmq-instance-mxfp4.ptx, line 122018; error   : Feature '.scale_vec::2X' not supported on .target 'sm_120'
ptxas fatal   : Ptx assembly aborted due to errors
gmake[2]: *** [ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/build.make:1502: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/template-instances/mmq-instance-mxfp4.cu.o] Ошибка 255
gmake[2]: *** Ожидание завершения заданий…
gmake[1]: *** [CMakeFiles/Makefile2:1926: ggml/src/ggml-cuda/CMakeFiles/ggml-cuda.dir/all] Ошибка 2
gmake: *** [Makefile:146: all] Ошибка 2

[am17an]

Can you confirm if this is after #18361? If yes, please share full command that doesn't work for you

[vektory79]

Same here

cd llama.cpp
git pull
rm -r ./build
cmake \
    -B ./build \
    -DGGML_CUDA=ON \
    -DGGML_BLAS=OFF \
    -DGGML_NATIVE=OFF \
    -DGGML_RPC=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CPU_ALL_VARIANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES="120f" \
    -DGGML_CUDA_FA_ALL_QUANTS=true \
    -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc --all)

And error:

-- Found CUDAToolkit: /usr/local/cuda/targets/x86_64-linux/include (found version "13.1.80") 
-- CUDA Toolkit found
-- Using CUDA architectures: 120f
CMake Error at /usr/share/cmake-3.28/Modules/Internal/CMakeCUDAArchitecturesValidate.cmake:9 (message):
  CMAKE_CUDA_ARCHITECTURES:

    120f

  is not one of the following:

    * a semicolon-separated list of integers, each optionally
      followed by '-real' or '-virtual'
    * a special value: all, all-major, native

Call Stack (most recent call first):
  /usr/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:116 (cmake_cuda_architectures_validate)
  ggml/src/ggml-cuda/CMakeLists.txt:42 (enable_language)

[vektory79]

But:

cd llama.cpp
git pull
rm -r ./build
cmake \
    -B ./build \
    -DGGML_CUDA=ON \
    -DGGML_BLAS=OFF \
    -DGGML_NATIVE=OFF \
    -DGGML_RPC=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CPU_ALL_VARIANTS=ON \
    -DCMAKE_CUDA_ARCHITECTURES="120a" \
    -DGGML_CUDA_FA_ALL_QUANTS=true \
    -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc --all)

works perfect...

[BahamutRU]

commit 83b3b1c271c78bd77664120431aa8c354d68daac (HEAD, origin/master, origin/HEAD)
Author: Aman Gupta <amangupta052@gmail.com>
Date:   Thu Dec 25 23:55:38 2025 +0800

    cuda: optimize cumsum cub path (#18362)

    * cuda: optimize cumsum cub path

    * remove heavy perf test

commit b0fb0f0aee2c931cbc08e95aef68f615c6de90a2 (tag: b7538, master)
Author: Aman Gupta <amangupta052@gmail.com>
Date:   Thu Dec 25 22:12:11 2025 +0800

    ggml-cuda: fix blackwell native builds (#18361)

    * ggml-cuda: fix blackwell native builds

    Replace 12x in native architectures by 12xa

    * replace for GGML_NATIVE=OFF too

    * only replace for native

    * remove 120f-virtual for default compilation

    ---------

    Co-authored-by: Aman Gupta <aman>

rm -rf build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON -DGGML_CUDA=ON -DGGML_VULKAN=OFF -DGGML_RPC=OFF -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS -DGGML_CUDA_F16=ON -DLLAMA_OPENSSL=ON -DGGML_SCHED_MAX_COPIES=4 -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CCACHE=OFF -DCMAKE_CUDA_ARCHITECTURES=120f-real
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE=Release
-- Found Git: /usr/bin/git (found version "2.43.0")
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
-- BLAS found, Libraries: /usr/lib/x86_64-linux-gnu/libopenblas.so
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.1")
-- Checking for module 'openblas64'
--   Found openblas64, version 0.3.26
-- BLAS found, Includes: /usr/include/x86_64-linux-gnu/openblas64-pthread/
-- Including BLAS backend
-- Found CUDAToolkit: /usr/local/cuda-12.8/targets/x86_64-linux/include (found version "12.8.93")
-- CUDA Toolkit found
-- Using CUDA architectures: 120f-real
CMake Error at /usr/share/cmake-3.28/Modules/Internal/CMakeCUDAArchitecturesValidate.cmake:9 (message):
  CMAKE_CUDA_ARCHITECTURES:

    120f-real

  is not one of the following:

    * a semicolon-separated list of integers, each optionally
      followed by '-real' or '-virtual'
    * a special value: all, all-major, native

Call Stack (most recent call first):
  /usr/share/cmake-3.28/Modules/CMakeDetermineCUDACompiler.cmake:116 (cmake_cuda_architectures_validate)
  ggml/src/ggml-cuda/CMakeLists.txt:42 (enable_language)


-- Configuring incomplete, errors occurred!

[am17an]

That's a CMake bug (it does not understand the f stuff). I think if you upgrade your CMake it should work

[am17an]

@BahamutRU either remove -DCMAKE_CUDA_ARCHITECTURES=120f-real or use -DCMAKE_CUDA_ARCHITECHTURES=120 (it's the same CMake bug)

[BahamutRU]

works perfect...

Yeah, but…

Ptx assembly aborted due to errors

=D

Lets update cmake…