Environment:
GPU: NVIDIA GeForce RTX 4060 Laptop GPU (SM 8.9, 8GB VRAM)
Driver: 591.74 → 610.62 (both affected)
CUDA: tested 12.4, 12.8, 13.3 (all affected)
ggml: v0.15.3, llama.cpp b9914 (both affected)
OS: Windows 10, MSVC 19.44/19.50
Reproduction:
Run depth-anything V3 BASE model (q4_k, 100MB gguf) with CUDA backend. Model loads and offloads 275 weights to GPU successfully, then crashes during inference.
Root cause tracked down:
Added cudaStreamSynchronize after each cublasGemmEx call to isolate the issue. The crash is an illegal memory access that happens during the 3rd cuBLAS call in the sequence:
#1 OK: (972×768) @ (588×768)^T
#2 OK: (972×768) @ (1536×768)^T
#3 CRASH: (252×768) @ (6912×768)^T ← illegal memory access
The illegal access is likely from a preceding kernel (dequant / mmvq / convert) in the same stream, not from cuBLAS itself. compute-sanitizer --tool memcheck reports 0 errors in Release build (likely optimized away). LLM inference works fine because the matrix shapes are larger and more regular.
Key observation:
The crash does NOT occur on cuBLAS calls for LLM workloads (which use large, power-of-2 matmul dimensions like 4096×4096). It only triggers on ViT-style shapes with odd dimensions (252, 588, 6912, 972, etc.). This suggests a shape-dependent buffer overflow or alignment issue in one of the CUDA matmul helper kernels.
Debug log:
DEBUG cublas #1: ne01=972 ne11=768 ne10=588 s01=588 s11=588 ne0=972
DEBUG cublas #1: OK
DEBUG cublas #2: ne01=972 ne11=768 ne10=1536 s01=1536 s11=1536 ne0=972
DEBUG cublas #2: OK
ggml_cuda_compute_forward: MUL_MAT failed
CUDA error: an illegal memory access was encountered
Environment:
GPU: NVIDIA GeForce RTX 4060 Laptop GPU (SM 8.9, 8GB VRAM)
Driver: 591.74 → 610.62 (both affected)
CUDA: tested 12.4, 12.8, 13.3 (all affected)
ggml: v0.15.3, llama.cpp b9914 (both affected)
OS: Windows 10, MSVC 19.44/19.50
Reproduction:
Run depth-anything V3 BASE model (q4_k, 100MB gguf) with CUDA backend. Model loads and offloads 275 weights to GPU successfully, then crashes during inference.
Root cause tracked down:
Added cudaStreamSynchronize after each cublasGemmEx call to isolate the issue. The crash is an illegal memory access that happens during the 3rd cuBLAS call in the sequence:
#1 OK: (972×768) @ (588×768)^T
#2 OK: (972×768) @ (1536×768)^T
#3 CRASH: (252×768) @ (6912×768)^T ← illegal memory access
The illegal access is likely from a preceding kernel (dequant / mmvq / convert) in the same stream, not from cuBLAS itself. compute-sanitizer --tool memcheck reports 0 errors in Release build (likely optimized away). LLM inference works fine because the matrix shapes are larger and more regular.
Key observation:
The crash does NOT occur on cuBLAS calls for LLM workloads (which use large, power-of-2 matmul dimensions like 4096×4096). It only triggers on ViT-style shapes with odd dimensions (252, 588, 6912, 972, etc.). This suggests a shape-dependent buffer overflow or alignment issue in one of the CUDA matmul helper kernels.
Debug log:
DEBUG cublas #1: ne01=972 ne11=768 ne10=588 s01=588 s11=588 ne0=972
DEBUG cublas #1: OK
DEBUG cublas #2: ne01=972 ne11=768 ne10=1536 s01=1536 s11=1536 ne0=972
DEBUG cublas #2: OK
ggml_cuda_compute_forward: MUL_MAT failed
CUDA error: an illegal memory access was encountered