[BUG] Video decoding freeze / GPU hang - NVIDIA 580.119.02 + cosmic-comp (Wayland)

[sebschopf]

Overview

Hardware-accelerated video playback causes a full video output freeze and subsequent GPU hang on hybrid graphics laptops using the NVIDIA 580.119.02 driver in a COSMIC Wayland session.

Environment

• OS: Pop!_OS 24.04 LTS
• Kernel: 6.18.7-76061807-generic
• DE: COSMIC (Wayland)
• Driver: NVIDIA 580.119.02 (New Feature Branch)
• Hardware: ASUS ROG Strix G713PU (Hybrid AMD Raphael iGPU + RTX 4050 Mobile dGPU)

Observed Behavior

VLC (APT)

• Playback freezes consistently after ~5 minutes of runtime.
• Audio continues for 10-20 seconds before total blackout of the window.
• Mouse cursor disappears when hovering over the window area.
• Process enters a defunct/zombie state; requires SIGKILL.

MPV (APT)

• Fails to initialize decoding properly.
• Spawns multiple hanging processes (visible in nvidia-smi with 0MiB usage).

Workaround

Disabling HW acceleration in VLC (Input / Codecs -> Hardware-accelerated decoding -> Disable) prevents the freeze. This confirms the issue is localized to the HW decoding path (NVDEC/DMA-BUF handoff).

Reproduction Steps

1. Boot into COSMIC Wayland session.
2. Verify NVIDIA 580.119.02 is active (modinfo nvidia).
3. Launch VLC and ensure Hardware Acceleration is set to "Automatic" or "VA-API/VDPAU".
4. Play any H.264/HEVC container for 5-10 minutes.
5. System/App will freeze during decoding.

Debug Information

Graphics Stack

01:00.0 VGA compatible controller: NVIDIA Corporation AD107M [GeForce RTX 4050 Max-Q / Mobile] (rev a1)
08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev d8)

OpenGL renderer: NVIDIA GeForce RTX 4050 Laptop GPU/PCIe/SSE2
OpenGL version: 4.6.0 NVIDIA 580.119.02

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.119.02             Driver Version: 580.119.02    CUDA Version: 13.0      |
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|=========================================================================================|
|    0   N/A  N/A      3024      G   cosmic-comp                                    80MiB |
|    0   N/A  N/A      6129    C+G   mpv                                            90MiB |
|    0   N/A  N/A      6149    C+G   mpv                                             0MiB |
+-----------------------------------------------------------------------------------------+

Technical Notes

The behavior suggests a regression in DMA-BUF sharing or sync-file handling between the 580.x driver series and cosmic-comp. Since software decoding works, the issue is likely in the NVDEC -> Wayland surface presentation path. High probability of a driver-level deadlock during buffer swap.