Megatron+sglang training occurs nccl timeout during rollout generation

[Xialinxuan]

my environment: 64 H20(8 nodes)+ NCCL version 2.21.5 + cuda12.4
Basic hyperparameter settings:

offload=True
tensor_model_parallel_size=8
pipeline_model_parallel_size=1
sgl_tensor_model_parallel_size=8

Some package version information:

torch                             2.6.0
torch_memory_saver                0.0.8
sgl-kernel                        0.1.4
sglang                            0.4.6.post5
verl                              0.4.1
vllm                              0.8.2
megatron-core                     0.12.1

The log indicates this error occurs when executing the code below:

I can use the same configuration to train with vLLM(tp=8) without issues, but when I switch to sglang and generate rollouts, this NCCL communication timeout occurs(enven with nccl_timeout=7200). I suspect some process is getting stuck for some reason. Are there any methods to debug and identify the specific cause? Let me know if additional details would help resolve this problem.

[qinguangming1999]

I met same issue with vLLM and fsdp, do you solve it?

[SuperCB]

I met same issue with Megatron and sglang

[SuperCB]

my environment: 64 H20(8 nodes)+ NCCL version 2.21.5 + cuda12.4 Basic hyperparameter settings:

offload=True
tensor_model_parallel_size=8
pipeline_model_parallel_size=1
sgl_tensor_model_parallel_size=8

Some package version information:

torch                             2.6.0
torch_memory_saver                0.0.8
sgl-kernel                        0.1.4
sglang                            0.4.6.post5
verl                              0.4.1
vllm                              0.8.2
megatron-core                     0.12.1

The log indicates this error occurs when executing the code below:

I can use the same configuration to train with vLLM(tp=8) without issues, but when I switch to sglang and generate rollouts, this NCCL communication timeout occurs(enven with nccl_timeout=7200). I suspect some process is getting stuck for some reason. Are there any methods to debug and identify the specific cause? Let me know if additional details would help resolve this problem.

Is your data volume too large, causing rollout times to be too long? I think setting nccl_timeout could solve this problem.

[Xialinxuan]

Is your data volume too large, causing rollout times to be too long? I think setting nccl_timeout could solve this problem.

I don't think it's a data loading issue, because I was able to train using the same data and model with the Megatron+vllm combination. If it were a data loading problem, the Megatron+vllm combination should also experience NCCL timeouts. Moreover, I set the nccl_timeout to 7200 seconds and it still timed out.

[SuperCB]

How long is your rollout time?

[Xialinxuan]

How long is your rollout time?

For Megatron + vllm, it takes about 490s to generate squence(response length mean: 1600) and 2000s to finish a step(both logged by wandb).

For Megatron + sglang, I cannot see any records logged(except system information in wandb). It appears that the program was getting stuck at an unknown location during rollout generation until it timed out.

[yvmilir]

I encountered a similar problem using sglang+fsdp.
In step 1, the rollout time was normal , and the difference compared to the experimental results of vllm+fsdp was not significant.
In step 2, the rollout time took a huge amount of time, nearly 30 minutes, and there were timeout errors.
Sglang is version 0.4.6. post4, while verl is version 0.3.1. dev.

step1 rollout time:

step2 rollout time:

errors:

vllm+fsdp is all right.

[Yangruipis]

@zhaochenyang20 I also observed the same problem, step 1 is normal and step 2 rollout too slow until nccl timeout is reached

here is my env

$ pip3 list | egrep "flash|sglang"
flash_attn                               2.8.0.post2
flashinfer-python                        0.2.6.post1
sglang                                   0.4.8.post1

$ python -m torch.utils.collect_env
PyTorch version: 2.7.1+cu126                            
Is debug build: False                                                                                                                       
CUDA used to build PyTorch: 12.6                      
ROCM used to build PyTorch: N/A                                       
                                                                      
OS: Ubuntu 24.04.2 LTS (x86_64)                                 
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0                                                                                          
Clang version: Could not collect             
CMake version: Could not collect                                                                                                            
Libc version: glibc-2.39                     
                                                                                                                                            
Python version: 3.11.13 (main, Jun  5 2025, 13:12:00) [GCC 11.2.0] (64-bit runtime)                                                         
Python platform: Linux-4.18.0-348.7.1.el8_5.x86_64-x86_64-with-glibc2.39                                                                   
Is CUDA available: True                         
CUDA runtime version: 12.9.86                                                                                                               
CUDA_MODULE_LOADING set to: LAZY                               
GPU models and configuration:                                         
GPU 0: NVIDIA H20-3e                                                                                                                        
GPU 1: NVIDIA H20-3e                             
GPU 2: NVIDIA H20-3e                                                                                                                        
GPU 3: NVIDIA H20-3e                         
GPU 4: NVIDIA H20-3e                                                                                                                        
GPU 5: NVIDIA H20-3e                           
GPU 6: NVIDIA H20-3e                                                                                                                        
GPU 7: NVIDIA H20-3e              
                                                                                                                                            
Nvidia driver version: 570.124.06                
cuDNN version: Probably one of the following:                                                                                               
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.10.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.10.2                                                                                            
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.10.2
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2                                                                            
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.10.2
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.10.2                    
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.10.2                                                                                      
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.10.2                     
HIP runtime version: N/A                                                                                                                                                                                                                                                                 
MIOpen runtime version: N/A                                  
Is XNNPACK available: True                                                                                                                                                                                                                                                               
                                                                      
CPU:                                                                                                                                        
Architecture:                    x86_64                                                                                                     
CPU op-mode(s):                  32-bit, 64-bit                                                                                             
Address sizes:                   46 bits physical, 57 bits virtual                                                                          
Byte Order:                      Little Endian                                                                                              
CPU(s):                          208                                                                                                        
On-line CPU(s) list:             0-207                                                                                                      
Vendor ID:                       GenuineIntel                                                                                               
BIOS Vendor ID:                  Intel(R) Corporation                                                                                       
Model name:                      INTEL(R) XEON(R) PLATINUM 8563C                                                                            
BIOS Model name:                 INTEL(R) XEON(R) PLATINUM 8563C  CPU @ 2.6GHz                                                              
BIOS CPU family:                 179                                                                                                        
CPU family:                      6                                                                                                          
Model:                           207                                                                                                        
Thread(s) per core:              2                                                                                                          
Core(s) per socket:              52                                                                                                         
Socket(s):                       2                                                                                                          
Stepping:                        2                                                                                                          
Frequency boost:                 enabled                                                                                                    
CPU(s) scaling MHz:              77%                                                                                                        
CPU max MHz:                     4000.0000                                                                                                  
CPU min MHz:                     800.0000                                                                                                   
BogoMIPS:                        5200.00 

Virtualization:                  VT-x
L1d cache:                       4.9 MiB (104 instances)
L1i cache:                       3.3 MiB (104 instances)
L2 cache:                        208 MiB (104 instances)
L3 cache:                        640 MiB (2 instances)
NUMA node(s):                    2
NUMA node0 CPU(s):               0-51,104-155
NUMA node1 CPU(s):               52-103,156-207
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Versions of relevant libraries:
[pip3] galore-torch==1.0
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] torch==2.7.1
[pip3] torch_memory_saver==0.0.8
[pip3] torchao==0.9.0
[pip3] torchaudio==2.7.1
[pip3] torchdata==0.11.0
[pip3] torchmetrics==1.7.3
[pip3] torchvision==0.22.1
[pip3] transformer_engine_torch==2.4.0
[pip3] triton==3.3.1
[conda] galore-torch                                1.0              pypi_0             pypi
[conda] numpy                                       2.2.6            pypi_0             pypi
[conda] nvidia-cublas-cu12                          12.6.4.1         pypi_0             pypi
[conda] nvidia-cuda-cupti-cu12                      12.6.80          pypi_0             pypi
[conda] nvidia-cuda-nvrtc-cu12                      12.6.77          pypi_0             pypi
[conda] nvidia-cuda-runtime-cu12                    12.6.77          pypi_0             pypi
[conda] nvidia-cudnn-cu12                           9.5.1.17         pypi_0             pypi
[conda] nvidia-cufft-cu12                           11.3.0.4         pypi_0             pypi
[conda] nvidia-curand-cu12                          10.3.7.77        pypi_0             pypi
[conda] nvidia-cusolver-cu12                        11.7.1.2         pypi_0             pypi
[conda] nvidia-cusparse-cu12                        12.5.4.2         pypi_0             pypi
[conda] nvidia-cusparselt-cu12                      0.6.3            pypi_0             pypi
[conda] nvidia-nccl-cu12                            2.26.2           pypi_0             pypi
[conda] nvidia-nvjitlink-cu12                       12.6.85          pypi_0             pypi
[conda] nvidia-nvtx-cu12                            12.6.77          pypi_0             pypi
[conda] torch                                       2.7.1            pypi_0             pypi
[conda] torch-memory-saver                          0.0.8            pypi_0             pypi
[conda] torchao                                     0.9.0            pypi_0             pypi
[conda] torchaudio                                  2.7.1            pypi_0             pypi
[conda] torchdata                                   0.11.0           pypi_0             pypi
[conda] torchmetrics                                1.7.3            pypi_0             pypi
[conda] torchvision                                 0.22.1           pypi_0             pypi
[conda] transformer-engine-torch                    2.4.0            pypi_0             pypi
[conda] triton                                      3.3.1            pypi_0             pypi

cc @SuperCB

[Xialinxuan]

step1 rollout time:

step2 rollout time:

@yvmilir Hi, I'd like to ask how to make the log output include such 'DEBUG' information?

[yvmilir]

step1 rollout time:
step2 rollout time:

@yvmilir Hi, I'd like to ask how to make the log output include such 'DEBUG' information?

You can set VERL_LOGGING_LEVEL=DEBUG before training to enable debug mode logging

[Yangruipis]

@zhaochenyang20 I also observed the same problem, step 1 is normal and step 2 rollout too slow until nccl timeout is reached

here is my env

换成 flashinfer 还是不行，非常奇怪，gpu util满的tensorcore空的，像是hang了，但是显卡温度又很高，他真的在跑东西...

[SuperCB]

all_reduce的时间有13ms

生成一个token的时间竟然有3s

[saldfj1921]

I encountered a similar problem using sglang+fsdp. In step 1, the rollout time was normal , and the difference compared to the experimental results of vllm+fsdp was not significant. In step 2, the rollout time took a huge amount of time, nearly 30 minutes, and there were timeout errors. Sglang is version 0.4.6. post4, while verl is version 0.3.1. dev.

step1 rollout time:

step2 rollout time:

errors: vllm+fsdp is all right.

same problem, step 2 rollout become very slow.... have you solved it ?