Batchsize bug when larger than 1 during training.

[lwpyh]

Hi there,

Thanks for this amazing work! I noticed that this project only takes 30 minutes to train on 8 GPUs, which is quite impressive. We tried running experiments on one or two A100 80GB GPUs, but I found that the required time exceeds 100 hours, which is very surprising. Have you encountered a similar issue? Could it be that we have not configured something correctly?

Best

[lwpyh]

I try to reproduce with 1 A100, and I got the results as shown in the image after 100 steps

Seems like I do not get the aha moment, is one A100 not suitable for it? Here is my script:

torchrun --nproc_per_node="1"
--nnodes="1"
--node_rank="0"
--master_addr="127.0.0.1"
--master_port="12355"
src/open_r1/grpo.py
--deepspeed local_scripts/zero3.json
--output_dir checkpoints/Qwen2-VL-2B-GRPO-8k
--model_name_or_path Qwen/Qwen2-VL-2B-Instruct
--dataset_name leonardPKU/clevr_cogen_a_train
--max_prompt_length 1024
--per_device_train_batch_size 2
--gradient_accumulation_steps 8
--logging_steps 1
--bf16
--report_to wandb
--gradient_checkpointing false
--attn_implementation flash_attention_2
--max_pixels 401408
--num_train_epochs 1
--run_name Qwen2-VL-2B-GRPO-CLEVR-70k
--save_steps 100
--save_only_model true

I keep gradient_accumulation_stepsper_device_train_batch_sizenproc_per_node as the same as the original training script.

[chenllliang]

hi，the time is for 100steps not 2 epoch in the script. the curve seems to me that the initial reward is too low. I think it might be related to batch decoding problem，can you try with per_device_train_batch_size =1？

Here are more training curves for reference. (train_reward almost equals to acc reward)

[lwpyh]

That works now! many thanks!

[chenllliang]

@lwpyh glad to hear it！Can you also share the screenshot of the new curve here？ and the difference in experiment setting compared to last one. It can help us locate the bugs. thanks a lot！

[lwpyh]

The only difference is the batch_size, I set as 1, while the gradient_accumulation_steps set as 16. I still work on 1 A100. Meanwhile, when I try to run with 2 A100, I encountered a bug as follows: 0%| | 0/8750 [00:00 + 0x145c0 (0x2b9ed1d415c0 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #8: + 0x7ea5 (0x2b9e977b7ea5 in /lib64/libpthread.so.0) frame #9: clone + 0x6d (0x2b9e981d3b0d in /lib64/libc.so.6) terminate called after throwing an instance of 'c10::DistBackendError' what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x2b9f1c065446 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x2b9f1c00f6e4 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libc10.so) frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x2b9f1bf7ca18 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libc10_cuda.so) frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x2b9ee7b81a46 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x2b9ee7b86710 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x2b9ee7b8de7a in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x2b9ee7b8f93d in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #7: + 0x145c0 (0x2b9ed1d415c0 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #8: + 0x7ea5 (0x2b9e977b7ea5 in /lib64/libpthread.so.0) frame #9: clone + 0x6d (0x2b9e981d3b0d in /lib64/libc.so.6) Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x2b9f1c065446 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libc10.so) frame #1: + 0xe7eb1b (0x2b9ee7804b1b in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) frame #2: + 0x145c0 (0x2b9ed1d415c0 in /data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/lib/libtorch.so) frame #3: + 0x7ea5 (0x2b9e977b7ea5 in /lib64/libpthread.so.0) frame #4: clone + 0x6d (0x2b9e981d3b0d in /lib64/libc.so.6) W0203 21:47:52.774000 19482 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 19510 closing signal SIGTERM E0203 21:47:52.938000 19482 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 0 (pid: 19509) of binary: /data/home/acw652/.conda/envs/multi_r1/bin/python3.10 Traceback (most recent call last): File "/data/home/acw652/.conda/envs/multi_r1/bin/torchrun", line 8, in sys.exit(main()) File "/data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper return f(*args, **kwargs) File "/data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main run(args) File "/data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run elastic_launch( File "/data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/data/home/acw652/.conda/envs/multi_r1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ====================================================== src/open_r1/grpo.py FAILED ------------------------------------------------------ Failures: ------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2025-02-03_21:47:52 host : rdg11.apocrita rank : 0 (local_rank: 0) exitcode : -6 (pid: 19509) error_file: traceback : Signal 6 (SIGABRT) received by PID 19509 ======================================================

I located tha bug is in Line 371-374 in grpo_trainer.py. But due to the time limit, I do not go deeper to check how to fix the bug. Anyway, From my side, it works on 1 80G A100, but have some problems on 2.

[chenllliang]

@lwpyh thanks a lot！we are looking at the bsz bug first！the second seems a nccl communication error related to machine.

[JunMa11]

Hi @chenllliang ,

Here are the results on the cell image dataset. I converted the segmentation task as a counting task.

The overall loss also looks good but accuracy reward loss is noising.

Will run the inference and share results. The inference pipeline should be the same as the base model qwen's pipeline, right?

[chenllliang]

@JunMa11 wow, this is amazing! But it seems that the main reward comes from format rather than acc. You can first try disable the format reward. Is the dataset open-source to access? How big are the images? Maybe our team member can help @0xvincii @Yifan-Song793. The inference pipeline is the same, just follow the prompt setting in https://github.com/Deep-Agent/R1-V/blob/412ced0d03da49131642e093281b01e67b401e3a/src/open-r1-multimodal/src/open_r1/grpo.py#L150

[0xvincii]

@chenllliang copy that. @JunMa11 the acc rewards seems very low at the begining~0. Is this task too hard for the baseline model? Maybe some model output and data point will help. Also, can you share the training script?