The missing guide for training Qwen3-VL MOE

[huaiyizhao]

Anyone interested in training qwen3-vl moe models but without success can try the following.

1. Use the official base image [base-verl0.6-cu128-cudnn9.8-torch2.8.0-fa2.7.4](https://hub.docker.com/layers/verlai/verl/base-verl0.6-cu128-cudnn9.8-torch2.8.0-fa2.7.4/images/sha256-6bcff875bfe58350b238ddac7f975fcda6f99f55dee42d8918199357ab3aa0ef)
2. pip install vllm==0.11.0 which supports qwen3 vl rollout
3. clone verl (optional: checkout to commit 2c9e19e which I have personally tested) and install
4. install mbridge pip install -U git+https://github.com/ISEEKYAN/mbridge.git
5. install latest transformers for qwen3vl support pip install 'transformers>=4.57.0'
6. install mcore pip install --no-deps --no-cache-dir git+https://github.com/NVIDIA/Megatron-LM.git@core_v0.13.1

Then you can run the example in examples/grpo_trainer/run_qwen3_vl-30b-megatron.sh

Good luck!

[baobaohanhan21]

thanks, can it run success with 8*H20 ? I suffered OOM issue When save ckpts after 20 steps.

[huaiyizhao]

Yes, I am using 8*H20 myself. But I lower the actor.megatron.expert_model_parallel_size to 4 and actor.megatron.tensor_model_parallel_size to 2 which I think is compatible with 8 GPUs.

[huaiyizhao]

I have built a docker image for it docker pull zhyict/verl:verl0.7.0-transformers4.57.1-vllm0.11.0-mcore0.13.1

[enbacoo]

I have built a docker image for it docker pull zhyict/verl:verl0.7.0-transformers4.57.1-vllm0.11.0-mcore0.13.1

which cudnn version is used? I met the error below. Other dependency versions are consistent with yours

File "/Megatron-LM/megatron/core/transformer/transformer_layer.py", line 875, in call
return super(MegatronModule, self).call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Megatron-LM/megatron/core/transformer/transformer_layer.py", line 441, in forward
hidden_states, context = self._forward_attention(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Megatron-LM/megatron/core/transformer/transformer_layer.py", line 501, in _forward_attention
attention_output_with_bias = self.self_attention(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Megatron-LM/megatron/core/transformer/attention.py", line 737, in forward
core_attn_out = self.core_attention(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Megatron-LM/megatron/core/extensions/transformer_engine.py", line 931, in forward
core_attn_out = super().forward(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/external_utils.py", line 198, in nonrecursive_disable_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/transformer_engine/pytorch/attention/dot_product_attention/dot_product_attention.py", line 1010, in forward
raise ValueError(
ValueError: No dot product attention backend is available for the provided inputs. Please run with NVTE_DEBUG=1 NVTE_DEBUG_LEVEL=2 to find out the reasons for disabling all backends.

[huaiyizhao]

nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.10.0
transformer_engine 2.2.0+d0c452c

[huaiyizhao]

For aync vllm rollout, see #3934

[YaguangGong]

Anyone interested in training qwen3-vl moe models but without success can try the following.

1. Use the official base image [base-verl0.6-cu128-cudnn9.8-torch2.8.0-fa2.7.4](https://hub.docker.com/layers/verlai/verl/base-verl0.6-cu128-cudnn9.8-torch2.8.0-fa2.7.4/images/sha256-6bcff875bfe58350b238ddac7f975fcda6f99f55dee42d8918199357ab3aa0ef)
2. pip install vllm==0.11.0 which supports qwen3 vl rollout
3. clone verl (optional: checkout to commit 2c9e19e which I have personally tested) and install
4. install mbridge pip install -U git+https://github.com/ISEEKYAN/mbridge.git
5. install latest transformers for qwen3vl support pip install 'transformers>=4.57.0'
6. install mcore pip install --no-deps --no-cache-dir git+https://github.com/NVIDIA/Megatron-LM.git@core_v0.13.1

Then you can run the example in examples/grpo_trainer/run_qwen3_vl-30b-megatron.sh

Good luck!

Followed your steps, but still got unsupported error messages, how to debug?

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/workspace/verl/verl/trainer/main_ppo.py", line 434, in
main()
File "/usr/local/lib/python3.12/dist-packages/hydra/main.py", line 94, in decorated_main
_run_hydra(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
raise ex
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
return func()
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/utils.py", line 458, in
lambda: hydra.run(
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/lib/python3.12/dist-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/trainer/main_ppo.py", line 42, in main
run_ppo(config)
File "/workspace/verl/verl/trainer/main_ppo.py", line 91, in run_ppo
ray.get(runner.run.remote(config))
File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2882, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 968, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(ValueError): �[36mray::TaskRunner.run()�[39m (pid=29694, ip=10.217.27.83, actor_id=bb636bb153de67c3caef0e7801000000, repr=<main_ppo.TaskRunner object at 0x7f93422479b0>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/trainer/main_ppo.py", line 333, in run
trainer.init_workers()
File "/workspace/verl/verl/trainer/ppo/ray_trainer.py", line 753, in init_workers
self.ref_policy_wg.init_model()
File "/workspace/verl/verl/single_controller/ray/base.py", line 48, in call
output = ray.get(output)
^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.RayTaskError(ValueError): �[36mray::WorkerDict.ref_init_model()�[39m (pid=34422, ip=10.217.27.83, actor_id=2300dbb93ab516914bb78a0d01000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7f1992d9ef00>)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/single_controller/ray/base.py", line 700, in func
return getattr(self.worker_dict[key], name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/single_controller/base/decorator.py", line 442, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/utils/transferqueue_utils.py", line 199, in dummy_inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/workers/megatron_workers.py", line 517, in init_model
self.ref_module, self.ref_model_config = self._build_model_optimizer(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/verl/verl/workers/megatron_workers.py", line 299, in _build_model_optimizer
self._init_hf_config_and_tf_config(
File "/workspace/verl/verl/workers/megatron_workers.py", line 161, in _init_hf_config_and_tf_config
bridge = AutoBridge.from_config(hf_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/mbridge/core/auto_bridge.py", line 50, in from_config
raise ValueError(
ValueError: Unregistered model type: qwen3_vl, now only support dict_keys(['deepseek_v3', 'llama', 'qwen2', 'mimo', 'mixtral', 'qwen2_5_vl', 'qwen2_moe', 'qwen3', 'qwen3_moe', 'glm4_moe', 'glm4v', 'glm4v_moe', 'gemma3', 'internvl_chat'])

[huaiyizhao]

Try to uninstall mbridge and reinstall from github pip install -U git+https://github.com/ISEEKYAN/mbridge.git

[ISEEKYAN]

thanks for the guidance, I am working in progress to support CP for long context, would support in a few days. Feel free to contact me with any question.
For now you can use docker image verlai/verl:vllm011.dev1

[ISEEKYAN]

I have refactored the implementation to support context parallel with #3998. I recommend to use this PR with image verlai/verl:vllm011.dev_qwenvl_cp

[ddy-ddy]

I get the error:
askRunner pid=128989, ip=127.0.0.1) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::WorkerDict.actor_rollout_compute_log_prob() (pid=143735, ip=127.0.0.1, actor_id=8412434267f92ed8ac3a66e201000000, repr=<verl.single_controller.ray.base.WorkerDict object at 0x7f628b064260>) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/single_controller/ray/base.py", line 700, in func (TaskRunner pid=128989, ip=127.0.0.1) return getattr(self.worker_dict[key], name)(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/single_controller/base/decorator.py", line 442, in inner (TaskRunner pid=128989, ip=127.0.0.1) return func(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/transferqueue_utils.py", line 199, in dummy_inner (TaskRunner pid=128989, ip=127.0.0.1) return func(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/profiler/performance.py", line 105, in f (TaskRunner pid=128989, ip=127.0.0.1) return self.log(decorated_function, *args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/profiler/performance.py", line 118, in log (TaskRunner pid=128989, ip=127.0.0.1) output = func(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/profiler/profile.py", line 256, in wrapper (TaskRunner pid=128989, ip=127.0.0.1) return func(self_instance, *args, **kwargs_inner) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/workers/megatron_workers.py", line 744, in compute_log_prob (TaskRunner pid=128989, ip=127.0.0.1) output, entropys = self.actor.compute_log_prob(data=data, calculate_entropy=True) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/profiler/performance.py", line 105, in f (TaskRunner pid=128989, ip=127.0.0.1) return self.log(decorated_function, *args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/utils/profiler/performance.py", line 118, in log (TaskRunner pid=128989, ip=127.0.0.1) output = func(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/workers/actor/megatron_actor.py", line 217, in compute_log_prob (TaskRunner pid=128989, ip=127.0.0.1) output = self.forward_backward_batch( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/workers/actor/megatron_actor.py", line 596, in forward_backward_batch (TaskRunner pid=128989, ip=127.0.0.1) losses_reduced = forward_backward_func( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 500, in forward_backward_no_pipelining (TaskRunner pid=128989, ip=127.0.0.1) output_tensor, num_tokens = forward_step( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/megatron/core/pipeline_parallel/schedules.py", line 289, in forward_step (TaskRunner pid=128989, ip=127.0.0.1) output_tensor, loss_func = forward_step_func(data_iterator, model) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/workers/actor/megatron_actor.py", line 559, in forward_step (TaskRunner pid=128989, ip=127.0.0.1) output = forward_fn( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/opt/tiger/open_verl/verl/models/mcore/model_forward.py", line 53, in model_forward (TaskRunner pid=128989, ip=127.0.0.1) output_orig = model( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl (TaskRunner pid=128989, ip=127.0.0.1) return self._call_impl(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl (TaskRunner pid=128989, ip=127.0.0.1) return forward_call(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/megatron/core/distributed/data_parallel_base.py", line 22, in forward (TaskRunner pid=128989, ip=127.0.0.1) return self.module(*inputs, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl (TaskRunner pid=128989, ip=127.0.0.1) return self._call_impl(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl (TaskRunner pid=128989, ip=127.0.0.1) return forward_call(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/megatron/core/transformer/module.py", line 237, in forward (TaskRunner pid=128989, ip=127.0.0.1) outputs = self.module(*inputs, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl (TaskRunner pid=128989, ip=127.0.0.1) return self._call_impl(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl (TaskRunner pid=128989, ip=127.0.0.1) return forward_call(*args, **kwargs) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/mbridge/models/qwen3_vl/model.py", line 340, in forward (TaskRunner pid=128989, ip=127.0.0.1) input_ids_thd, _ = preprocess_packed_seqs( (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) File "/home/tiger/.local/lib/python3.12/site-packages/mbridge/core/util.py", line 631, in preprocess_packed_seqs (TaskRunner pid=128989, ip=127.0.0.1) seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32) (TaskRunner pid=128989, ip=127.0.0.1) ^^^^^^^^^^^^^^^^^^ (TaskRunner pid=128989, ip=127.0.0.1) AttributeError: 'NoneType' object has no attribute 'sum'

Has anyone else encountered this problem, my environment is installed according to the above material~