[Bug]: AssertionError: local_world_size must be less than or equal to the number of visible devices

[jthomson04]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

/opt/dynamo/venv/lib/python3.12/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
Collecting environment information...

    System Info

==============================
OS : Ubuntu 24.04.2 LTS (aarch64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Clang version : Could not collect
CMake version : version 4.1.2
Libc version : glibc-2.39

==============================
PyTorch Info

PyTorch version : 2.9.0+cu129
Is debug build : False
CUDA used to build PyTorch : 12.9
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] (64-bit runtime)
Python platform : Linux-6.14.0-1011-nvidia-64k-aarch64-with-glibc2.39

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 12.9.86
CUDA_MODULE_LOADING set to :
GPU models and configuration :
GPU 0: NVIDIA GB200
GPU 1: NVIDIA GB200
GPU 2: NVIDIA GB200
GPU 3: NVIDIA GB200

Nvidia driver version : 580.105.05
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 144
On-line CPU(s) list: 0-143
Vendor ID: ARM
Model name: Neoverse-V2
Model: 0
Thread(s) per core: 1
Core(s) per socket: 72
Socket(s): 2
Stepping: r0p0
Frequency boost: disabled
CPU(s) scaling MHz: 100%
CPU max MHz: 3393.0000
CPU min MHz: 81.0000
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache: 9 MiB (144 instances)
L1i cache: 9 MiB (144 instances)
L2 cache: 144 MiB (144 instances)
L3 cache: 228 MiB (2 instances)
NUMA node(s): 34
NUMA node0 CPU(s): 0-71
NUMA node1 CPU(s): 72-143
NUMA node2 CPU(s):
NUMA node3 CPU(s):
NUMA node4 CPU(s):
NUMA node5 CPU(s):
NUMA node6 CPU(s):
NUMA node7 CPU(s):
NUMA node8 CPU(s):
NUMA node9 CPU(s):
NUMA node10 CPU(s):
NUMA node11 CPU(s):
NUMA node12 CPU(s):
NUMA node13 CPU(s):
NUMA node14 CPU(s):
NUMA node15 CPU(s):
NUMA node16 CPU(s):
NUMA node17 CPU(s):
NUMA node18 CPU(s):
NUMA node19 CPU(s):
NUMA node20 CPU(s):
NUMA node21 CPU(s):
NUMA node22 CPU(s):
NUMA node23 CPU(s):
NUMA node24 CPU(s):
NUMA node25 CPU(s):
NUMA node26 CPU(s):
NUMA node27 CPU(s):
NUMA node28 CPU(s):
NUMA node29 CPU(s):
NUMA node30 CPU(s):
NUMA node31 CPU(s):
NUMA node32 CPU(s):
NUMA node33 CPU(s):
Vulnerability Gather data sampling: Not affected
Vulnerability Ghostwrite: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, but not BHB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

==============================
Versions of relevant libraries

[pip3] flashinfer-python==0.5.2
[pip3] mypy==1.18.2
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.2.1
[pip3] nvidia-ml-py==13.580.65
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pynvml==13.0.1
[pip3] pytest-mypy==1.0.1
[pip3] pyzmq==26.4.0
[pip3] torch==2.9.0+cu129
[pip3] torchaudio==2.9.0
[pip3] torchvision==0.24.0
[pip3] transformers==4.57.1
[pip3] triton==3.5.0
[pip3] tritonclient==2.62.0
[conda] Could not collect

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.11.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NODE NODE SYS SYS 0-71 0 2
GPU1 NV18 X NV18 NV18 NODE NODE SYS SYS 0-71 0 10
GPU2 NV18 NV18 X NV18 SYS SYS NODE NODE 72-143 1 18
GPU3 NV18 NV18 NV18 X SYS SYS NODE NODE 72-143 1 26
NIC0 NODE NODE SYS SYS X NODE SYS SYS
NIC1 NODE NODE SYS SYS NODE X SYS SYS
NIC2 SYS SYS NODE NODE SYS SYS X NODE
NIC3 SYS SYS NODE NODE SYS SYS NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_4
NIC3: mlx5_5

==============================
Environment Variables

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
CUDA_CACHE_DISABLE=1
NCCL_VERSION=2.26.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VERSION=12.9.0
VLLM_ALLREDUCE_USE_SYMM_MEM=0
VLLM_USE_TRTLLM_ATTENTION=1
NCCL_IB_SL=1
LD_LIBRARY_PATH=/opt/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem_install/lib:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu:/opt/nvidia/nvda_nixl/lib/aarch64-linux-gnu/plugins:/usr/local/ucx/lib:/usr/local/ucx/lib/ucx:/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

I'm running TP8 across two GB200 nodes (4 gpus each) using ray. I'm encountering assertion errors regarding the world size. I only encountered this on 0.11.1; 0.11.0 works fine.

Output of ray status:

Node status
---------------------------------------------------------------
Active:
 1 node_503b8417b7f0e134daceefd694dbbbf80c1b94194a48bb38c2a920cb
 1 node_1d1a7daf21dfe323fecb012b70211f8d0de58027abbabd0b6b3576e6
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Total Usage:
 0.0/288.0 CPU
 0.0/8.0 GPU
 0B/2.80TiB memory
 0B/372.53GiB object_store_memory

From request_resources:
 (none)
Pending Demands:
 (no resource demands)

Run command: vllm serve --model <path to gpt-oss-120b> --tensor-parallel-size 8 --distributed-executor-backend ray

vLLM debug log output (can't post full log due to character limit, so limiting to relevant stack traces instead):

(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 343, in execute_method
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     raise e
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 332, in execute_method
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     return run_method(self, method, args, kwargs)
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 479, in run_method
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 324, in init_device
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 210, in init_device
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]     assert self.parallel_config.local_world_size <= visible_device_count, (
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=1773285) ERROR 11-18 22:45:34 [v1/engine/core.py:842] AssertionError: local_world_size (8) must be less than or equal to the number of visible devices (4).

(APIServer pid=1773137) DEBUG 11-18 22:45:36 [v1/engine/utils.py:954] Waiting for 1 local, 0 remote core engine proc(s) to start.
(APIServer pid=1773137) Traceback (most recent call last):
(APIServer pid=1773137)   File "/opt/dynamo/venv/bin/vllm", line 10, in <module>
(APIServer pid=1773137)     sys.exit(main())
(APIServer pid=1773137)              ^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=1773137)     args.dispatch_function(args)
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=1773137)     uvloop.run(run_server(args))
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
(APIServer pid=1773137)     return __asyncio.run(
(APIServer pid=1773137)            ^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=1773137)     return runner.run(main)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1773137)     return self._loop.run_until_complete(task)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
(APIServer pid=1773137)     return await main
(APIServer pid=1773137)            ^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 2024, in run_server
(APIServer pid=1773137)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 2043, in run_server_worker
(APIServer pid=1773137)     async with build_async_engine_client(
(APIServer pid=1773137)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1773137)     return await anext(self.gen)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 195, in build_async_engine_client
(APIServer pid=1773137)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1773137)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1773137)     return await anext(self.gen)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 236, in build_async_engine_client_from_engine_args
(APIServer pid=1773137)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1773137)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/utils/func_utils.py", line 116, in inner
(APIServer pid=1773137)     return fn(*args, **kwargs)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 203, in from_vllm_config
(APIServer pid=1773137)     return cls(
(APIServer pid=1773137)            ^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 133, in __init__
(APIServer pid=1773137)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1773137)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=1773137)     return AsyncMPClient(*client_args)
(APIServer pid=1773137)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 808, in __init__
(APIServer pid=1773137)     super().__init__(
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 469, in __init__
(APIServer pid=1773137)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=1773137)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1773137)     next(self.gen)
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 907, in launch_core_engines
(APIServer pid=1773137)     wait_for_engine_startup(
(APIServer pid=1773137)   File "/opt/dynamo/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 964, in wait_for_engine_startup
(APIServer pid=1773137)     raise RuntimeError(
(APIServer pid=1773137) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Yang1032]

I've encountered a similar issue in vLLM version 0.11.1. When launching a distributed inference job with a pipeline parallel size of 2 (PP=2) and a tensor parallel size of 8 (TP=8) across two nodes, the system incorrectly throws the following error:

Tensor parallel size (16) cannot be larger than the number of available GPUs (8)

The key point is that this configuration worked correctly and this specific error was not present in versions prior to 0.11.1.

[cjackal]

Upcoming PR #28873 handles your issue.

[Yang1032]

Upcoming PR #28873 handles your issue.

Just wanted to note that this particular bug was identified several days prior to release. Was there a specific reason for pushing v0.11.1 today, considering the known issues?

[cjackal]

I'm not on the maintainer's side, but if I had to say we can include the hotfix in the post-release, just as in v0.10.1.1. The hotfix I mentioned was labled as ready but a few test is flaky these days and the maintainers forgot to cherrypick by mistake I guess.

[njhill]

Sorry about this. There was some misunderstanding and we didn't realize the recent PR that introduced this problem would be included in the release. We'll likely do a post1 hotfix release for this.

[alec-flowers]

Thanks @njhill

[njhill]

Fixed by #28873, we are releasing v0.11.2 with this.

[AIR-hl]

Fixed by #28873, we are releasing v0.11.2 with this.

This problem is still existing in v0.11.2 for me, only v0.11.0 works well