[V1] Support MP Executor for multi node distributed inference

[luccafong]

Purpose

Support MP Executor for multi node distributed inference when no ray setup. (Compatible with DP hybrid/internal/external lb)

• How to use:
  vllm serve "model_name" -tp <TP_Size> -dp <DP_Size> -pp <PP_Size> --nnodes <# nodes> --node-rank <rank of node> --master-addr <leader_host_ip> [--master-port <port>] [--headless (if not exposing API endpoint)>] [--data-parallel-external-lb (for external lb) or --data-parallel-hybrid-lb]
  See concrete examples in test plans

Note for DP Compatibility:

• We don't implement new DP coordinator/backend in this PR, we reuse what we already have.
• This PR support auto inference of --data-parallel-rank --data-parallel-local-size --data-parallel-start-rank based on -distributed-node-rank for all 3 dp lb modes, and it works the same with using them explicitly before.
• For engines that do not expose api server (e.g. leader or external dp engine), we need to add --headless (it will auto start a headless DP engine if it is leader(driver) of a DP group; or start a headless Executor Instance if it is non-driver workers)
• This is backward compatible with existing interface for other data parallel.

Architecture change

Test Plan

TP=4 (2 Instances )

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -tp=4 --max-model-len=32768 --nnodes 2 --node-rank 0 --master-addr 127.0.0.1 --port 8000 > /tmp/test_rank0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -tp=4 --max-model-len=32768 --nnodes 2 --node-rank 1 --master-addr 127.0.0.1 --headless > /tmp/test_rank1.log 2>&1 &

PP=2 x TP=2 (2 Instances)

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -pp=2 -tp=2 --max-model-len=32768 --nnodes 2 --node-rank 0 --master-addr 127.0.0.1 --port 8000 > /tmp/test_rank0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -pp=2 -tp=2 --max-model-len=32768 --nnodes 2 --node-rank 1 --master-addr 127.0.0.1 --headless > /tmp/test_rank1.log 2>&1 &

DP 2 (External) * TP 4 (2 in node, 2 across node) (4 Instances)

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 0 --master-addr 127.0.0.1 --port 8000 --data-parallel-external-lb > /tmp/test_dp0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 1 --master-addr 127.0.0.1 --headless > /tmp/test_dp1.log 2>&1 &

CUDA_VISIBLE_DEVICES=4,5 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 2 --master-addr 127.0.0.1 --port 8001 --data-parallel-external-lb > /tmp/test_dp2.log 2>&1 &

CUDA_VISIBLE_DEVICES=6,7 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 3 --master-addr 127.0.0.1 --headless > /tmp/test_dp3.log 2>&1 &

DP 2 (Internal with headless) * TP 4 (2 in node, 2 across node) (4 Instances)

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 0 --master-addr 127.0.0.1 --port 8000 > /tmp/test_dp0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 1 --master-addr 127.0.0.1 --headless > /tmp/test_dp1.log 2>&1 &

CUDA_VISIBLE_DEVICES=4,5 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 2 --master-addr 127.0.0.1 --headless > /tmp/test_dp2.log 2>&1 &

CUDA_VISIBLE_DEVICES=6,7 vllm serve "Qwen/Qwen3-1.7B" -dp=2 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 3 --master-addr 127.0.0.1 --headless > /tmp/test_dp3.log 2>&1 &

DP * 4 (Internal) * TP 2 (inner node) (4 Instances)

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 0 --master-addr 127.0.0.1 --port 8000 > /tmp/test_dp0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=4 --max-model-len=32768 --nnodes 4 --node-rank 1 --master-addr 127.0.0.1 --headless > /tmp/test_dp1.log 2>&1 &

CUDA_VISIBLE_DEVICES=4,5 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 2 --master-addr 127.0.0.1 --headless > /tmp/test_dp2.log 2>&1 &

CUDA_VISIBLE_DEVICES=6,7 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 3 --master-addr 127.0.0.1 --headless > /tmp/test_dp3.log 2>&1 &

DP * 4 (External) * TP 2 (inner node) (4 Instances)

CUDA_VISIBLE_DEVICES=0,1 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 0 --master-addr 127.0.0.1 --port 8000 --data-parallel-external-lb > /tmp/test_dp0.log 2>&1 &

CUDA_VISIBLE_DEVICES=2,3 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 1 --master-addr 127.0.0.1 --port 8001 --data-parallel-external-lb > /tmp/test_dp1.log 2>&1 &

CUDA_VISIBLE_DEVICES=4,5 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 2 --master-addr 127.0.0.1 --port 8002 --data-parallel-external-lb > /tmp/test_dp2.log 2>&1 &

CUDA_VISIBLE_DEVICES=6,7 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 4 --node-rank 3 --master-addr 127.0.0.1 --port 8003 --data-parallel-external-lb > /tmp/test_dp3.log 2>&1 &

DP * 2(external) * 2(internal) Hybrid. * TP * 2 (2 Instances)

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 2 --node-rank 0 --master-addr 127.0.0.1 --data-parallel-hybrid-lb > /tmp/test_dp0.log 2>&1 &

CUDA_VISIBLE_DEVICES=4,5,6,7 vllm serve "Qwen/Qwen3-1.7B" -dp=4 -tp=2 --max-model-len=32768 --nnodes 2 --node-rank 1 --master-addr 127.0.0.1 --data-parallel-hybrid-lb --port 8002 > /tmp/test_dp1.log 2>&1 &

DP * 2 (internal) * PP * 2 * TP * 2(across nodes) (8 Instances)

for i in {0..7}; do
     HEADLESS=""
     if [ $i -gt 0 ]; then HEADLESS="--headless";fi
    CUDA_VISIBLE_DEVICES=$i vllm serve Qwen/Qwen3-1.7B -dp=2  -pp=2 -tp=2 --max-model-len=32768 -n 8 -r  $i --master-addr 127.0.0.1  --port 8000 $HEADLESS > /tmp/test_mp${i}.log 2>&1 &
done

Test Result

Eval tested on all above combinations and on par with baseline, below are some examples.

TP Eval

lm_eval --model local-completions --tasks gsm8k     --model_args model=Qwen/Qwen3-1.7B,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=200,max_retries=3,tokenized_requests=False     --limit 1000

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.681	±	0.0147
		strict-match	5	exact_match	↑	0.678	±	0.0148

Baseline

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.681	±	0.0147
		strict-match	5	exact_match	↑	0.679	±	0.0148

TP Perf

vllm bench serve --model Qwen/Qwen3-1.7B  --port 8003  --dataset-name random  --ignore-eos  --num-prompts 512  --request-rate inf  --random-input-len 1024  --random-output-len 128  --max-concurrency 128

============ Serving Benchmark Result ============
Successful requests:                     512
Maximum request concurrency:             128
Benchmark duration (s):                  5.22
Total input tokens:                      522598
Total generated tokens:                  65536
Request throughput (req/s):              98.05
Output token throughput (tok/s):         12550.62
Total Token throughput (tok/s):          112631.95
---------------Time to First Token----------------
Mean TTFT (ms):                          225.67
Median TTFT (ms):                        235.68
P99 TTFT (ms):                           517.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.39
Median TPOT (ms):                        8.42
P99 TPOT (ms):                           9.68
---------------Inter-token Latency----------------

Baseline

============ Serving Benchmark Result ============
Successful requests:                     512
Maximum request concurrency:             128
Benchmark duration (s):                  5.17
Total input tokens:                      522598
Total generated tokens:                  65536
Request throughput (req/s):              99.08
Output token throughput (tok/s):         12682.43
Total Token throughput (tok/s):          113814.84
---------------Time to First Token----------------
Mean TTFT (ms):                          229.18
Median TTFT (ms):                        231.62
P99 TTFT (ms):                           510.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.25
Median TPOT (ms):                        8.33
P99 TPOT (ms):                           9.50
---------------Inter-token Latency----------------
Mean ITL (ms):                           8.25
Median ITL (ms):                         6.32
P99 ITL (ms):                            31.12
==================================================

TPx PP Eval

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	↑	0.689	±	0.0146
		strict-match	5	exact_match	↑	0.686	±	0.0147

TPx PP Perf

Perf (Multi Instance)

============ Serving Benchmark Result ============
Successful requests:                     512
Maximum request concurrency:             128
Benchmark duration (s):                  7.90
Total input tokens:                      522598
Total generated tokens:                  65536
Request throughput (req/s):              64.77
Output token throughput (tok/s):         8291.08
Total Token throughput (tok/s):          74405.90
---------------Time to First Token----------------
Mean TTFT (ms):                          146.42
Median TTFT (ms):                        123.13
P99 TTFT (ms):                           406.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.30
Median TPOT (ms):                        14.43
P99 TPOT (ms):                           15.01
---------------Inter-token Latency----------------
Mean ITL (ms):                           14.30
Median ITL (ms):                         12.82
P99 ITL (ms):                            43.65
==================================================

Baseline (Single Instance)

============ Serving Benchmark Result ============
Successful requests:                     512
Maximum request concurrency:             128
Benchmark duration (s):                  7.76
Total input tokens:                      522598
Total generated tokens:                  65536
Request throughput (req/s):              66.02
Output token throughput (tok/s):         8450.47
Total Token throughput (tok/s):          75836.33
---------------Time to First Token----------------
Mean TTFT (ms):                          144.65
Median TTFT (ms):                        105.95
P99 TTFT (ms):                           406.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.02
Median TPOT (ms):                        14.20
P99 TPOT (ms):                           14.63
---------------Inter-token Latency----------------
Mean ITL (ms):                           14.02
Median ITL (ms):                         12.54
P99 ITL (ms):                            42.69
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[facebook-github-bot]

@luccafong has imported this pull request. If you are a Meta employee, you can view this in D81078278.

[facebook-github-bot]

@luccafong has imported this pull request. If you are a Meta employee, you can view this in D81078278.

[facebook-github-bot]

@luccafong has imported this pull request. If you are a Meta employee, you can view this in D81078278.

[facebook-github-bot]

@luccafong has imported this pull request. If you are a Meta employee, you can view this in D81078278.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @luccafong.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[luccafong]

@njhill @youkaichao I have addressed comments and args, retested all combinations on lm_eval.

[njhill]

Thanks @luccafong. A few small things...

[houseroad]

Offline synced with @youkaichao and @njhill, and aligned.

Let's land this for now.

[patrickvonplaten]

I don't think it's a good idea to add nnodes as an attribute to the very general ParallelConfig class where it is just meaningful when distributed_executor_backend is mp. Many AI labs rely heavily on RAY and when the distibuted executor backend is ray, nnodes will always be wrong here is it'll stay 1 even if tensor_parallel is set to something like 16 on 8xH200.

It's not intuitive when doing self.vllm_config.parallel_config.nnodes to get 1 here for ray backend and it's not enough to just state in the comment that it's only for mp. IMO we need to make sure that nnodes always displays the correct number of nodes no matter the parallel backend. Similarly node_rank also needs to make sense for ray. If it's impossible to put good values for ray we should at least prefix it with nnodes_for_mp_backend

[luccafong]

nnodes_for_mp_backend might be pretty hard to use, does set them to None be a better way to resolve the conflicts and confusion? and we can raise error if user set it for ray. @patrickvonplaten