Speed up when having padding tokens in DeepEP

[fzyzcjy]

Motivation

test

PYTHONUNBUFFERED=1 SGLANG_TORCH_PROFILER_DIR=/host_home/temp_sglang_server2local python3 -m sglang.launch_server --model-path /dev/shm/DeepSeek-R1 --trust-remote-code --dist-init-addr 192.168.0.55:5757 --nnodes 2 --node-rank ${MY_NODE_RANK} --tp-size ${num_gpu} --dp-size ${num_gpu} --enable-dp-attention --mem-fraction-static 0.8 --chunked-prefill-size $((128*${num_gpu})) --max-running-requests $((${num_gpu}*128)) --context-length 4096 --disable-radix-cache --enable-deepep-moe --deepep-mode low_latency --cuda-graph-bs 128 --decode-log-interval 1

python3 -m sglang.bench_one_batch_server --model-path /dev/shm/DeepSeek-R1 --base-url http://localhost:30000 --batch-size 16 --input-len 1 --output-len 2048 --skip-warmup

• baseline: 6 tok/s/gpu
• PR: 29 tok/s/gpu

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[lambert0312]

This pr will significantly reduce DeepSeek's inference performance (15%+). Need to look at the specific reasons.

[fzyzcjy]

@lambert0312 Looks bad. Could you please show your commands, and would be great to have a profile. My first guess is that, we need to fuse it.

[lambert0312]

@lambert0312 Looks bad. Could you please show your commands, and would be great to have a profile. My first guess is that, we need to fuse it.

@fzyzcjy I tried to modify it. You can see the PR I linked above. Thank you.

[fzyzcjy]

Interesting, I thought this line already makes no extra kernels are executed.