Add DeepSeek V3/R1 shared experts fusion

[BBuf]

Motivation

I gain idea mainly from vllm-project/vllm#15502 , thanks for the author's work.I will add references in the modifications for grouped_topk function and DeepSeek v2 model weight_loader. And sgl-kernel's moe_align_kernel kernel has already supported num_experts > 256 in pr, so it's easy to implement fuse shared experts into 256 experts now.

Conclusion

parser.add_argument(
            "--n-share-experts-fusion",
            type=int,
            default=None,
            help="The number of shared_experts need to be replica to fuse with normal experts in deepseek v3/r1 "
            "we use tp_size by default.",
        )

Setting ServerArgs.n_share_experts_fusion=tp_size can yield maximum benefits, in the test data presented below, as the QPS increased to 4, the throughput improved by 4%, while both TTFT and ITL decreased by approximately 15% to 20%.

Acc in H200

➜  sglang git:(support_r1_shared_expers_fusion) ✗ python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 --num-shots 8                                            
100%|████████████████████████████████████████████████████████████████████████| 1319/1319 [01:08<00:00, 19.14it/s]
Accuracy: 0.952
Invalid: 0.000
Latency: 69.547 s
Output throughput: 1998.856 token/s

Benchmark in H200

random

share-gpt

QPS	指标	Baseline (--disable-shared-experts-fusion)	优化版本	改进百分比
1	总吞吐量 (tok/s)	483.47	485.72	+0.5%
	平均TTFT (ms)	949.18	664.25	+30.0%
	平均ITL (ms)	54.69	50.20	+8.2%
4	总吞吐量 (tok/s)	1088.59	1132.73	+4.0%
	平均TTFT (ms)	2630.26	2144.08	+18.5%
	平均ITL (ms)	156.21	132.75	+15.0%
8	总吞吐量 (tok/s)	1188.77	1235.63	+3.9%
	平均TTFT (ms)	6320.67	3443.59	+45.5%
	平均ITL (ms)	188.29	178.94	+5.0%

qps=1

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 --disable-shared-experts-fusion
python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 1 --port 30001

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    1.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  322.06    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60147     
Request throughput (req/s):              0.93      
Input token throughput (tok/s):          295.89    
Output token throughput (tok/s):         187.58    
Total token throughput (tok/s):          483.47    
Concurrency:                             11.07     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   11879.96  
Median E2E Latency (ms):                 7582.23   
---------------Time to First Token----------------
Mean TTFT (ms):                          949.18    
Median TTFT (ms):                        351.17    
P99 TTFT (ms):                           8154.20   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           54.69     
Median ITL (ms):                         40.97     
P95 ITL (ms):                            53.34     
P99 ITL (ms):                            297.26    
Max ITL (ms):                            7614.16   
==================================================

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 
python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 1--port 30001

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    1.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  320.56    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60132     
Request throughput (req/s):              0.94      
Input token throughput (tok/s):          297.27    
Output token throughput (tok/s):         188.45    
Total token throughput (tok/s):          485.72    
Concurrency:                             10.01     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10695.58  
Median E2E Latency (ms):                 6544.49   
---------------Time to First Token----------------
Mean TTFT (ms):                          664.25    
Median TTFT (ms):                        313.28    
P99 TTFT (ms):                           5128.61   
---------------Inter-Token Latency----------------
Mean ITL (ms):                           50.20     
Median ITL (ms):                         39.50     
P95 ITL (ms):                            47.16     
P99 ITL (ms):                            268.81    
Max ITL (ms):                            6598.42   
=================================================

• Total token throughput (tok/s)
  • main: 483.47
  • pr: 485.72
• ttft
  • main: 949.18 ms
  • pr: 664.25 ms(40%)
• itl
  • main: 54.69 ms
  • pr: 50.20 ms(8.9%)

qps=4

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 --disable-shared-experts-fusion
 python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 4 --port 30001

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    4.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  143.03    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60162     
Request throughput (req/s):              2.10      
Input token throughput (tok/s):          666.23    
Output token throughput (tok/s):         422.36    
Total token throughput (tok/s):          1088.59   
Concurrency:                             70.99     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   33845.89  
Median E2E Latency (ms):                 26233.23  
---------------Time to First Token----------------
Mean TTFT (ms):                          2630.26   
Median TTFT (ms):                        997.26    
P99 TTFT (ms):                           13312.57  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           156.21    
Median ITL (ms):                         68.18     
P95 ITL (ms):                            570.93    
P99 ITL (ms):                            908.09    
Max ITL (ms):                            11241.39  
==================================================

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 
python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 4 --port 30001
============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    4.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  137.46    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60056     
Request throughput (req/s):              2.18      
Input token throughput (tok/s):          693.25    
Output token throughput (tok/s):         439.48    
Total token throughput (tok/s):          1132.73   
Concurrency:                             62.55     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   28660.35  
Median E2E Latency (ms):                 21592.16  
---------------Time to First Token----------------
Mean TTFT (ms):                          2144.08   
Median TTFT (ms):                        641.09    
P99 TTFT (ms):                           16040.76  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           132.75    
Median ITL (ms):                         66.72     
P95 ITL (ms):                            506.70    
P99 ITL (ms):                            914.90    
Max ITL (ms):                            8829.84   
==================================================

• Total token throughput (tok/s)
  • main: 1088.59
  • pr: 1132.73 (4.0%)
• ttft
  • main: 2630.26 ms
  • pr: 2144.08 ms(20%)
• itl
  • main: 156.2ms
  • pr: 132.75ms (17%+)

qps=8

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 --disable-shared-experts-fusion
python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 8 --port 30001

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    8.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  130.98    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60151     
Request throughput (req/s):              2.29      
Input token throughput (tok/s):          727.54    
Output token throughput (tok/s):         461.23    
Total token throughput (tok/s):          1188.77   
Concurrency:                             100.71    
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   43969.47  
Median E2E Latency (ms):                 41886.32  
---------------Time to First Token----------------
Mean TTFT (ms):                          6320.67   
Median TTFT (ms):                        5567.51   
P99 TTFT (ms):                           17773.40  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           188.29    
Median ITL (ms):                         67.75     
P95 ITL (ms):                            490.78    
P99 ITL (ms):                            1261.62   
Max ITL (ms):                            11864.62  
==================================================

python3 -m sglang.launch_server --model /DeepSeek-V3 --tp 8 --trust-remote-code --port 30001 
python3 -m sglang.bench_serving --backend sglang --num-prompts 300 --request-rate 8 --port 30001

============ Serving Benchmark Result ============
Backend:                                 sglang    
Traffic request rate:                    8.0       
Max reqeuest concurrency:                not set   
Successful requests:                     300       
Benchmark duration (s):                  126.01    
Total input tokens:                      95293     
Total generated tokens:                  60411     
Total generated tokens (retokenized):    60139     
Request throughput (req/s):              2.38      
Input token throughput (tok/s):          756.23    
Output token throughput (tok/s):         479.41    
Total token throughput (tok/s):          1235.63   
Concurrency:                             93.33     
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   39204.10  
Median E2E Latency (ms):                 37838.59  
---------------Time to First Token----------------
Mean TTFT (ms):                          3443.59   
Median TTFT (ms):                        2325.78   
P99 TTFT (ms):                           14249.98  
---------------Inter-Token Latency----------------
Mean ITL (ms):                           178.94    
Median ITL (ms):                         67.67     
P95 ITL (ms):                            581.96    
P99 ITL (ms):                            1626.09   
Max ITL (ms):                            7362.30   
==================================================

• Total token throughput (tok/s)
  • main: 1188.77
  • pr: 1235.63(3.9%)
• ttft
  • main: 6320.67 ms
  • pr: 3443.59 ms(80%)
• itl
  • main: 188.29 ms
  • pr: 178.94ms(5%)

[fzyzcjy]

Just some nits

[fzyzcjy]

nit: wondering whether randint will be a little bit slower - shall we use something like round-robin

[fzyzcjy]

nit: is it possible that we remove the original shared_experts weight after this, then we save a bit of memory by making that shared_expert never load, and we can remove the logic of

if self.n_shared_experts is not None and self.n_share_fusion_experts == 0:
            shared_output = self.shared_experts(...)

above, since now it is directly None

[ch-wan]

The naming is a little bit confusing. Is it identical with n_share_fusion_experts in the previous files? Why do we need a different name?

[BBuf]

Yeah, I'll handle it.

[ch-wan]

DeepSeek-V2 has 2 shared experts. Should we multiple the number of replica with the number of shared experts?

[ch-wan]

we can call torch.randint(.., out=topk_ids[...]) to save data copy

[ch-wan]

@BBuf I have finished my review. My major concern is how it is compatible with deepseek-v2 where the model has 2 shared experts?

[BBuf]

@BBuf I have finished my review. My major concern is how it is compatible with deepseek-v2 where the model has 2 shared experts?

Thank you. Currently, I am facing an issue with garbled output with the pr. I will address your feedback once I debug the issue and identify the cause.

[fzyzcjy]

Currently, I am facing an issue with garbled output with the pr.

Wondering whether the "gets a bit confused about the numbers" comment above may be a little bit related (again I just used eyes to look at it so can be wildly wrong)

[BBuf]

Currently, I am facing an issue with garbled output with the pr.

Wondering whether the "gets a bit confused about the numbers" comment above may be a little bit related (again I just used eyes to look at it so can be wildly wrong)

No, it's because the logic for share_fusion is not handled in the biased_grouped_topk function. I assumed that the grouped_topk function was being used instead of the biased_grouped_topk function. I will check later when h200 is free.

[fzyzcjy]

Ah I see

[xihuai18]

Is this pr compatible with nextn？

[xihuai18]

Is this pr compatible with nextn？

It seems not:

[2025-04-05 14:23:25 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/path/to/your/sglang/python/sglang/srt/managers/scheduler.py", line 1993, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/managers/scheduler.py", line 261, in __init__
    self.draft_worker = EAGLEWorker(
                        ^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/speculative/eagle_worker.py", line 104, in __init__
    super().__init__(
  File "/path/to/your/sglang/python/sglang/srt/managers/tp_worker.py", line 74, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 170, in __init__
    self.initialize(min_per_gpu_memory)
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 180, in initialize
    self.load_model()
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 392, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_loader/__init__.py", line 22, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_loader/loader.py", line 371, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/path/to/your/sglang/python/sglang/srt/models/deepseek_nextn.py", line 259, in load_weights
    param = params_dict[name]
            ~~~~~~~~~~~^^^^^^
KeyError: 'model.decoder.mlp.shared_experts.down_proj.weight'

[yiakwy-xpu-ml-framework-team]

We should add this to the documentation and to do lists (tunning for mi300x, easy to forget). D

@BBuf

[yiakwy-xpu-ml-framework-team]

Is this pr compatible with nextn？

It seems not:

[2025-04-05 14:23:25 TP3] Scheduler hit an exception: Traceback (most recent call last):
  File "/path/to/your/sglang/python/sglang/srt/managers/scheduler.py", line 1993, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/managers/scheduler.py", line 261, in __init__
    self.draft_worker = EAGLEWorker(
                        ^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/speculative/eagle_worker.py", line 104, in __init__
    super().__init__(
  File "/path/to/your/sglang/python/sglang/srt/managers/tp_worker.py", line 74, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 170, in __init__
    self.initialize(min_per_gpu_memory)
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 180, in initialize
    self.load_model()
  File "/path/to/your/sglang/python/sglang/srt/model_executor/model_runner.py", line 392, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_loader/__init__.py", line 22, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/path/to/your/sglang/python/sglang/srt/model_loader/loader.py", line 371, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/path/to/your/sglang/python/sglang/srt/models/deepseek_nextn.py", line 259, in load_weights
    param = params_dict[name]
            ~~~~~~~~~~~^^^^^^
KeyError: 'model.decoder.mlp.shared_experts.down_proj.weight'

@xihuai18 yep the weights name has been updated from model.layers.{moe_layer}.mlp.shared_experts.{suffix} to model.layers.{moe_layer}.mlp.experts.{n_ronted_experts + shared_expert_id}.{suffix}.

It can be fixed very quickly.

Better to setup a recording list and dependency list:

weights name mangling : model(deepseekv2), nextn(optimiztion module)... should be checked.

[lambert0312]

@xihuai18 yep the weights name has been updated from model.layers.{moe_layer}.mlp.shared_experts.{suffix} to model.layers.{moe_layer}.mlp.experts.{n_ronted_experts + shared_expert_id}.{suffix}.

It can be fixed very quickly.

Better to setup a recording list and dependency list:

weights name mangling : model(deepseekv2), nextn(optimiztion module)... should be checked.

@xihuai18 @yiakwy-xpu-ml-framework-team I also encountered this problem. I modified a version according to the same solution and submitted a PR: #5143

[dongyibo]

@BBuf I have finished my review. My major concern is how it is compatible with deepseek-v2 where the model has 2 shared experts?

hello，How can I integrate with the two shared experts, for example, DeepSeekV2 has two shared experts? @ch-wan @BBuf