[Bug] Has anyone encountered machine reboot issues when using mooncake-based pd disaggregation?

[wehyy]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

When running bench_serving tests with mooncake-based pd disaggregation, I frequently experience machine crashes and reboots. I'm wondering if anyone else has encountered the same issue?

Interestingly, when I use nixl instead, the system doesn't crash or reboot at all - which is quite puzzling.

Reproduction

MC_TCP_BIND_ADDRESS=192.168.1.1 MOONCAKE_CONFIG_PATH=/data/sglang/mooncake.json
python3 -m sglang.launch_server --model-path /models/DeepSeek-R1 --trust-remote-code
--tp-size 8 --disaggregation-mode prefill --port 30000 --host 192.168.1.1
--mem-fraction-static 0.9 --attention-backend flashinfer

MC_TCP_BIND_ADDRESS=192.168.1.1 MOONCAKE_CONFIG_PATH=/data/sglang/mooncake.json
python3 -m sglang.launch_server --model-path /models/DeepSeek-R1 --trust-remote-code
--tp-size 8 --disaggregation-mode decode --port 30001 --host 192.168.1.2
--mem-fraction-static 0.9 --attention-backend flashinfer

python3 -m sglang.srt.disaggregation.mini_lb --prefill http://192.168.1.1:30000 --decode http://192.168.1.2:30001 --host 0.0.0.0 --port 8000

python3 ./bench_serving.py
--backend sglang --dataset-name random --model /models/DeepSeek-R1
--tokenizer ./tokenizer
--dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json
--random-input-len 2000 --random-output-len 2000 --random-range-ratio 1
--request-rate 16 --max-concurrency 16 --num-prompts 50
--base-url http://127.0.0.1:8000
--host 0.0.0.0 --port 8000

Environment

Python: 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H20
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.61
CUDA Driver Version: 535.161.08
PyTorch: 2.6.0+cu124
sglang: 0.4.5.post2
sgl_kernel: 0.0.9.post2
flashinfer: Module Not Found
triton: 3.2.0
transformers: 4.51.1
torchao: 0.10.0
numpy: 1.26.4
aiohttp: 3.11.11
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.30.2
interegular: 0.3.3
modelscope: 1.25.0
orjson: 3.10.16
outlines: 0.1.11
packaging: 23.2
psutil: 6.1.1
pydantic: 2.10.6
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.2
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.17
openai: 1.75.0
tiktoken: 0.9.0
anthropic: 0.49.0
litellm: 1.67.0
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PHB PIX SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX PHB SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE PIX NODE NODE SYS SYS SYS SYS 0-95,192-287 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS NODE NODE PIX NODE 96-191,288-383 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX NODE NODE 96-191,288-383 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS PHB NODE NODE PIX 96-191,288-383 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS PIX NODE NODE PHB 96-191,288-383 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS
NIC1 NODE NODE NODE PIX SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS
NIC2 NODE PHB PIX NODE SYS SYS SYS SYS NODE NODE X PHB SYS SYS SYS SYS
NIC3 NODE PIX PHB NODE SYS SYS SYS SYS NODE NODE PHB X SYS SYS SYS SYS
NIC4 SYS SYS SYS SYS NODE NODE PHB PIX SYS SYS SYS SYS X NODE NODE PHB
NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE
NIC6 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE
NIC7 SYS SYS SYS SYS NODE NODE PIX PHB SYS SYS SYS SYS PHB NODE NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_bond_1
NIC1: mlx5_bond_2
NIC2: mlx5_bond_3
NIC3: mlx5_bond_4
NIC4: mlx5_bond_5
NIC5: mlx5_bond_6
NIC6: mlx5_bond_7
NIC7: mlx5_bond_8

ulimit soft: 1000000

[ShangmingCai]

MOONCAKE_CONFIG_PATH has been deprecated one week ago: #5460. We recommend regularly merging from the main branch and installing the newest Mooncake pip package to include the latest changes.

Could you please share the crash logs? console output or any error files.

If you suspect NIC issues, you might want to try specifying a working IB device using --disaggregation-ib-device: #5460

[wehyy]

update, The issue of the machine crashing has not been resolved, and the same problem has also occurred on the nixl backend. No obvious error logs were captured on the machine, the socket connection showed no abnormalities, and both memory and CPU utilization appeared normal—though it's possible that I failed to properly capture the error. Below are the software versions encountering this freezing issue:

1. sglang 0.4.5-post3 + mooncake aa31072
2. sglang 0.4.6 + mooncake e0f74db
3. sglang 0.4.6-post4 + nixl e8257bb

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.