[Bug] Accuracy issue with SGLang using DeepSeek-R1-AWQ

[TheTinyTeddy]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The inference result of a simple question such as "9.11 and 9.8 which is greater?" can often time (~4 out of 5 times) result in progressively meaningless texts as more tokens are being generated.

The model checkpoint is: https://huggingface.co/cognitivecomputations/DeepSeek-R1-AWQ

SGlang installation from pip install "sglang[all]>=0.4.3.post4" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

Reproduction

python3 -m sglang.launch_server --model /model_ckpt/DeepSeek-R1-AWQ --trust-remote --tp 8 --dtype float16

Environment

sglang[all]>=0.4.3.post4
flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

[jhinpan]

Thank you for raising this issue! We've met with this issue before and our quantization team is investigating it.

[wangjiyang]

Same issue here.

[WALLE-AI]

Same issue here.

[WALLE-AI]

https://huggingface.co/cognitivecomputations/DeepSeek-R1-AWQ
The model deployment test model has garbled characters and large deviations in multiple rounds of dialogue

[slchenchn]

This issue can be resolved if the data type is set to bf16 (refer to vllm-project/vllm#15002). However, my devices do not support bf16. Are there any alternative solutions?

[wangjiyang]

This issue can be resolved if the data type is set to bf16 (refer to vllm-project/vllm#15002). However, my devices do not support bf16. Are there any alternative solutions?

H20*8 has same issue.

[zhyncs]

#4556
#4625
#4719

[AniZpZ]

This issue can be resolved if the data type is set to bf16 (refer to vllm-project/vllm#15002). However, my devices do not support bf16. Are there any alternative solutions?

bf16 provides a larger dynamic range than fp16, there might be a overflow when casting bf16 to fp16. A quantization for activation might solve the problem and we are working on w4a8 quantization for ds r1.

[AniZpZ]

This issue can be resolved if the data type is set to bf16 (refer to vllm-project/vllm#15002). However, my devices do not support bf16. Are there any alternative solutions?

H20*8 has same issue.

H20*8 should be compatible with bf16, and the accuracy issue will be resolved once our PRs are merged

#4462 (comment)

[AniZpZ]

The awq accuracy issue has been fixed in 0.4.4.post4, you can enable "--quantization moe_wna16" option instead of "--dtype float16" to address the problem.

[zilogo]

This issue can be resolved if the data type is set to bf16 (refer to vllm-project/vllm#15002). However, my devices do not support bf16. Are there any alternative solutions?

bf16 provides a larger dynamic range than fp16, there might be a overflow when casting bf16 to fp16. A quantization for activation might solve the problem and we are working on w4a8 quantization for ds r1.

@AniZpZ Hi AniZpZ, it's ok now to support w4a8 for ds r1 671B?