[Misc] Fused MoE Marlin support for GPTQ

[dsikka]

Summary

• Add GPTQ Marlin MoE Support; marlin MoE kernels currently support int4
• Update/add optional testing for large MoE models for GPTQ and llm-compressor

Co-authored by @ElizaWszola, from Neural Magic

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mgoin]

I need to run a quick test on it myself, but this looks good to land for 4bit support!

[mgoin]

Performance looks great!

python benchmarks/benchmark_latency.py --model mistralai/Mixtral-8x7B-Instruct-v0.1 --tensor-parallel-size 2 --input-len 128 --output-len 512 --batch-size 1 --num-iters-warmup 2 --num-iters 10
Avg latency: 5.879553547129035 seconds

python benchmarks/benchmark_latency.py --model nm-testing/Mixtral-8x7B-Instruct-v0.1-W4A16-quantized --tensor-parallel-size 2 --input-len 128 --output-len 512 --batch-size 1 --num-iters-warmup 2 --num-iters 10
Avg latency: 4.726654114946723 seconds

python benchmarks/benchmark_latency.py --model TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ --tensor-parallel-size 2 --input-len 128 --output-len 512 --batch-size 1 --num-iters-warmup 2 --num-iters 10
Avg latency: 4.787863119132817 seconds

Before this PR, GPTQ Mixtral would be much slower:

python benchmarks/benchmark_latency.py --model TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ --tensor-parallel-size 2 --input-len 128 --output-len 512 --batch-size 1 --num-iters-warmup 2 --num-iters 10
Avg latency: 8.206900223530829 seconds

[fengyang95]

Does this support deepseek-v2?

[xiaoqi35]

Thanks !
That's an important feature for deepseek-v2. Quantized deepseek-v2 models need FuseMoE that supports int4 quantization.
So Awq quantization is also OK?

[ElizaWszola]

Sonnet benchmark results (no act order, 4-bit):

// 4-bit quantized moe without act order
llm = LLM(model="TheBloke/Mixtral-8x7B-v0.1-GPTQ")

TTFT	TPOT

[ElizaWszola]

Thanks ! That's an important feature for deepseek-v2. Quantized deepseek-v2 models need FuseMoE that supports int4 quantization. So Awq quantization is also OK?

AWQ is currently in our future work scope!