Skip to content

[AWQ][MoE] Support AWQ MoE - #1644

Merged
kyuyeunk merged 23 commits into
vllm-project:mainfrom
catswe:support-awq-moe
Mar 6, 2026
Merged

[AWQ][MoE] Support AWQ MoE#1644
kyuyeunk merged 23 commits into
vllm-project:mainfrom
catswe:support-awq-moe

Conversation

@catswe

@catswe catswe commented Feb 6, 2026

Copy link
Copy Markdown
Contributor

Description

Enabled AWQ. Ran on v6e-8.

Changes:

To preserve the memory benefit of AWQ, I dequantized on the fly and added zero_point parameters. The changes from adding zero_point is similar in logic as scale parameters

tpu_inference/layers/vllm/quantization/awq.py

  • Referenced vllm/model_executor/layers/quantization/awq_marlin.py
  • Referenced tpu_inference/layers/vllm/quantization/fp8.py
  • Referenced tpu_inference/layers/vllm/quantization/mxfp4.py
  • Defined get_fused_moe_quant_config since otherwise attribute access error
  • Implemented create_weights directly and uses extra_weight_attrs because upstream has no base AWQ MoE method to inherit from, unlike the other TPU MoE methods

Tests

Due to changes from #1659, doing pip install -e . may be needed before running the following command

MODEL_IMPL_TYPE=vllm vllm serve QuantTrio/MiniMax-M2-AWQ --gpu-memory-utilization=0.95 --tensor_parallel_size=8 --enable-expert-parallel --dtype bfloat16 --download-dir /dev/shm
Screenshot 2026-02-27 at 6 26 31 PM

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have made or will make corresponding changes to any relevant documentation.

@lk-chen lk-chen changed the title Support AWQ MoE [AWQ][MoE] Support AWQ MoE Feb 6, 2026
@catswe
catswe force-pushed the support-awq-moe branch 2 times, most recently from 5c91f5a to 0a5b4fa Compare February 11, 2026 23:43
@catswe
catswe force-pushed the support-awq-moe branch 5 times, most recently from 50bc01a to 421ab1b Compare February 21, 2026 22:10
@catswe catswe changed the title [AWQ][MoE] Support AWQ MoE [Draft][AWQ][MoE] Support AWQ MoE Feb 23, 2026
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
@catswe catswe changed the title [Draft][AWQ][MoE] Support AWQ MoE [AWQ][MoE] Support AWQ MoE Feb 26, 2026
@catswe
catswe marked this pull request as ready for review February 26, 2026 17:40
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Comment thread tpu_inference/kernels/megablox/gmm.py Outdated
Comment thread tpu_inference/kernels/megablox/gmm.py Outdated
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
@kyuyeunk kyuyeunk added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 28, 2026
Comment thread tpu_inference/kernels/megablox/gmm.py Outdated
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Signed-off-by: catswe <212922539+catswe@users.noreply.github.com>
Comment thread tpu_inference/layers/vllm/quantization/awq.py
Comment thread tpu_inference/layers/vllm/quantization/awq.py
@catswe

catswe commented Mar 6, 2026

Copy link
Copy Markdown
Contributor Author

i prefer creating a follow up pr!

@kyuyeunk

kyuyeunk commented Mar 6, 2026

Copy link
Copy Markdown
Collaborator

i prefer creating a follow up pr!

thanks! will merge it!

@kyuyeunk
kyuyeunk merged commit fd433f8 into vllm-project:main Mar 6, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants