GPTQ Fused MoE class#8

Closed

ElizaWszola wants to merge 82 commits intomarlin-moe-8-bitfrom

Commits on Sep 3, 2024

Commits on Sep 4, 2024

Commits on Sep 5, 2024

[bugfix] >1.43 constraint for openai (#8169)

SolitaryThinker
and
mgoin
authored
[Misc] Clean up RoPE forward_native (#8076)
WoosukKwon
authored
[ci] Mark LoRA test as soft-fail (#8160)
khluu
authored
[Core/Bugfix] Add query dtype as per FlashInfer API requirements. (#8173)
elfiegg
authored
[Doc] Indicate more information about supported modalities (#8181)
DarkLight1337
authored
Inclusion of InternVLChatModel In PP_SUPPORTED_MODELS(Pipeline Parallelism) (#7860)
Manikandan-Thangaraj-ZS0321
authored
[MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (#8029)

alex-jw-brooks
and
DarkLight1337
authored
Move verify_marlin_supported to GPTQMarlinLinearMethod (#8165)
mgoin
authored
[Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (#7962)
sroy745
authored
update/fix weight loading to support tp
dsikka
committed

Commits on Sep 6, 2024

Commits on Sep 7, 2024

Commits on Sep 8, 2024

[Bugfix] Fix async postprocessor in case of preemption (#8267)
alexm-redhat
authored

Commits on Sep 9, 2024

Enable 8-bit weights in Fused Marlin MoE

ElizaWszola
authored and
dsikka
committed
fix rocm

ElizaWszola
authored and
dsikka
committed
bad paste

ElizaWszola
authored and
dsikka
committed
add test case; fix imports for tests
dsikka
committed
fix to adapt custom_routin_function
dsikka
committed
Use select_experts to compute top_k tensors in fused moe

ElizaWszola
authored and
dsikka
committed
bring back fused_moe_marlin -> fused_marlin_moe

ElizaWszola
authored and
dsikka
committed
GPTQ Fused MoE class

ElizaWszola
authored and
dsikka
committed
Add GPTQMarlinMoEMethod to gptq_marlin.py

ElizaWszola
authored and
dsikka
committed
Use FusedMoE layer for all loads

ElizaWszola
authored and
dsikka
committed
Make sure that GPTQ runs through mixtral.py

ElizaWszola
authored and
dsikka
committed
enforce float16A/scales for marlin moe

ElizaWszola
authored and
dsikka
committed
remove large model
dsikka
committed
Cleanup, comments

ElizaWszola
authored and
dsikka
committed
cleanup

ElizaWszola
authored and
dsikka
committed
remove 8-bit stuff for now

ElizaWszola
authored and
dsikka
committed
update/fix weight loading to support tp
dsikka
committed
fix; update large model testing cases
dsikka
committed
add hack to support unfused mixtral pathway for int8
dsikka
committed
fix install for tpu test
dsikka
committed
Move float16 typecast hack to gptq marlin moe method

ElizaWszola
authored and
dsikka
committed
Move output type conversion to gptq method as well

ElizaWszola
authored and
dsikka
committed
typo fix; fix comment
dsikka
committed
Merge branch 'gptq_fused_moe' of https://github.com/neuralmagic/vllm into gptq_fused_moe
ElizaWszola
committed
Clarify comment, change how we process bias
ElizaWszola
committed
[Bugfix] Streamed tool calls now more strictly follow OpenAI's format; ensures Vercel AI SDK compatibility (#8272)
K-Mistele
authored
[Frontend] Add progress reporting to run_batch.py (#8060)

alugowski
and
Adam Lugowski
authored
[Bugfix] Correct adapter usage for cohere and jamba (#8292)
vladislavkruglikov
authored
[Misc] GPTQ Activation Ordering (#8135)
kylesayrs
authored
Merge branch 'main' into gptq_fused_moe
dsikka
authored