[ROCm] [Feature] [Doc] [Dockerfile] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing#12499

Closed

tjtanaa wants to merge 104 commits intovllm-project:mainfrom

EmbeddedLLM:ptpc-fp8-rocm

Commits on Jan 28, 2025

add Per-token-activation per-channel-weight on-the-fly quantization fp8

kliuae
authored and
tjtanaa
committed
add ptpc fp8 unittests
tjtanaa
committed
remove is_navi check for now
tjtanaa
committed
update rocm gpu installation readme; remove navi check
tjtanaa
committed
update PyTorch version to enable torch._scaled_mm rowwise
tjtanaa
committed
[Misc] Rename `MultiModalInputsV2 -> MultiModalInputs` (#12244)

DarkLight1337
authored and
tjtanaa
committed
[Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (#12237)

jeejeelee
authored and
tjtanaa
committed
[Misc] Remove redundant TypeVar from base model (#12248)

DarkLight1337
authored and
tjtanaa
committed
[Bugfix] Fix mm_limits access for merged multi-modal processor (#12252)

DarkLight1337
authored and
tjtanaa
committed
[torch.compile] transparent compilation with more logging (#12246)

youkaichao
authored and
tjtanaa
committed
[V1][Bugfix] Fix data item ordering in mixed-modality inference (#12259)

ywang96
authored and
tjtanaa
committed
Remove pytorch comments for outlines + compressed-tensors (#12260)

tdoublep
authored and
tjtanaa
committed
[Platform] improve platforms getattr (#12264)

MengqingCao
authored and
tjtanaa
committed
[ci/build] update nightly torch for gh200 test (#12270)

youkaichao
authored and
tjtanaa
committed
[Bugfix] fix race condition that leads to wrong order of token returned (#10802)

joennlae
authored and
tjtanaa
committed
[Kernel] fix moe_align_block_size error condition (#12239)

jinzhen-lin
authored and
tjtanaa
committed
[v1][stats][1/n] Add RequestStatsUpdate and RequestStats types (#10907)

rickyyx
authored and
tjtanaa
committed
[Bugfix] Multi-sequence broken (#11898)

andylolu2
authored and
tjtanaa
committed
[Misc] Remove experimental dep from tracing.py (#12007)

codefromthecrypt
authored and
tjtanaa
committed
[Misc] Set default backend to SDPA for get_vit_attn_backend (#12235)

wangxiyuan
authored and
tjtanaa
committed
[Core] Free CPU pinned memory on environment cleanup (#10477)

janimo
authored and
tjtanaa
committed
[BUGFIX] When skip_tokenize_init and multistep are set, execution crashes (#12277)

authored and
tjtanaa
committed
[Documentation][AMD] Add information about prebuilt ROCm vLLM docker for perf validation purpose (#12281)

hongxiayang
authored and
tjtanaa
committed
[VLM] Simplify post-processing of replacement info (#12269)

DarkLight1337
authored and
tjtanaa
committed
[ci/lint] Add back default arg for pre-commit (#12279)

khluu
authored and
tjtanaa
committed
[CI] add docker volume prune to neuron CI (#12291)

liangfu
authored and
tjtanaa
committed
[Ci/Build] Fix mypy errors on main (#12296)

DarkLight1337
authored and
tjtanaa
committed
[Benchmark] More accurate TPOT calc in `benchmark_serving.py` (#12288)

njhill
authored and
tjtanaa
committed
[core] separate builder init and builder prepare for each batch (#12253)

youkaichao
authored and
tjtanaa
committed
[Build] update requirements of no-device (#12299)

MengqingCao
authored and
tjtanaa
committed
[Core] Support fully transparent sleep mode (#11743)

youkaichao
authored and
tjtanaa
committed
[VLM] Avoid unnecessary tokenization (#12310)

DarkLight1337
authored and
tjtanaa
committed
[Model][Bugfix]: correct Aria model output (#12309)

xffxff
authored and
tjtanaa
committed
[Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (#12313)

ywang96
authored and
tjtanaa
committed
[Doc] Add docs for prompt replacement (#12318)

DarkLight1337
authored and
tjtanaa
committed
[Misc] Fix the error in the tip for the --lora-modules parameter (#12319)

WangErXiao
authored and
tjtanaa
committed
[Misc] Improve the readability of BNB error messages (#12320)

jeejeelee
authored and
tjtanaa
committed
[Bugfix] Fix HPU multiprocessing executor (#12167)

kzawora-intel
authored and
tjtanaa
committed
[Core] Support `reset_prefix_cache` (#12284)

comaniac
authored and
tjtanaa
committed
[Frontend][V1] Online serving performance improvements (#12287)

njhill
authored and
tjtanaa
committed
[AMD][Quantization] Add TritonScaledMMLinearKernel since int8 is broken for AMD (#12282)

rasmith
authored and
tjtanaa
committed
[Bugfix] Fixing AMD LoRA CI test. (#12329)

Alexei-V-Ivanov-AMD
authored and
tjtanaa
committed
[Docs] Update FP8 KV Cache documentation (#12238)

authored and
tjtanaa
committed
[Docs] Document vulnerability disclosure process (#12326)

russellb
authored and
tjtanaa
committed
[V1] Add `uncache_blocks` (#12333)

comaniac
authored and
tjtanaa
committed
[doc] explain common errors around torch.compile (#12340)

youkaichao
authored and
tjtanaa
committed
[Hardware][Gaudi][BugFix] Fix dataclass error due to triton package update (#12338)

zhenwei-intel
authored and
tjtanaa
committed
[Bugfix] Fix k_proj's bias for whisper self attention (#12342)

Isotr0py
authored and
tjtanaa
committed
[Kernel] Flash Attention 3 Support (#12093)

LucasWilkinson
authored and
tjtanaa
committed
[Doc] Troubleshooting errors during model inspection (#12351)

DarkLight1337
authored and
tjtanaa
committed
[V1] Simplify M-RoPE (#12352)

authored and
tjtanaa
committed
[Bugfix] Fix broken internvl2 inference with v1 (#12360)

Isotr0py
authored and
tjtanaa
committed
[core] add wake_up doc and some sanity check (#12361)

youkaichao
authored and
tjtanaa
committed
[torch.compile] decouple compile sizes and cudagraph sizes (#12243)

youkaichao
authored and
tjtanaa
committed
[FP8][Kernel] Dynamic kv cache scaling factors computation (#11906)

authored and
tjtanaa
committed
[TPU] Update TPU CI to use torchxla nightly on 20250122 (#12334)

lsy323
authored and
tjtanaa
committed
[Docs] Document Phi-4 support (#12362)

Isotr0py
authored and
tjtanaa
committed
[BugFix] Fix parameter names and `process_after_weight_loading` for W4A16 MoE Group Act Order (#11528)

authored and
tjtanaa
committed
[Misc] Fix OpenAI API Compatibility Issues in Benchmark Script (#12357)

jsato8094
authored and
tjtanaa
committed
[Docs] Add meetup slides (#12345)

WoosukKwon
authored and
tjtanaa
committed
[Docs] Update spec decode + structured output in compat matrix (#12373)

russellb
authored and
tjtanaa
committed
[V1][Frontend] Coalesce bunched `RequestOutput`s (#12298)

authored and
tjtanaa
committed
Set weights_only=True when using torch.load() (#12366)

russellb
authored and
tjtanaa
committed
[Bugfix] Path join when building local path for S3 clone (#12353)

omer-dayan
authored and
tjtanaa
committed
Update compressed-tensors version (#12367)

dsikka
authored and
tjtanaa
committed
[V1] Increase default batch size for H100/H200 (#12369)

WoosukKwon
authored and
tjtanaa
committed
[perf] fix perf regression from #12253 (#12380)

youkaichao
authored and
tjtanaa
committed
[Misc] Use VisionArena Dataset for VLM Benchmarking (#12389)

ywang96
authored and
tjtanaa
committed
[ci/build] fix wheel size check (#12396)

youkaichao
authored and
tjtanaa
committed
[Hardware][Gaudi][Doc] Add missing step in setup instructions (#12382)

MohitIntel
authored and
tjtanaa
committed
[ci/build] sync default value for wheel size (#12398)

youkaichao
authored and
tjtanaa
committed
[Misc] Enable proxy support in benchmark script (#12356)

jsato8094
authored and
tjtanaa
committed
[Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (#12375)

LucasWilkinson
authored and
tjtanaa
committed
[Misc] Remove deprecated code (#12383)

DarkLight1337
authored and
tjtanaa
committed
[Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only supports pack_gqa (for build size reasons). (#12405)

LucasWilkinson
authored and
tjtanaa
committed
[Bugfix] Fix BLIP-2 processing (#12412)

DarkLight1337
authored and
tjtanaa
committed
[ROCm][MoE] MI300 tuned configs Mixtral-8x(7B,22B) | fp16, fp8 (#12408)

divakar-amd
authored and
tjtanaa
committed
[Misc] Add FA2 support to ViT MHA layer (#12355)

Isotr0py
authored and
tjtanaa
committed
[TPU][CI] Update torchxla version in requirement-tpu.txt (#12422)

lsy323
authored and
tjtanaa
committed
[Misc][Bugfix] FA3 support to ViT MHA layer (#12435)

authored and
tjtanaa
committed
[V1][Perf] Reduce scheduling overhead in model runner after cuda sync (#12094)

youngkent
authored and
tjtanaa
committed
[V1][Bugfix] Fix assertion when mm hashing is turned off (#12439)

ywang96
authored and
tjtanaa
committed
[Misc] Revert FA on ViT #12355 and #12435 (#12445)

ywang96
authored and
tjtanaa
committed
[Frontend] generation_config.json for maximum tokens(#12242)

authored and
tjtanaa
committed
[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (#12417)

authored and
tjtanaa
committed
[Bugfix/CI] Fix broken kernels/test_mha.py (#12450)

tlrmchlsmth
authored and
tjtanaa
committed
[Bugfix][Kernel] Fix perf regression caused by PR #12405 (#12434)

LucasWilkinson
authored and
tjtanaa
committed
[Build/CI] Fix libcuda.so linkage (#12424)

tlrmchlsmth
authored and
tjtanaa
committed
[Frontend] Rerank API (Jina- and Cohere-compatible API) (#12376)

K-Mistele
authored and
tjtanaa
committed
[DOC] Add link to vLLM blog (#12460)

terrytangyuan
authored and
tjtanaa
committed
[V1] Avoid list creation in input preparation (#12457)

WoosukKwon
authored and
tjtanaa
committed
[Frontend] Support scores endpoint in run_batch (#12430)

pooyadavoodi
authored and
tjtanaa
committed
[Bugfix] Fix Granite 3.0 MoE model loading (#12446)

DarkLight1337
authored and
tjtanaa
committed
[Bugfix] Fix missing seq_start_loc in xformers prefill metadata (#12464)

Isotr0py
authored and
tjtanaa
committed
[V1][Minor] Minor optimizations for update_from_output (#12454)

WoosukKwon
authored and
tjtanaa
committed
[Bugfix] Fix gpt2 GGUF inference (#12467)

Isotr0py
authored and
tjtanaa
committed
[Build] Only build 9.0a for scaled_mm and sparse kernels (#12339)

LucasWilkinson
authored and
tjtanaa
committed
[V1][Metrics] Add initial Prometheus logger (#12416)

markmc
authored and
tjtanaa
committed
[V1][CI/Test] Do basic test for top-p & top-k sampling (#12469)

WoosukKwon
authored and
tjtanaa
committed
[FlashInfer] Upgrade to 0.2.0 (#11194)

authored and
tjtanaa
committed
[Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs` with ChunkedPrefill (#10132)

authored and
tjtanaa
committed
Update `pre-commit` hooks (#12475)

hmellor
authored and
tjtanaa
committed
[Neuron][Kernel] NKI-based flash-attention kernel with paged KV cache (#11277)

authored and
tjtanaa
committed
Fix bad path in prometheus example (#12481)

mgoin
authored and
tjtanaa
committed

Uh oh!

[ROCm] [Feature] [Doc] [Dockerfile] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing#12499tjtanaa wants to merge 104 commits intovllm-project:mainvllm-project/vllm:mainfrom EmbeddedLLM:ptpc-fp8-rocmEmbeddedLLM/vllm:ptpc-fp8-rocmCopy head branch name to clipboard

Commits

Commits on Jan 28, 2025

[ROCm] [Feature] [Doc] [Dockerfile] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing#12499
tjtanaa wants to merge 104 commits intovllm-project:mainfrom
EmbeddedLLM:ptpc-fp8-rocm