[RFC]: Remove VL Modeling Files

[shen-shanshan]

Motivation.

To avoid maintaining a variety of modeling files in vllm-ascend, we propose to remove all files in models dir in vllm-ascend. After this, the only thing a vllm plugin need to do is just registering their custom device-specific OOT ops to vllm when adding a new model.

To achieve this, there are some refactors need to be done both in vllm and vllm-ascend, such as extracting some general layers as CustomOp

Proposed Change.

vllm:

• Extract MMEncoderAttention as CustomOp. @shen-shanshan [CustomOp][MM] Extract MMEncoderAttention as CustomOp and replace the backend of QwenVisionAttention with it. vllm#30125
• Extract ApplyRotaryEmb as CustomOp. @shen-shanshan [CustomOp] Extract apply_rotary_emb as CustomOp and unify the dispatch logic vllm#29873
• Extract conv layer as custom op. @shen-shanshan [Model][MM] Extract conv layer as CustomOp vllm#28455
• Use caching to remove repeated sin/cos computations. @gcanlin [Model][Perf] Use cos and sin cache in QwenVL vllm#28798
• Remove redundant TP logic in split_qkv. @gcanlin [Refactor] Remove redundant TP gather/split in split_qkv in QwenVL vllm#28271

vllm-ascend:

• Patch VisionAttention layer and remove Qwen2.5-VL modeling files. @shen-shanshan [MM][Model][Perf] Remove Qwen2.5-VL modeling files and add patch for VisionAttention #4349
• Remove Qwen2-VL modeling files. @shen-shanshan [MM][Model] Remove Qwen2-VL modeling files #4534
• Remove Qwen3-VL and Qwen3-VL-MoE modeling files. @shen-shanshan [MM][Model] Remove Qwen3-VL modeling files #4577
• Implement CustomOp for MMEncoderAttention and remove related patch. @shen-shanshan [CustomOp][MM] Register AscendMMEncoderAttention CustomOp and remove related patch #4750
• Implement CustomOp for ApplyRotaryEmb and remove related patch. @shen-shanshan [CustomOp] Implement ApplyRotaryEmb CustomOp and register it #4667
• Implement multimodal_cpu_fields in model runner to guarantee that grid_thw should be moved to cpu before converting to numpy. @zhangxinyuehfad
• Refactor set_ascend_forward_context() to remove patch for ViT embedding. @gcanlin
• Remove patch for cos/sin cache. @shen-shanshan [MM][Patch] Remove patch for cos/sin cache #4672
• Clear all remained patch. @shen-shanshan

Other related:

• Make mamba backend pluggable. @shen-shanshan [Model][Mamba] Add selector for mamba attention backend and make it pluggable for other device vllm#26487

Feedback Period.

No response

CC List.

@Yikun @wangxiyuan @gcanlin

Any Other Things.

No response