[RFC]: Support Ming-Flash-Omni in vLLM-Omni

[raindaywhu]

Motivation.

Ming-flash-omni Preview, an upgraded version of Ming-Omni, built upon a sparser Mixture-of-Experts (MoE) variant of Ling-Flash-2.0. Ming-flash-omni Preview shows competitive performance in vision-text understanding, image generation, audio understanding and text-to-speech capabilitiestechinical report.
The primary objective of this proposal is to adapt this model to the vllm-omni framework.

Proposed Change.

The implementation follows a three-phase roadmap:
Phase 1 : This phase focuses on adapting Ming-flash-omni runing on Ascend with current vllm adaptor.
Phase 2: This phase integrates Ming-flash-omni enabling the full multi-stage pipeline. Thinker for multi-modal data encoder, Talker for llm, Show for VisionDecoder or AudioDecoder.
Phase 3: This phase performance tuning to maximize NPU throughput.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hsliuustc0106]

please check #203 and related PRs #595 #588 #319. You may also need to use OmniConnector #79.

[yinpeiqi]

I think currently the framework can not support ming-flash-omni, as it contains one "talker"+DiT for audio generation and one DiT for vision generation, after a LLM backbone. The current framework only support a sequence of stage lists, which makes it hard to support the branching of audio output and image output. Would ming model require output image and audio in one request?

[hsliuustc0106]

I think currently the framework can not support ming-flash-omni, as it contains one "talker"+DiT for audio generation and one DiT for vision generation, after a LLM backbone. The current framework only support a sequence of stage lists, which makes it hard to support the branching of audio output and image output. Would ming model require output image and audio in one request?

from the paper, I have not seen "output image and audio in one request".

[raindaywhu]

I think currently the framework can not support ming-flash-omni, as it contains one "talker"+DiT for audio generation and one DiT for vision generation, after a LLM backbone. The current framework only support a sequence of stage lists, which makes it hard to support the branching of audio output and image output. Would ming model require output image and audio in one request?

Ming model require output image OR audio in one request.
Multi-modal data encoder => Ling-flash (AR) => Image Decoder
Multi-modal data encoder => Ling-flash (AR) => Audio Decoder

[yinpeiqi]

For online serving the model, we still need to load both model and set an image stage and an audio stage at the same time, while the query output modality info leads to specific stage for generation. Thus we still need to check how our system can solve the branching problem.

[Gaohan123]

Please check the Q1 Roadmap issue #677 . We plan to support custom and Graph-based stage pipeline.