[Perf] Optimize the Qwen2.5-Omni Model thinker-to-talker-proj with nn.Linear

[kechengliu97]

Summary of Changes

In this submission, we replaced the thinker_to_talker_proj from the original ColumnParallelLinear layer with nn.Linear. This change resulted in a significant performance improvement, particularly in the forward pass of the Qwen2_5OmniTalkerForConditionalGeneration model, where the overall execution time was reduced by over 200 microseconds.

Performance Improvements

• thinker_to_talker_proj latency:

  • Reduced from 283μs 390ns to 223μs 840ns, a decrease of approximately 21%.

• Overall forward time:

  • The total execution time decreased by more than 200 microseconds, significantly improving inference performance.

Configuration Comparison & Reference

In our comparison with the Transformer prototype, we observed that the corresponding linear layers also use nn.Linear in that model. Based on this observation, the updated implementation aligns with mainstream architectures, ensuring higher compatibility and optimizability.

Precision Verification

Using the same prompt, "What is the origin of the United States?", we obtained identical outputs. Both the text and the wav file generated were almost identical, proving that the accuracy has not been affected by this change.

The output text (wav file read the following paragraph):

Prompt:
<|im_start|>system
You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.<|im_end|>
<|im_start|>user
What is the origin of the United States?<|im_end|>
<|im_start|>assistant

vllm_text_output:
Well, you know, the United States has its roots in European colonization. The first Europeans to settle there were mainly from England. They started colonizing the eastern part of North America in the 17th century. People like the Pilgrims landed at Plymouth in 1620.Over time, these colonies grew and developed their own governments. Eventually, they declared independence from Britain in 1776. And then they formed the United States as we know it today.If you want to know more about specific events during that colonial period or how the states were formed, just let me know.

Conclusion

By replacing ColumnParallelLinear with nn.Linear, we not only improved performance but also ensured consistency with Transformer architecture. This optimization will result in lower latency and higher throughput for various inference tasks, with no loss in precision.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 308583dda7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Keep tensor-parallel/quantized linear for projector

Replacing ColumnParallelLinear with nn.Linear drops both tensor-parallel sharding and quant_config handling. In runs with tensor_parallel_size > 1 or quantized checkpoints, this projector will now instantiate full‑precision weights on every rank (no partitioning/quantized kernels), which can inflate memory or prevent quantized weights from loading correctly. Consider keeping the parallel/quantized linear wrapper or providing an equivalent vLLM-aware linear here.

Useful? React with 👍 / 👎.

[kechengliu97]

Meanwhile, Qwen-2.5-Omni-7B is a small model which almost all kinds of NPU/GPU can hold with only one.

[hsliuustc0106]

any accuracy difference?

[kechengliu97]

any accuracy difference?

With the pure text prompt as input, no difference has been observed so far. More importantly, this is what is used in the corresponding part of the Transformer .

[hsliuustc0106]

lgtm

[ZJY0516]

@kechengliu97 Do you have any idea that where the performance improvements come from? ColumnParallelLinear call nn.linear inside actually

[kechengliu97]

@kechengliu97 Do you have any idea that where the performance improvements come from? ColumnParallelLinear call nn.linear inside actually

Our Device can not transfer the profiling swim lane to the outer Internet, but I noticed that using these like ColumnParallelLinear or RowParallelLinear we found some communication cost, which may cause the higher latency.