[RFC]: TTS Model Performance Optimization on NPU

[gcanlin]

Motivation.

The performance of the current TTS model is poor on NPU.

Proposed Change.

To be done.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ZhaoJiangJiang]

Hello, I'm setting up the environment by building from source with the following specific versions:

• vLLM: tags/v0.16.0
• vLLM-Ascend: main branch (commit ID: e2175d9c7e62b437391dfee996b1375674ba7c18)
• vLLM-omni: tags/v0.16.0

I start up the online server by vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --port 8091 --stage-configs-path ~/vllm-omni-workspace/vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml. when I send a curl request, the response is extremely slow and eventually ends in a Request Timeout error.

Server's log:

onnxruntime cpuid_info warning: Unknown CPU vendor. cpuinfo_vendor value: 15
/bin/sh: 1: sox: not found
(APIServer pid=25786) [2026-03-04 10:46:05] WARNING __init__.py:10: SoX could not be found!
(APIServer pid=25786) 
(APIServer pid=25786)     If you do not have SoX, proceed here:
(APIServer pid=25786)      - - - http://sox.sourceforge.net/ - - -
(APIServer pid=25786) 
(APIServer pid=25786)     If you do (or think that you should) have SoX, double-check your
(APIServer pid=25786)     path variables.
(APIServer pid=25786)     
(APIServer pid=25786) The tokenizer you are loading from '/home/data/models/Qwen3-TTS/Qwen3-TTS-12Hz-0.6B-CustomVoice/' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=25786) INFO 03-04 10:46:06 [serving_speech.py:514] TTS speech request speech-a1e9562398c8d41a: text='你好，我是通义千问', task_type=CustomVoice

Client's log:

(base) [root@localhost qwen3_tts]# bash run_client.sh 
Model: /home/data/models/Qwen3-TTS/Qwen3-TTS-12Hz-0.6B-CustomVoice/
Task type: CustomVoice
Text: 你好，我是通义千问
Voice: vivian
Generating audio...
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
    return self._connection.handle_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
    raise exc
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
    ) = self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
    data = self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read
    with map_exceptions(exc_map):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/root/miniconda3/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/code/qwen3_tts/vllm-omni/examples/online_serving/qwen3_tts/openai_speech_client.py", line 245, in <module>
    run_tts_generation(args)
  File "/home/code/qwen3_tts/vllm-omni/examples/online_serving/qwen3_tts/openai_speech_client.py", line 107, in run_tts_generation
    response = client.post(api_url, json=payload, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/root/miniconda3/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

[gcanlin]

@ZhaoJiangJiang That looks like be not a performance problem actually.

Could you try this command?

vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --port 8091 --stage-configs-path /root/vllm-workspace/vllm-omni/vllm_omni/platforms/npu/stage_configs/qwen3_tts.yaml

curl -X POST http://localhost:8091/v1/audio/speech     -H "Content-Type: application/json"     -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English"
    }' --output output.wav

[ZhaoJiangJiang]

@ZhaoJiangJiang That looks like be not a performance problem actually.

Could you try this command?

vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --port 8091 --stage-configs-path /root/vllm-workspace/vllm-omni/vllm_omni/platforms/npu/stage_configs/qwen3_tts.yaml

curl -X POST http://localhost:8091/v1/audio/speech     -H "Content-Type: application/json"     -d '{
        "input": "Hello, how are you?",
        "voice": "vivian",
        "language": "English"
    }' --output output.wav

Thanks for the suggestion! That command works perfectly, the curl request now returns a response as expected, and the audio file is generated correctly.
Looking at the difference between the two commands, it seems that need to use the YAML file of vllm-omni/vllm_omni/platforms/npu/stage_configs/qwen3_tts.yaml, rather than the one in vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml.

[gcanlin]

Looking at the difference between the two commands, it seems that need to use the YAML file of vllm-omni/vllm_omni/platforms/npu/stage_configs/qwen3_tts.yaml, rather than the one in vllm-omni/vllm_omni/model_executor/stage_configs/qwen3_tts.yaml.

Oh, yes. vllm-omni/vllm_omni/platforms/npu/stage_configs/qwen3_tts.yaml is adapted for NPU by default.