[Bug]:Text-to-image model parallel inference failed: concurrent /v1/images/generations calls – one request never gets HTTP response (generation succeeds twice, but only one 200 OK)

[iheee-google]

Your current environment

The output of python collect_env.py

Collecting environment information...
WARNING 01-29 18:23:47 [mooncake_connector.py:18] Mooncake not available, MooncakeOmniConnector will not work

    System Info

==============================
OS : Ubuntu 22.04.4 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : Could not collect
CMake version : version 3.22.1
Libc version : glibc-2.35

==============================
PyTorch Info

PyTorch version : 2.9.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A

==============================
Python Environment

Python version : 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform : Linux-5.15.0-86-generic-x86_64-with-glibc2.35

==============================
CUDA / GPU Info

Is CUDA available : True
CUDA runtime version : 12.4.131
CUDA_MODULE_LOADING set to :
GPU models and configuration : GPU 0: NVIDIA A100-PCIE-40GB
Nvidia driver version : 550.90.07
cuDNN version : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True

==============================
CPU Info

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Vendor ID: GenuineIntel
Model name: Intel Xeon Processor (Cascadelake)
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
Stepping: 6
BogoMIPS: 5986.16
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat pku ospke avx512_vnni
L1d cache: 2.5 MiB (80 instances)
L1i cache: 2.5 MiB (80 instances)
L2 cache: 160 MiB (40 instances)
L3 cache: 32 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-39
NUMA node1 CPU(s): 40-79
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown: Vulnerable
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

==============================
Versions of relevant libraries

[pip3] flashinfer-python==0.5.3
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.17.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.5
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.0
[pip3] torchaudio==2.9.0
[pip3] torchsde==0.2.6
[pip3] torchvision==0.24.0
[pip3] transformers==4.57.6
[pip3] triton==3.5.0
[conda] flashinfer-python 0.5.3 pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
[conda] nvidia-cudnn-frontend 1.17.0 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
[conda] nvidia-cutlass-dsl 4.3.5 pypi_0 pypi
[conda] nvidia-ml-py 13.590.48 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.27.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
[conda] nvidia-nvshmem-cu12 3.3.20 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
[conda] pyzmq 27.1.0 pypi_0 pypi
[conda] torch 2.9.0 pypi_0 pypi
[conda] torchaudio 2.9.0 pypi_0 pypi
[conda] torchsde 0.2.6 pypi_0 pypi
[conda] torchvision 0.24.0 pypi_0 pypi
[conda] transformers 4.57.6 pypi_0 pypi
[conda] triton 3.5.0 pypi_0 pypi

==============================
vLLM Info

ROCM Version : Could not collect
vLLM Version : 0.12.0
vLLM-Omni Version : 0.12.0rc1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-79 0-1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

==============================
Environment Variables

NVIDIA_VISIBLE_DEVICES=GPU-048d485d-0791-39d9-394c-17bff1ef670c
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NCCL_VERSION=2.21.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics,video
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.4.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
OMP_NUM_THREADS=10
MKL_NUM_THREADS=10
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

Your code version

The commit id or version of vllm

0.12.0

The commit id or version of vllm-omni

0.12.0rc1

🐛 Describe the bug

vllm serve /root/autodl-tmp/models/Z-Image-Turbo \
  --omni \
  --port 8091 \
  --gpu_memory_utilization 0.9

• Mode: pure diffusion (single diffusion stage)
• Hardware:
  • GPU: [please fill in, e.g. A100 GPU memory size]
  • Number of GPUs: 1
• OS / Python:
  • Python: 3.12 (from container)
• Client: requests (Python)

import time
import threading
import requests

URL = "http://localhost:8091/v1/images/generations"
PAYLOAD = {
"prompt": "可爱的卡通橘猫，白色背景",
"n": 1,
"size": "512x512",
"response_format": "b64_json",
"num_inference_steps": 8,
}

def run(i):
start = time.time()
try:
resp = requests.post(URL, json=PAYLOAD, timeout=60)
cost = time.time() - start
print(f"[req {i}] status={resp.status_code} cost={cost:.2f}s")
except Exception as e:
cost = time.time() - start
print(f"[req {i}] ERROR cost={cost:.2f}s err={repr(e)}")

threads = [threading.Thread(target=run, args=(i,)) for i in range(2)]
for t in threads:
t.start()
for t in threads:
t.join()

Output (representative):

[req 1] status=200 cost=1.14s
[req 0] ERROR cost=60.06s err=ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8091): Read timed out. (read timeout=60)"))

---

Server Logs During This Repro

For the above two concurrent requests, the server logs show:

(APIServer pid=...) INFO ... Generating 1 image(s) 512x512    # request A
(APIServer pid=...) INFO ... Generating 1 image(s) 512x512    # request B

[Stage-0] INFO ... Generation completed successfully.         # generation A done
[Stage-0] INFO ... Post-processing completed ...

(APIServer pid=...) INFO [log_utils.py:549] {'type': 'request_level_metrics',
(APIServer pid=...) INFO [log_utils.py:549]  'request_id': 'img_gen_1769674978',
(APIServer pid=...) INFO [log_utils.py:549]  'e2e_time_ms': 1033.39,
(APIServer pid=...) INFO [log_utils.py:549]  'stages': {0: {...}}}

(APIServer pid=...) INFO [async_omni.py:468] [Summary] {'e2e_requests': 1, ...}
(APIServer pid=...) INFO [api_server.py:696] Successfully generated 1 image(s)
(APIServer pid=...) INFO: 127.0.0.1:44958 - "POST /v1/images/generations HTTP/1.1" 200 OK

[Stage-0] INFO ... Generation completed successfully.         # generation B done
[Stage-0] INFO ... Post-processing completed ...
# (no second request_level_metrics, no second 200 OK log)

So from the logs:

• Both requests are accepted (Generating 1 image(s) 512x512 printed twice).
• Both generations complete successfully in Stage-0.
• But only one request is counted in e2e_requests: 1 and only one 200 OK is written to the HTTP log.

---

Expected Behavior

• For two concurrent /v1/images/generations requests with the same payload:
  • Both should either succeed with HTTP 200 in ~1–2 seconds, or
  • If there is some kind of queueing, the second one should still get an HTTP response well within the client timeout (60s in this repro).

Most importantly, for every HTTP request, there should be a corresponding HTTP response.

---

• With 2 concurrent requests:

  • One request returns 200 OK in ~1.1s.

  • The other request hangs until the client-side requests.post(..., timeout=60) hits a ReadTimeout:

    ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8091): Read timed out. (read timeout=60)")
    

• On the server side:

  • Both diffusion generations complete successfully (Stage-0 logs).
  • Only one of the two requests is reflected in request_level_metrics and e2e_requests: 1.
  • Only one 200 OK is logged by the API server.
  • The other request appears to never get an HTTP response written, even though the diffusion stage finished.

This reproduces consistently with 2 concurrent requests (and also when vLLM is called behind a FastAPI gateway).

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ZJY0516]

@iheee-google Could you please try the latest main branch code. It seems you are using outdated vllm-omni and vllm

[iheee-google]

@iheee-google Could you please try the latest main branch code. It seems you are using outdated vllm-omni and vllm

@ZJY0516 I appreciate your advice, I've already tested the v0.14 release, but unfortunately, the issue persists. Are there any specific additional parameters needed to enable vLLM parallel inference and ensure proper result return?

[iheee-google]

@iheee-google Could you please try the latest main branch code. It seems you are using outdated vllm-omni and vllm@iheee 谷歌请您试试最新的主分行代码。看起来你用的是过时的 VLLM-omni 和 VLLM

@ZJY0516 I appreciate your advice, I've already tested the v0.14 release, but unfortunately, the issue persists. Are there any specific additional parameters needed to enable vLLM parallel inference and ensure proper result return?感谢你的建议，我已经测试过 v0.14 版本，但遗憾的是，问题依旧存在。是否需要额外的特定参数来实现 vLLM 并行推断并确保正确结果返回？

The log becomes：

(APIServer pid=81674) INFO 01-30 09:55:06 [api_server.py:1000] Generating 1 image(s) 512x512
(APIServer pid=81674) INFO 01-30 09:55:06 [async_omni.py:317] [AsyncOrchestrator] Entering scheduling loop: stages=1, final_stage=0
(APIServer pid=81674) INFO 01-30 09:55:06 [api_server.py:1000] Generating 1 image(s) 512x512
(APIServer pid=81674) INFO 01-30 09:55:06 [async_omni.py:317] [AsyncOrchestrator] Entering scheduling loop: stages=1, final_stage=0
[Stage-0] INFO 01-30 09:55:06 [manager.py:538] Deactivating all adapters: 0 layers
[Stage-0] WARNING 01-30 09:55:06 [kv_transfer_manager.py:356] No connector available for receiving KV cache
[Stage-0] INFO 01-30 09:55:07 [diffusion_engine.py:80] Generation completed successfully.
[Stage-0] INFO 01-30 09:55:07 [diffusion_engine.py:98] Post-processing completed in 0.0072 seconds
[Stage-0] INFO 01-30 09:55:07 [manager.py:538] Deactivating all adapters: 0 layers
[Stage-0] WARNING 01-30 09:55:07 [kv_transfer_manager.py:356] No connector available for receiving KV cache
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550] {'type': 'request_level_metrics',
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'request_id': 'img_gen_1769738106',
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'e2e_time_ms': 864.7818565368652,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'e2e_tpt': 0.0,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'e2e_total_tokens': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'transfers_total_time_ms': 0.0,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'transfers_total_bytes': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]  'stages': {0: {'stage_gen_time_ms': 854.4080257415771,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]                 'num_tokens_out': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [log_utils.py:550]                 'num_tokens_in': 0}}}
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352] [Summary] {'e2e_requests': 1,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'e2e_total_time_ms': 864.8297786712646,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'e2e_sum_time_ms': 864.7818565368652,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'e2e_total_tokens': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'e2e_avg_time_per_request_ms': 864.7818565368652,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'e2e_avg_tokens_per_s': 0.0,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'wall_time_ms': 864.8297786712646,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'final_stage_id': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'stages': [{'stage_id': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]              'requests': 1,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]              'tokens': 0,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]              'total_time_ms': 864.7880554199219,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]              'avg_time_per_request_ms': 864.7880554199219,
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]              'avg_tokens_per_s': 0.0}],
(APIServer pid=81674) INFO 01-30 09:55:07 [async_omni.py:352]  'transfers': []}
(APIServer pid=81674) INFO 01-30 09:55:07 [api_server.py:1058] Successfully generated 1 image(s)
(APIServer pid=81674) INFO:     127.0.0.1:46726 - "POST /v1/images/generations HTTP/1.1" 200 OK
[Stage-0] INFO 01-30 09:55:08 [diffusion_engine.py:80] Generation completed successfully.
[Stage-0] INFO 01-30 09:55:08 [diffusion_engine.py:98] Post-processing completed in 0.0083 seconds

[ZJY0516]

cc @david6666666

[iheee-google]

cc @david6666666

I’ve identified the root cause and fixed the issue .

The core problem lies in how the /v1/images/generations endpoint generates request_id:

• Previously, it used a second-level timestamp (e.g., img_gen_1769738106) to create the ID. This meant concurrent requests arriving in the same second would receive identical request_ids.
• When this happened, the later request would overwrite the earlier one’s state in self.request_states.
• The original request’s coroutine would then get stuck indefinitely on await queue.get() until the client timed out, and the logs would only count 1 request.
• This explains why saw Generation completed successfully. twice in the backend, but only one 200 response in the frontend, with the other request timing out.

---

I’ve modified the file vllm_omni/entrypoints/openai/api_server.py:

1. Added an import for the uuid module at the top of the file.
2. Changed the request_id generation logic: instead of using a timestamp, it now generates a globally unique ID with uuid.uuid4().hex, keeping the img_gen_ prefix.
3. This ensures every request gets a unique ID even under high concurrency, preventing state overwrites and resolving the timeout issue.

---

[ZJY0516]

@iheee-google Thanks a lot. Could you please open a pr to fix this? I can fix it either

[iheee-google]

@iheee-google Thanks a lot. Could you please open a pr to fix this? I can fix it either

No problem — you can take care of putting up the PR，I’ll leave it to you

By the way, after applying the request_id fix, I ran a concurrent test with a script to verify, and found a new issue: the 8 inference requests are not actually running in parallel on the GPU — they’re being processed serially instead.

Test Script

import time
import threading
import requests

URL = "http://localhost:8091/v1/images/generations"
PAYLOAD = {
    "prompt": "可爱的卡通橘猫，白色背景",
    "n": 1,
    "size": "512x512",
    "response_format": "b64_json",
    "num_inference_steps": 8,
}

def run(i):
    start = time.time()
    try:
        resp = requests.post(URL, json=PAYLOAD, timeout=60)
        cost = time.time() - start
        print(f"[req {i}] status={resp.status_code} cost={cost:.2f}s")
    except Exception as e:
        cost = time.time() - start
        print(f"[req {i}] ERROR cost={cost:.2f}s err={repr(e)}")

threads = [threading.Thread(target=run, args=(i,)) for i in range(8)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Test Result

[req 1] status=200 cost=1.14s
[req 3] status=200 cost=2.08s
[req 4] status=200 cost=2.98s
[req 0] status=200 cost=3.83s
[req 5] status=200 cost=4.73s
[req 2] status=200 cost=5.68s
[req 6] status=200 cost=6.52s
[req 7] status=200 cost=7.42s

You can see the response time increases sequentially for each request, which confirms the inferences are consumed one after another on the GPU, not in true parallel.

So does this mean the GPU concurrent inference feature hasn’t been implemented yet?

[ZJY0516]

@iheee-google Yes. The diffusion inference is not like LLM can use chunk prefill, continuous batching and so on, because it's already compute bound at most time. We've already work on batch inference, But it's not ready.

We recommend using parallel inference instead of batch.

[iheee-google]

@iheee-google Yes. The diffusion inference is not like LLM can use chunk prefill, continuous batching and so on, because it's already compute bound at most time. We've already work on batch inference, But it's not ready.

We recommend using parallel inference instead of batch.

---

Got it, thanks for the clarification! I have a few follow-up questions about parallel inference:

1. How exactly do I enable and use the parallel inference feature? Could you share the specific steps or configuration needed?
2. Can this parallel inference work in an online service setting (like our current /v1/images/generations endpoint)? Our main requirement is to handle concurrent requests efficiently in our production service.

---

[ZJY0516]

https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --usp 2 --ring 2
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  11.25          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.44           
Latency Mean (s):                        2.2505         
Latency Median (s):                      2.2485         
Latency P99 (s):                         2.2574         

============================================================

vllm serve Tongyi-MAI/Z-Image-Turbo --omni -tp 2
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  13.93          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.36           
Latency Mean (s):                        2.7851         
Latency Median (s):                      2.7754         
Latency P99 (s):                         2.8329         

============================================================

vllm serve Tongyi-MAI/Z-Image-Turbo --omni
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  21.36          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.23           
Latency Mean (s):                        4.2721         
Latency Median (s):                      4.2446         
Latency P99 (s):                         4.3785         

============================================================

The benchmark command is

python diffusion_benchmark_serving.py --port 8000 --model Tongyi-MAI/Z-Image-Turbo --task t2i --dataset random --num-prompts 5 --num-inference-steps 9

You can find diffusion_benchmark_serving.py in this repo

[ZJY0516]

Our main requirement is to handle concurrent requests efficiently in our production service.

The key question is whether a FIFO scheduling policy would violate your SLO.

[iheee-google]

https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --usp 2 --ring 2
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  11.25          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.44           
Latency Mean (s):                        2.2505         
Latency Median (s):                      2.2485         
Latency P99 (s):                         2.2574         

============================================================

vllm serve Tongyi-MAI/Z-Image-Turbo --omni -tp 2
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  13.93          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.36           
Latency Mean (s):                        2.7851         
Latency Median (s):                      2.7754         
Latency P99 (s):                         2.8329         

============================================================

vllm serve Tongyi-MAI/Z-Image-Turbo --omni
================= Serving Benchmark Result =================
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 random         
Task:                                    t2i            
--------------------------------------------------
Benchmark duration (s):                  21.36          
Request rate:                            inf            
Max request concurrency:                 1              
Successful requests:                     5/5              
--------------------------------------------------
Request throughput (req/s):              0.23           
Latency Mean (s):                        4.2721         
Latency Median (s):                      4.2446         
Latency P99 (s):                         4.3785         

============================================================

The benchmark command is基准命令为

python diffusion_benchmark_serving.py --port 8000 --model Tongyi-MAI/Z-Image-Turbo --task t2i --dataset random --num-prompts 5 --num-inference-steps 9

You can find diffusion_benchmark_serving.py in this repo你可以在这个仓库中找到 diffusion_benchmark_serving.py

@ZJY0516 Thanks again for your reply! I apologize for forgetting to mention this earlier — actually, I only have one GPU on my end..Do these parallel inference features all require multiple GPUs to work? Is there any way to use them with a single GPU?