[feat]: adapt batch request for flux

[nuclearwu]

Signed-off-by: wuzhongjian [email protected]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

adapt batch request for flux #853 and #797

Test Plan

Vllm-Omni:

1. offline_inference

python examples/offline_inference/text_to_image/text_to_image.py \
  --model black-forest-labs/FLUX.1-dev \
  --prompt "Beautiful illustration of The ocean. in a serene landscape, magic realism, narrative realism, beautiful matte painting, heavenly lighting, retrowave, 4 k hd wallpaper" \
  --seed 42 \
  --cfg_scale 4.0 \
  --tensor_parallel_size 4 \
  --num_images_per_prompt 1 \
  --num_inference_steps 50 \
  --guidance_scale 4.0 \
  --height 1024\
  --width 1024\
  --output outputs/ocean.png

2. online_inference

export VLLM_WORKER_MULTIPROC_METHOD=spawn
MODEL_NAME_OR_PATH=/workspace/cache/ymttest/johnjan/models/black-forest-labs/FLUX___1-dev/

vllm serve ${MODEL_NAME_OR_PATH} \
   --omni \
   --port 8092 \
   --vae_use_slicing \
   --vae_use_tiling

curl -s http://localhost:8092/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Beautiful illustration of The ocean. in a serene landscape, magic realism, narrative realism, beautiful matte painting, heavenly lighting, retrowave, 4 k hd wallpaper"}
    ],
    "extra_body": {
      "height": 1024,
      "width": 1024,
      "num_inference_steps": 50,
      "guidance_scale": 4.0,
      "seed": 42
    }
  }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > ocean-serve.png

3. Benchmark

python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
	--base-url http://localhost:8092 \
	--model /workspace/cache/ymttest/johnjan/models/black-forest-labs/FLUX___1-dev/ \
	--task t2i \
	--dataset vbench \
	--num-prompts 100

Test Result

1.offline_inference

2.online_inference

3. Benchmark
   

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[nuclearwu]

cc @david6666666 @ZJY0516 @hsliuustc0106

[david6666666]

@fhfuih ptal thx

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f108c48697

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

add test plan&results please

[fhfuih]

Looks good to me. Thanks for keeping up with the recent code update

[nuclearwu]

add test plan&results please

@hsliuustc0106 done, please review, thanks

[nuclearwu]

cc @david6666666 @ZJY0516

[hsliuustc0106]

could you provide benchmark results using https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/diffusion

[nuclearwu]

could you provide benchmark results using https://github.com/vllm-project/vllm-omni/tree/main/benchmarks/diffusion

@hsliuustc0106 done, ptal thx

[david6666666]

LGTM, I've already tested it locally, thanks for the fix.