[Diffusion] Refactor diffusion models weights loading

[Isotr0py]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

• This PR introduce DiffusersPipelineLoader to load weights under the subfolder of standard diffusers mode repo.
• This PR also reuses the AutoWeightsLoader from main repo to load weights recursively.

Test Plan

python examples/offline_inference/qwen_image/text_to_image.py

python examples/offline_inference/qwen_image/text_to_image.py --model Tongyi-MAI/Z-Image-Turbo --num_inference_steps 10

Test Result

Both Z-Image and Qwen-Image pipeline can still work.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

The account who enabled Codex for this repo no longer has access to Codex. Please contact the admins of this repo to enable Codex again.

[Isotr0py]

I think we should also unifiy text_encoder, vae and other submodule's weights loading along this way in following PRs.

Otherwise AutoModel.from_pretrained has loaded their weights at model initialization stage here, which will make weights-based parallism integration a bit difficult to be handled.

[ZJY0516]

Agreed

[ZJY0516]

LGTM. Does this PR support modelscope weight load?

[ZJY0516]

Agreed

[Isotr0py]

Does this PR support modelscope weight load?

Yes, it works with the modelscope patch omni_snapshot_download at entrypont level:

vllm-omni/vllm_omni/entrypoints/omni.py

Lines 14 to 22 in f73791d

	def omni_snapshot_download(model_id) -> str:
	# TODO: this is just a workaround for quickly use modelscope, we should support
	# modelscope in weight loading feature instead of using `snapshot_download`
	if os.environ.get("VLLM_USE_MODELSCOPE", False):
	from modelscope.hub.snapshot_download import snapshot_download
	return snapshot_download(model_id)
	else:
	return _dummy_snapshot_download(model_id)

It still can't work without this patch, because some vllm-omni specific weights utils functions don't have modelscope workaround:

vllm-omni/vllm_omni/model_executor/model_loader/weight_utils.py

Lines 13 to 19 in f73791d

	def download_weights_from_hf_specific(
	model_name_or_path: str,
	cache_dir: Optional[str],
	allow_patterns: list[str],
	revision: Optional[str] = None,
	ignore_patterns: Optional[Union[str, list[str]]] = None,
	) -> str:

[ZJY0516]

Does this PR support modelscope weight load?

Yes, it works with the modelscope patch omni_snapshot_download at entrypont level:

vllm-omni/vllm_omni/entrypoints/omni.py

Lines 14 to 22 in f73791d

	def omni_snapshot_download(model_id) -> str:
	# TODO: this is just a workaround for quickly use modelscope, we should support
	# modelscope in weight loading feature instead of using `snapshot_download`
	if os.environ.get("VLLM_USE_MODELSCOPE", False):
	from modelscope.hub.snapshot_download import snapshot_download
	return snapshot_download(model_id)
	else:
	return _dummy_snapshot_download(model_id)

It still can't work without this patch, because some vllm-omni specific weights utils functions don't have modelscope workaround:

vllm-omni/vllm_omni/model_executor/model_loader/weight_utils.py

Lines 13 to 19 in f73791d

	def download_weights_from_hf_specific(
	model_name_or_path: str,
	cache_dir: Optional[str],
	allow_patterns: list[str],
	revision: Optional[str] = None,
	ignore_patterns: Optional[Union[str, list[str]]] = None,
	) -> str:

Do you know does vllm already provides similar modelscope-compatible utilities? also cc @MengqingCao

[SamitHuang]

it will be good to show the memory usage and time cost for model loading, similar to vllm

(EngineCore_DP0 pid=2964220) INFO 12-04 06:28:57 [default_loader.py:267] Loading weights took 3.21 seconds
(EngineCore_DP0 pid=2964220) INFO 12-04 06:28:57 [gpu_model_runner.py:2653] Model loading took 1.6714 GiB and 5.854078 seconds

[Isotr0py]

Done in 22f0f84

INFO 12-04 17:45:34 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO:vllm_omni.diffusion.worker.gpu_worker:Worker 0: Initialized device and distributed environment.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00,  1.58s/it]
WARNING 12-04 17:45:39 [__init__.py:755] Current vLLM config is not set.
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:03<00:06,  3.48s/it]
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:06<00:03,  3.34s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:08<00:00,  2.50s/it]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:08<00:00,  2.74s/it]

INFO:vllm_omni.diffusion.model_loader.diffusers_loader:Loading weights took 8.44 seconds
INFO:vllm_omni.diffusion.worker.gpu_worker:Model loading took 19.2180 GiB and 14.825355 seconds
INFO:vllm_omni.diffusion.worker.gpu_worker:Worker 0: Model loaded successfully.
INFO:vllm_omni.diffusion.worker.gpu_worker:Worker 0: Scheduler loop started.
INFO:vllm_omni.diffusion.worker.gpu_worker:Worker 0 ready to receive requests via shared memory

[ZJY0516]

LGTM, generally