Bug & Fix: Multi-GPU results in static noise

[pftq]

Only 1 part of the image is clear. If 2 GPUs, it's the top half the image. If 4, it's the top 4th of the image. So it looks like there's something wrong with the way the multi-GPU stitches the other GPU output back together. I did a fresh pull and download the models to be sure.

This is my prompt:

GPUs=4
CFG=6
Steps=30
Frames=129
torchrun --nproc_per_node=$GPUs \
sample_image2video.py \
    --model HYVideo-T/2 \
    --prompt "" \
    --i2v-mode \
    --i2v-image-path "image.jpg" \
    --i2v-resolution 720p \
    --i2v-stability \
    --video-length $Frames \
    --infer-steps $Steps \
    --flow-reverse \
    --flow-shift 7.0 \
    --embedded-cfg-scale $CFG \
    --save-path ./results \
    --ulysses-degree $GPUs \
    --ring-degree 1 \
    --video-size 960 960 \
    --xdit-adaptive-size

[grimulkan]

Yes, I have the same issue, and actually had it in the original T2V model as well for multi-GPU.

[pftq]

I've been trying to debug this issue this week, and it seems like the main issue is that the repo code attempts to split the video spatially (top half for one GPU, bottom half for the other). This is done in inference.py's parallelize_transformer.

However, it seems like xfuser is meant for sequence parallelism (splitting the video by time), so it's strange that Hunyuan's multi-GPU handling is coded for spatial parallelism instead. The Skyreels fine-tune doesn't have this issue, and their approach is also to split sequentially instead of spatially. So I wonder if there was a mistake for whoever was in charge of adding the multi-GPU support to the public repo that doesn't line up with Hunyuan's multi-GPU code internally.

Lastly, it seems like a lot of the parallelism and splitting to GPUs is being done manually in inference.py instead of actually letting xfuser (xDiT) do it. So it's all kind of weird here and doesn't seem like what Hunyuan actually used for their benchmarks/internal

[pftq]

@TianQi-777 Any thoughts on this? I went down a pretty deep rabbithole trying to debug it, but no luck. I tried rewriting the parallelism transformer code to do sequential parallelism as well but oddly enough then the video ends up static noise in the second half instead of the bottom half of the video. However there was a point where I got the rank 1 (second GPU) output to show in between the frames of the rank 0 (main GPU) output so it seems that the other GPUs are producing actual video. It's in the piecing together back to the full video that's getting messed up.

[TianQi-777]

Thank you for your attention. We are testing this problem.

[pftq]

Thanks - let me know if you'd like more tests from my side since it seems some others are using this fine. I am using Runpod with 2, 4, and 8 H100s.

[xibosun]

The issue is fixed in pr #53

[pftq]

Thanks - I'm inspecting the changes and notice the xdit_adaptive_size was removed. Is that a final change and should we remove it from our commandline options as well?

[grimulkan]

Appears to work now! I had to update xfusers, yunchang and flash-attn to the latest versions beyond what is in requirements.txt.

OT: without an official fp8 model for I2V (yet), it still ends up OOM in cases where the single-GPU variant does not, egs., during vae decode. VAE tiling or other approaches will probably help, or releasing the fp8 weights like was done for T2V.

[pftq]

Appears to work now! I had to update xfusers, yunchang and flash-attn to the latest versions beyond what is in requirements.txt.

OT: without an official fp8 model for I2V (yet), it still ends up OOM in cases where the single-GPU variant does not, egs., during vae decode. VAE tiling or other approaches will probably help, or releasing the fp8 weights like was done for T2V.

VAE tiling would probably do it. I believe the HunyuanGP version by deepbeepmeep has an implementation that could probably fit here but I haven't had a chance to look. As a work around at the moment, I'm just setting the resolution to 540p instead (still renders a decent 720x720) which gets me at least 241 frames on a H100 (with Riflex extrapolation applied).

[pftq]

I've given this a test and it doesn't seem to be a complete fix imo. I'm testing 4xGPUs on 480x480x49f with 360p resolution and it resizes to 480x448 only to leave vertical black bars on the sides (even with the ALLOW_RESIZE_FOR_SP disabled). It seems like 480x480 should easily divide into 4 GPUs.

Seems ok at 2 GPUs though (480x480) so maybe an edge case with 4.

A bigger issue is it seems like when the static noise comes back it's after the 4th generation in a row. This is for both 2 GPUs and 4 GPUs. The video height/width seem the same across all videos. Is there some kind of clearing that needs to happen to reset the RAM or something? This is by doing a loop around the inference call like this:

    for idx in range(40): 
            outputs = hunyuan_video_sampler.predict(
            prompt=args.prompt, 
            height=args.video_size[0],
            width=args.video_size[1],
            video_length=args.video_length,
            seed=args.seed,
            negative_prompt=args.neg_prompt,
            infer_steps=args.infer_steps,
            guidance_scale=args.cfg_scale,
            num_videos_per_prompt=args.num_videos,
            flow_shift=args.flow_shift,
            batch_size=args.batch_size,
            embedded_guidance_scale=args.embedded_cfg_scale,
            i2v_mode=args.i2v_mode,
            i2v_resolution=args.i2v_resolution,
            i2v_image_path=args.i2v_image_path,
            i2v_condition_type=args.i2v_condition_type,
            i2v_stability=args.i2v_stability,
            ulysses_degree=args.ulysses_degree,
            ring_degree=args.ring_degree,
            xdit_adaptive_size=args.xdit_adaptive_size
        )
        samples = outputs['samples']

[xibosun]

I've given this a test and it doesn't seem to be a complete fix imo. I'm testing 4xGPUs on 480x480x49f with 360p resolution and it resizes to 480x448 only to leave vertical black bars on the sides (even with the ALLOW_RESIZE_FOR_SP disabled). It seems like 480x480 should easily divide into 4 GPUs.

Seems ok at 2 GPUs though (480x480) so maybe an edge case with 4.

A bigger issue is it seems like when the static noise comes back it's after the 4th generation in a row. This is for both 2 GPUs and 4 GPUs. The video height/width seem the same across all videos. Is there some kind of clearing that needs to happen to reset the RAM or something? This is by doing a loop around the inference call like this:

    for idx in range(40): 
            outputs = hunyuan_video_sampler.predict(
            prompt=args.prompt, 
            height=args.video_size[0],
            width=args.video_size[1],
            video_length=args.video_length,
            seed=args.seed,
            negative_prompt=args.neg_prompt,
            infer_steps=args.infer_steps,
            guidance_scale=args.cfg_scale,
            num_videos_per_prompt=args.num_videos,
            flow_shift=args.flow_shift,
            batch_size=args.batch_size,
            embedded_guidance_scale=args.embedded_cfg_scale,
            i2v_mode=args.i2v_mode,
            i2v_resolution=args.i2v_resolution,
            i2v_image_path=args.i2v_image_path,
            i2v_condition_type=args.i2v_condition_type,
            i2v_stability=args.i2v_stability,
            ulysses_degree=args.ulysses_degree,
            ring_degree=args.ring_degree,
            xdit_adaptive_size=args.xdit_adaptive_size
        )
        samples = outputs['samples']

The PR #57 solves this edge case. For the second "bigger issue", I think it's because the args.i2v_image_path is the same across all rounds of the loop.