Fix lora tag check and add device maps and precision for diffusion pipelines #1829

ariG23498 · 2025-11-03T12:34:32Z

Making lora tag check case insensitive.

packages/tasks/src/model-libraries-snippets.ts

ariG23498 · 2025-11-03T12:53:07Z

packages/tasks/src/model-libraries-snippets.ts

 	} else if (model.tags.includes("controlnet")) {
 		codeSnippets = diffusers_controlnet(model);
-	} else if (model.tags.includes("lora")) {
+	} else if (model.tags.some(t => t.toLowerCase() === "lora")) {


I am unsure if this is the right way to tackle this.

My rationale was to lowercase the tags and check against "lora".

Suggested change

} else if (model.tags.some(t => t.toLowerCase() === "lora")) {

} else if (model.tags.includes("lora")) {

Hey @ariG23498 , tags are considered case sensitive in the HF ecosystem and we usually prefer to not lowercase them when checking for a value. So the best here would be to keep it has done before and if some repos have LoRA or LORA tag instead, they should be updated directly (i.e. open a pull request on their model card)

Interesting! @linoytsaban this would mean we need your automated scripts to send PRs on such repos.

Thanks for this @Wauplin

pcuenca · 2025-11-03T12:54:32Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image, export_to_video

-pipe = DiffusionPipeline.from_pretrained("${model.id}", torch_dtype=torch.float16)
+pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")


ah, float16 here. Is this intended?

I wanted to keep it as is, and not play around the already existing bits.

I see your point @ariG23498, I do however think this might be worth changing as most current popular image-to-video models are usually loaded in bf16 in diffusers (e.g. wan)

packages/tasks/src/model-libraries-snippets.ts

pcuenca · 2025-11-03T12:56:30Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image

-pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16").to("cuda")
+pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="cuda")


Do we only need the variant in this particular case? Also: flagging the use of float16 as well.

Here again I did not want to change things. I am unsure how that might play. But open to suggestions.

here I think it's ok to stay with float16 as it's only used for SDXL pipelines

packages/tasks/src/model-libraries-snippets.ts

Vaibhavs10

let's move to auto so that people can use this on mps and distributed setups as well.

Vaibhavs10 · 2025-11-03T13:22:59Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image, export_to_video

-pipe = DiffusionPipeline.from_pretrained("${model.id}", torch_dtype=torch.float16)
+pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")

Vaibhavs10 · 2025-11-03T13:42:08Z

packages/tasks/src/model-libraries-snippets.ts

+from diffusers import DiffusionPipeline

-pipe = DiffusionPipeline.from_pretrained("${model.id}")
+pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:42:17Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image

-pipe = DiffusionPipeline.from_pretrained("${model.id}")
+pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:42:26Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image, export_to_video

-pipe = DiffusionPipeline.from_pretrained("${model.id}", torch_dtype=torch.float16)
+pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:42:35Z

packages/tasks/src/model-libraries-snippets.ts

+from diffusers import DiffusionPipeline

-pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}")
+pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:42:52Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import export_to_video

-pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}")
+pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:43:01Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image, export_to_video

-pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}")
+pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:43:07Z

packages/tasks/src/model-libraries-snippets.ts

+from diffusers import DiffusionPipeline

-pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}")
+pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="cuda")

pipe = DiffusionPipeline.from_pretrained("${get_base_diffusers_model(model)}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:43:16Z

packages/tasks/src/model-libraries-snippets.ts

 mask = load_image("https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/cup_mask.png")

-pipe = FluxFillPipeline.from_pretrained("${model.id}", torch_dtype=torch.bfloat16).to("cuda")
+pipe = FluxFillPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")


Suggested change

pipe = FluxFillPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")

pipe = FluxFillPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="auto")

Vaibhavs10 · 2025-11-03T13:43:31Z

packages/tasks/src/model-libraries-snippets.ts

 from diffusers.utils import load_image

-pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16").to("cuda")
+pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="cuda")


Suggested change

pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="cuda")

pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="auto")

sayakpaul · 2025-11-03T13:53:54Z

let's move to auto so that people can use this on mps and distributed setups as well.

Not yet for a DiffusionPipeline since it can contain multiple models underlying. So, strategies known to work for the model-level device_map implementation won't work right out of the box.

For a DiffusionPipeline, "auto", therefore, isn't a valid input type for device_map, yet.

Distributed setups are also impacted by the same reason (a DiffusionPipeline can contain multiple models) and hence, it's a bit more involved than just specifying device_map="auto". Docs for that are here.

Vaibhavs10 · 2025-11-03T14:02:41Z

@sayakpaul - thanks for the context, how do you distinguish between cuda or mps then? users just have to explicitly specify it?

sayakpaul · 2025-11-03T14:38:35Z

how do you distinguish between cuda or mps then? users just have to explicitly specify it?

Either that (i.e., to("cuda"), to("mps"), etc.) or another pattern that is very common, where users just call enable_model_cpu_offload().

ariG23498 · 2025-11-04T10:26:42Z

@sayakpaul @linoytsaban can any one of you approve the PR?

That way we know the code snippets don't have any issue. The only thing left would be to check for the failures in the CI 😓.

linoytsaban

LGTM! thanks @ariG23498

Making lora tag check case insensitive.

Co-authored-by: vb <[email protected]>

Co-authored-by: Lucain <[email protected]>

Vaibhavs10

Good for me to merge - let's put a comment like this for MPS people in the snippets

Vaibhavs10 · 2025-11-05T09:54:06Z

packages/tasks/src/model-libraries-snippets.ts

-	`from diffusers import DiffusionPipeline
+	`import torch
+from diffusers import DiffusionPipeline



Suggested change

# switch to `mps` for apple devices

Also updated `torch_dtype` to `dtype`

ariG23498 requested review from SBrandeis, Wauplin, gary149, julien-c, ngxson and pcuenca as code owners November 3, 2025 12:34

ariG23498 changed the title ~~Fix lora tag check~~ Fix lora tag check and add device maps and precision for diffusion pipelines Nov 3, 2025

Vaibhavs10 reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

Vaibhavs10 reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Show resolved Hide resolved

ariG23498 commented Nov 3, 2025

View reviewed changes

pcuenca reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

pcuenca reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

pcuenca reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

pcuenca reviewed Nov 3, 2025

View reviewed changes

packages/tasks/src/model-libraries-snippets.ts Outdated Show resolved Hide resolved

Vaibhavs10 reviewed Nov 3, 2025

View reviewed changes

linoytsaban approved these changes Nov 4, 2025

View reviewed changes

ariG23498 and others added 8 commits November 4, 2025 16:14

Fix lora tag check

20d6cc4

Making lora tag check case insensitive.

Remove dtype argument

d9f8e20

Update DiffusionPipeline initialization with dtype and device_map

252e9bc

Update packages/tasks/src/model-libraries-snippets.ts

1bb8b79

Co-authored-by: vb <[email protected]>

Change device_map from 'auto' to 'cuda' in DiffusionPipeline

7bde2a9

Update packages/tasks/src/model-libraries-snippets.ts

31283de

Co-authored-by: Lucain <[email protected]>

Update diffusers installation command to include accelerate

eca1350

Change dtype from float16 to bfloat16 in pipeline

08481b3

ariG23498 force-pushed the ariG23498-patch-1 branch from 1265baf to 08481b3 Compare November 4, 2025 10:44

Vaibhavs10 approved these changes Nov 5, 2025

View reviewed changes

ariG23498 added 2 commits November 5, 2025 16:01

Adding comments for mps

c4d505e

Also updated `torch_dtype` to `dtype`

chore: fix comments

30dae3d

ariG23498 merged commit ed62b85 into main Nov 5, 2025
3 of 5 checks passed

ariG23498 deleted the ariG23498-patch-1 branch November 5, 2025 12:08

	} else if (model.tags.some(t => t.toLowerCase() === "lora")) {
	} else if (model.tags.includes("lora")) {

	pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.float16, device_map="cuda")
	pipe = DiffusionPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")

	pipe = FluxFillPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="cuda")
	pipe = FluxFillPipeline.from_pretrained("${model.id}", dtype=torch.bfloat16, device_map="auto")

	pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="cuda")
	pipe = AutoPipelineForInpainting.from_pretrained("${model.id}", torch_dtype=torch.float16, variant="fp16", device_map="auto")

Fix lora tag check and add device maps and precision for diffusion pipelines #1829

Fix lora tag check and add device maps and precision for diffusion pipelines #1829

Uh oh!

Conversation

ariG23498 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linoytsaban Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Vaibhavs10 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Nov 3, 2025

Uh oh!

Vaibhavs10 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Nov 3, 2025

Uh oh!

ariG23498 commented Nov 4, 2025

Uh oh!

linoytsaban left a comment

Choose a reason for hiding this comment

Uh oh!

Vaibhavs10 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

ariG23498 commented Nov 3, 2025 •

edited

Loading

linoytsaban Nov 3, 2025 •

edited

Loading

Vaibhavs10 commented Nov 3, 2025 •

edited

Loading