support multiple ranks and alphas for LoRA

[pacman100]

What does this PR do?

1. Add support for different ranks and alphas for LoRA tuner.
2. Add support for multiple active adapters

To Do:

• Tests
• Documentation
• Example

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[BenjaminBossan]

LGTM, thanks for adding this so quickly.

There are a few things I'd like to see in relation to this PR, but given that it helps with the diffusers PR, I'm fine with adding those later to speed things up:

• unit tests
• an example in the docs or examples folder
• similar config options for other tuners where it makes sense, like AdaLoRA
• maybe: regex matching

[kovalexal]

Hi, @pacman100! This feature is a super useful addition!

I am currently working on proper support of LoRA-C3Lier for Stable Diffusion. It's a special type of LoRA for which the Conv2d 3x3 layers may have different ranks and alphas. So this code will be super-useful for me to finish implementing this type of LoRA.

Do you by chance know what is the ETA for this feature to get into main branch?

[kovalexal]

@pacman100, @BenjaminBossan sorry to disturb you, I've been trying out the code from this branch, and I've faced and issue which is related to the usage of LoRAs with provided rank_pattern and alpha_pattern.

In short - if you need to create 2 adapters for model - first one without provided rank_pattern and alpha_pattern, second one with rank_pattern and alpha_pattern provided. After adding second adapter you can see, that ranks (and also alphas) for custom layers are not taken into account.

Here is a demo notebook which demonstrates how to reproduce this issue.

As far as I see, there is a possible problem in this specific lines:

peft/src/peft/tuners/lora/model.py

Lines 164 to 166 in 95fa3c1

	current_key = optional_kwargs["current_key"]
	r = lora_config.rank_pattern.get(current_key, lora_config.r)
	alpha = lora_config.alpha_pattern.get(current_key, lora_config.lora_alpha)

When the first adapter is initialized - the name of the module is passed as usual (e.g. "path.to.module"), but when the second adapter is added - the name of the module is nested in peft prefix (e.g. "base_model.model.path.to.module"), so that's why custom alphas and ranks are not propagated properly.

[kovalexal]

@pacman100 @BenjaminBossan I've addressed this issue and worked on 3 out of 4 comments @BenjaminBossan left (all except AdaLoRA).

Please let me know if there a way for me to somehow add my modifications to this PR.

[pacman100]

Hello @kovalexal, Thank you for actively working on this! Please raise a PR to merge your changes in this PR, we will be happy to quickly review it and incorporate the changes.

[pacman100]

Do you by chance know what is the ETA for this feature to get into main branch?

Next week is the plan

[younesbelkada]

Thanks a lot Sourab! I left few minor comments as a first pass ! Let me know what do you think

[BenjaminBossan]

Great work, this is a huge addition to PEFT.

I left a couple of comments, please take a look if they make sense. I have mostly reviewed the lora code closely, as I think IA³ and AdaLora just follow the same logic, so some of my comments apply there as well.

[BenjaminBossan]

Overall, this LGTM, thanks for addressing the issues. I think there is one bug, however, please check my comment.

It is important that we also add tests for multiple active adapters, but if this is time critical, I'm okay with adding them later. Docs/examples can also be added later.

[BenjaminBossan]

LGTM, thanks for addressing the comments.

Let's not forget to add tests + docs + examples later.

I think using multiple adapters with dropout could be problematic, as more and more values will be set to 0. But I guess that's on the user to address, not sure if there is anything that we could do about it.

[pacman100]

I think using multiple adapters with dropout could be problematic, as more and more values will be set to 0. But I guess that's on the user to address, not sure if there is anything that we could do about it.

I believe the support for multiple active adapters should be used only during inference in eval mode. In that case, dropout won't be an issue.

[BenjaminBossan]

I believe the support for multiple active adapters should be used only during inference in eval mode. In that case, dropout won't be an issue.

I see. But in theory, it would also work during training, right? And there isn't anything in the code right now that would prevent that.

[younesbelkada]

Great work @pacman100 , as always! Thanks!

[whybeyoung]

Any clearly examples for this pr?

[mcc311]

Could anyone provide any example here? Thanks!

[BenjaminBossan]

Sorry that we haven't added any documentation yet. The idea is that you can do something like this:

config = LoraConfig(
    r=8,
    rank_pattern={"foo": 16},
    lora_alpha=10,
    alpha_pattern={"bar": 20, "baz": 30},
)

In this case, all LoRA layers would use rank 8, except for layers that match "foo", which would have rank 16. Similarly, LoRA alpha would be 10 by default except for "bar", which would be 20, and "baz", which would be 30.