Add New lr scheduler

[sdbds]

Change Lr schedulers library from diffusers to transformers, they use same library but transformers had more lr schedulers.
Considering the community has always wanted custom learning rate, I think these newly added regulators will meet the demand.

Add inverse sqrt learning rate scheduler

new argument lr_scheduler_timescale,default to warms_up_steps

Add cosine with min lr scheduler
When set the num_steps=100, num_warmup_steps=10, lr=0.2, min_lr=0.01. The learning rate looks like:

new argument lr_scheduler_min_lr_ratio,default to 0 as cosine lr scheduler.

Add WSD scheduler
The ladder scheduler that so many people have been wanting.

new argument lr_decay_steps,
new argument lr_scheduler_min_lr_ratio,default to 0.

need update requirement transformers==4.41.2

[sdbds]

Fix bugs now, i think it works well.
@kohya-ss

[kohya-ss]

Thank you! However, removing PIECEWISE_CONSTANT will likely impact users. I would like to update the code after merging so that it can be used, but it will take some time. I'd like to prioritize some other PRs. If you could fix this, that would be great.

[sdbds]

Thank you! However, removing PIECEWISE_CONSTANT will likely impact users. I would like to update the code after merging so that it can be used, but it will take some time. I'd like to prioritize some other PRs. If you could fix this, that would be great.

OK，just keep diffusers import for backup using, use

name = SchedulerType(name) or DiffusersSchedulerType(name)
schedule_func = TYPE_TO_SCHEDULER_FUNCTION[name] or DIFFUSERS_TYPE_TO_SCHEDULER_FUNCTION[name]

to judge if using PIECEWISE_CONSTANT.
And i add parser type for input float with warmup and decay ratio, don't need to calculate training total steps.

[kohya-ss]

Sorry for bothering you again. Is there a reason to upgrade the version of the library other than transformers? It requires some more comprehensive testing, which takes time.

[sdbds]

Sorry for bothering you again. Is there a reason to upgrade the version of the library other than transformers? It requires some more comprehensive testing, which takes time.

The main thing is to upgrade the transformers version, and after upgrading the transformers version the other two dependencies will also ask for an update, so it's three version updates.
Since the PR is older, I reviewed the latest three dependency update histories, and there don't seem to be any major bug fixes as well as disruptive changes.

[kohya-ss]

Thanks, I understand. Then it seems like there is no big problem. I've updated accelerate and transformers in the sd3 branch, so maybe I can match that. I'll do some checks and merge :)

[kohya-ss]

Sorry for the delay, I have merged this.