training stability

[blvosh2]

Hi, I am asking as one that tries to train SD3, how did you manage to stabilize the training ?.
for me after few hours of training the loss goes up and I get nan's

[AvalEng]

i also have wrote a code for training the model but after hours of training i got nothing !!!
@blvosh2

[blvosh2]

So mine start to converge but after a few hours there is sudden divergence.
It looks like adding the qk_norm as the paper says helps but i can see that here they didn't added it and still manage to train the network, i wonder how they did it.

[AvalEng]

Alright, can we schedule a meeting to discuss our approaches? This would be very useful @blvosh2

[AnurajG]

@AvalEng @blvosh2 Did you guys get your training working. My training process fails to stabilise in the end. After hours of convergence it just keeps on diverging towards the end. Any help in this regard will be extremely great and helpful for my work.

[blvosh2]

Hi @AvalEng sorry for the kong response about the meeting, i am pretty busy, but we can talk on other platforms.
@AnurajG the only thing that worked for me divergence wise is to insert the qk_norm and the training is now stable although its very weird because the FitDit weights doesn’t include the norm so i wonder how they solved it.
Also i am having troubles with the fine preservation, its seems is very hard for the vton model to take the text from garment.
I haven’t implemented the Fourier loss yet, maybe this is the one to blame ?

[AnurajG]

Hi @AvalEng sorry for the kong response about the meeting, i am pretty busy, but we can talk on other platforms. @AnurajG the only thing that worked for me divergence wise is to insert the qk_norm and the training is now stable although its very weird because the FitDit weights doesn’t include the norm so i wonder how they solved it. Also i am having troubles with the fine preservation, its seems is very hard for the vton model to take the text from garment. I haven’t implemented the Fourier loss yet, maybe this is the one to blame ?

Which dataset have you been training on. Your custom dataset or the one mentioned in the repo including CVDD?

[blvosh2]

@AnurajG i have been testing my own custom dataset

[AnurajG]

@AnurajG i have been testing my own custom dataset

@blvosh2 Would it be possible to show your training code (over email or on a fork?)

It'll be great help for my work. Thanks

[Holmes2002]

@AnurajG @blvosh2 @AvalEng Does anyone success in training ? And can u share code ?

[Holmes2002]

@AnurajG @blvosh2 @AvalEng Hi, I retrain stage 1 in FitDit model but seems that loss hasn't changed in training process. Any solution

[AllanGitGood]

Hey, was trying to train the script based on the given inference code. Training on a single GPU seems to be working so far, but I’m not sure how many epochs the original script was trained for. Could someone let me know?

Also, I ran into some errors when trying to do distributed data parallel (DDP) training. If anyone has tips or knows common issues with DDP in this setup, that’d be awesome.

Thanks!

[Holmes2002]

@AnurajG @blvosh2 @AllanGitGood @AvalEng Any news for training code ?
I have reimplemented Fitdit with lora training. However I didn't see the stability between Noise and Frequency loss and I still struggled with it. I appreciate any advice or help