Would SGD w/ momentum or NAG be a good fit for the gradient masking?

[kaimatzu]

Just wondering if it's a good idea in theory to try this with SGD since theoretically, it should behave better than first+second moment optimization. Thoughts?

[kyleliang919]

I think it will improve upon SGD with momentum baseline when the gradient is noisy. But I have never tried.
I have been trying to understand the first + second moment optimization, especially something like muon and LION, but I still have not gotten a good answer to why it goes faster beyond intuitions, usually the best analysis can say is that it converges...