[Cross-entropy-loss] return mean token accuracy metric with CE loss

[kashif]

Summary

Returns the mean token accuracy metric when minimizing the cross-entropy loss without materializing the logits

https://x.com/jeremyphoward/status/1703246293802586155

Testing Done

• Hardware Type:
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[kashif]

@vaibhavjindal would you be able to kindly review?

[kashif]

@shimizust this will be a breaking change i believe BTW

[vaibhavjindal]

@kashif could you please elaborate on how it will be a breaking change? Will it break the intergration with transformers or trl?

[kashif]

yes if someone is using the raw functions in their lib. then now that functions returns one more thing... but on the HF side this PR takes care of this

[kashif]

@vaibhavjindal see here https://github.com/linkedin/Liger-Kernel/pull/910/files#diff-7654a885e261ec4986c229d953681635ab4c8761c89342d2dde604f169783b35L350

[vaibhavjindal]

@kashif got it. So if i understand correctly, it will make sure that liger remains compatible with newer versions from HF. However, just want to confirm it will break liger support with older transformers/trl versions?

[kashif]

no i believe my changes here will work with older version of HF.. i just meant non-HF frameworks

[kashif]

TRL relies on HF integration for the CE loss so in TRL I will just pin to the liger version that has these changes

[kashif]

@vaibhavjindal let me fix up the new qwen3-vl model to update its API

[kashif]

@vaibhavjindal all good from my side

[vaibhavjindal]

@vaibhavjindal all good from my side

Thanks a lot! I will do some final checks on correctness and benchmarks and will try to get it merged soon.

[kashif]

thank you so much.. also see here: huggingface/trl#4302 (comment)

[kashif]

thanks @vaibhavjindal for the typo fix and making it more robust!