🚨🚨🚨 [Refactor] Move third-party related utility files into integrations/ folder 🚨🚨🚨

[younesbelkada]

What does this PR do?

As per title and to address #25077 (comment)
Let's move all third party libs (outside HF ecosystem) related utility files inside lib_integrations/, currently to the best of my knowledge bitsandbytes and deepspeed are the only 2 third party libs that we use as a plugin integration and is outside HF ecosystem

cc @sgugger and @stas00 as it touches DS related code

Let me see first if the CI passes

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[stas00]

Hi Younes,

Unless I'm mistaken these would be BC breaking changes.

$ PYTHONPATH=src python -c "from transformers.deepspeed import HfDeepSpeedConfig"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers.deepspeed'

Changing the doc to use the new API won't unbreak users' code.

(In general for deepspeed integration issues please tag @pacman100 who is the current maintainer of the integration.)

[stas00]

unrelated - perhaps there could be a neater name than lib_integrations - this is awkward IMHO.

Perhaps just integrations or integration or external - in other words one word would be smoother. it's obvious that those are libs. all python packages are.

- from transformers.lib_integrations.deepspeed import HfDeepSpeedConfig
+ from transformers.integration.deepspeed import HfDeepSpeedConfig

[sgugger]

Yes I had mentioned integrations as well as a name for the folder. Note that we usually do not guarantee backward compatibility with imports not at the init level (anything not imported in the main init or a subfolder init is considered private), but we can keep a deepspeed module that reimports HfDeepSpeedConfig if this line is in a lot of places.

[younesbelkada]

I did not used integrations as there is already a module named integrations.py in the same place. Maybe renaming lib_integrations to external would be better, WDYT?

[stas00]

Use a single integration? Since once you add the specifics it's a clear transformers.integration.what

as in transformers.integration.deepspeed aka transformers' integration of deepspeed

[sgugger]

As long as the objects are re-imported, it's completely fine (and non-breaking) to change integrations.py to an integrations folder.

[sgugger]

Not sur we need subfolders for DeepSpeed, bitandbytes and peft, they can jus be modules no?

[younesbelkada]

Ah yes makes sense, modified it accordingly

[younesbelkada]

Actually sorry, forgot to do it for peft, will do it now

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[pacman100]

Thank you @younesbelkada for iterating!

[younesbelkada]

Thanks everyone for your reviews, regarding the large diff on the doctest file my intuition is that it got large because I added a new line that changed the alphabetical order of the file but I am not sure. (maybe @ydshieh can confirm)
I also realised we might need to keep utils/bitsandbytes.py otherwise it would be too breaking, so I reverted it back and did a similar approach than deepspeed.py

[younesbelkada]

The large diff is fixed with #25680 thanks to @ydshieh

[younesbelkada]

@sgugger , ran the daily CI (with a reduced number of models tests) thanks to @ydshieh and you can see the results here: https://github.com/huggingface/transformers/actions/runs/5965296109
compared the report with the report of the same day and I don't see any surprising difference / test failure caused by this PR. Therefore I think we can merge it ! Can you please have a final look ? 🙏 Thanks!

[sgugger]

Thanks for all the work on this!