Add HfFileSystemStreamFile

[lhoestq]

This allows faster file streaming. This is useful for streaming WebDatasets for example

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[codecov]

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (b79882a) 82.55% compared to head (acca803) 82.20%.
Report is 18 commits behind head on main.

❗ Current head acca803 differs from pull request most recent head 1c47722. Consider uploading reports for the commit 1c47722 to get more accurate results

Files	Patch %	Lines
src/huggingface_hub/hf_file_system.py	24.39%	31 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1967      +/-   ##
==========================================
- Coverage   82.55%   82.20%   -0.36%     
==========================================
  Files          66       66              
  Lines        8159     8187      +28     
==========================================
- Hits         6736     6730       -6     
- Misses       1423     1457      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Wauplin]

Thanks for opening the PR @lhoestq. Left some comments but mostly to be sure was is a implementation decision and what is required by fsspec. In general when there is a doubt I'd prefer to have an explicit comment in the code for the future ourselves (I feel fsspec is quite opinionated on some things).

Also, would it be possible to update the Usage part of this guide to showcase how to use HfFileSystemStreamFile and why.

Thanks in advance!

[mariosasko]

A comment from me :)

[lhoestq]

added annotations + a retry when streaming + close connection :)

[mariosasko]

Thanks! One nit.

EDIT:

Maybe let's also add a test.

[lhoestq]

Adding a test was useful 🙃 I found two bugs and fixed them

[Wauplin]

Thanks for iterating on this PR @lhoestq! Looks good to me except for a failing test. Can you have a look at it please? (or remove the new line if not relevant). Thanks in advance!

[Wauplin]

Thanks for fixing the test @lhoestq. Failing tests are now unrelated. Let's merge this :)