Support fsspec 2023.10.0

[albertvillanova]

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:

• Temporarily pin fsspec < 2023.10.0 #6331

Related to issue:

• Latest fsspec==2023.10.0 issue with streaming datasets #6330

As @ZachNagengast suggested, the issue might be related to:

• Implementation of 'local://' URI Scheme Support fsspec/filesystem_spec#1381

[tomscholz]

Hi @albertvillanova @lhoestq

I believe the pull request that pins the fsspec version (#6331) was merged by mistake. Another fix for the issue was merged on the same day an hour apart. See #6334

I'm now having an issue in my project where I can't use newer versions of fsspec.

Can we remove the pin?

Have a nice day! :)

[albertvillanova]

Hi @tomscholz,

Thanks for pointing this out. I think you are right.

I am doing some cross-checks and fixing it.

[albertvillanova]

Hi again, @tomscholz.

After a more cautious investigation, I think the pin is OK because there are other reasons for it. Chronologically:

• Temporarily pin fsspec < 2023.10.0 #6331
• datasets.filesystems: fix is_remote_filesystems #6334
• unpin-fsspec #6336
• Pin supported upper version of fsspec #6337

The reason is that after version 2023.10.0, they changed again the behavior of their glob function. See: #6337 (comment)
We are working on our side to support both previous and new glob behavior.

Note:

• First pin was < 2023.10.0
• Last pin is <= 2023.10.0

[albertvillanova]

Fixed by #6334 and #6336.