`map` with any joblib backend

[lhoestq]

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for map yet.

Right now we're using our iflatmap_unordered implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have iflatmap_unordered for joblib and therefore a map with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).

[ArjunJagdale]

#self-assign

Hi @lhoestq 👋🏼

I’d like to work on this!

Planning to support progress tracking with map() using any joblib backend (like "loky") by replacing the Queue-based approach in iflatmap_unordered with a file-based progress tracking mechanism (e.g. shared temp file with periodic updates).

This would allow the progress bar to work even in backends where inter-process Queues aren't supported. Let me know if this sounds good — I’ll get started!

[lhoestq]

I think ideally it should have a general solution, since some joblib backends don't have a shared filesystem