Limit the number of open fds in recursive directory traversal to a defined value

[adithyaov]

A lot of open FDs cause the flow to fail with:

       uncaught exception: IOException of type ResourceExhausted
       benchmark-tmp/dir-structure/dir_4/dir_3/dir_2/dir_5/dir_5: openDirStream: resource exhausted (Too many open files)

This is true in case of BFS.

[freckletonj]

Relatedly, I'm also wondering if we can globally limit maxThreads?

I can limit threads locally per map. I can manually collapse and compose my business logic, eg print . inc . inc and map that, but that requires devs to optimize according to execution concerns, and I like how streamly often obviates the need to think about execution concerns.

I'm thinking something like the following could work?

f :: IO ()
f = do
  fromList [1,2,3]
  & parMapM (maxThreads 2) (pure . (+1))
  & parMapM (maxThreads 2) (pure . (+1))
  & parMapM (maxThreads 2) print
  & fold Fold.drain


fWithGlobalMaxThreads :: IO ()
fWithGlobalMaxThreads = do
  withConfig (maxThreads 2) $ \cfg ->
    fromList [1,2,3]
    & parMapM cfg (pure . (+1))
    & parMapM cfg (pure . (+1))
    & parMapM cfg print
    & fold Fold.drain



fWithGlobalMaxThreads2 :: IO ()
fWithGlobalMaxThreads2 = do
    fromList [1,2,3]
    & parMapM id (pure . (+1))
    & parMapM id (pure . (+1))
    & parMapM id print
    & fold (maxThreads 2) Fold.drain