Failed creating session: EMFILE, open 'some/file/path.js' (in MacOS)

[gregtillbrook]

This error had me scratching my head until we figured out it wasn't directly a buster issue. Im listing it here as there's a good chance it'll affect someone else when using buster.

The error: Buster execution fails with the error Failed creating session: EMFILE, open 'some/file/path.js' (in MacOS)

The fix: The error relates to the number of files that can be open simultaneously. Up this limit to fix by typing the following in the terminal;

ulimit -n 10000

[gregtillbrook]

Note also that the ulimit change lasts only for the current session. This stack overflow article mentions a more permanent solution;

http://superuser.com/questions/261023/how-to-change-default-ulimit-values-in-mac-os-x-10-6

[c089]

Why close the issue? That's a workaround, not a fix - I guess at some point buster reads too many files in parallel which could be fixed by using a queue...

[gregtillbrook]

I closed the issue as I viewed it as more of an OS problem that just happened to manifest i buster.

That said though we've had lots of trouble since then with stalling tests (as raised in #299 #235 #303) which also seems to be related to large file numbers so maybe this points to a wider common issue.

So yes, i'll reopen.

[GCheung55]

I'm also experiencing this issue. I'm working with buster-amd and the documentation states that all the sources needs to be loaded before they can be used, which seems to be the issue.

[augustl]

We should check ulimit and read resource sets from file system and into memory in chunks. We currently do a when.all over all the resources in the resource set. This causes a parallel file system read for each resource to happen concurrently.

https://github.com/busterjs/ramp-resources/blob/1bef274083e96c0c2ff868b4815d6cb9a1856759/lib/resource-set.js#L299

We can chunk this operation on "ulimit -n". Not sure how to get that other than shelling out, and not sure how to do that on Windows.

Perhaps we should just always chunk by 100 files or something like that. This will cause the initial test run to be a little slower than it has to (?) but subsequent test runs will be faster due to the etag checking + caching we do - we'll only read the files that has changed since the last test run.

[alexindigo]

Had you consider using this module https://www.npmjs.org/package/graceful-fs?

And in my case it dies silently during manifest build (-vv flag is on) and I had to inject bunch of console.logs to find the problem. And seems like it tries to open same files multiply times.
My ulimit is 256 (default osx) and generated manifest (it works on linux) contains four identical array of 219 files each.

Oh, and funny thing it dies with code 0, so broken test suite stays unnoticed. :)

[alexindigo]

For those who suffer similar problem, while it's being decided on how to approach the issue,
here is workaround – https://gist.github.com/alexindigo/e67ac202d9e385600f88
Allows to use buster-test with graceful-fs without touching buster's source, but rather stubbing it's submodules, for greater flexibility.

Although it was tested only with 0.7.13 version of buster, it should continue to work other versions, survive npm updates and withstand shrinkwrapping.

[augustl]

Resolved in busterjs/ramp-resources@b2743e8