nyc issues and workarounds

[davidlehn]

I'm not sure what the maintenance status is of esm, but for future readers, here's some notes on nyc related issues and workarounds with nyc v15, esm v3.2.25, and node v14.

Problem

When esm is in use, running regular tests creates node_modules/.cache/esm, then running nyc apparently tries to reuse those files and the coverage output is wrong, usually reported at always 100%, but is not really 100% and the coverage output is nonsense in some way (incorrect source lines or other issues).

Something simple like this fails:

# npm test
# nyc npm test

Running the reverse way works since instrumented nyc code gets cached, and regular tests can run it.

ESM_DISABLE_CACHE

One might think you could avoid cache issues with:

# ESM_DISABLE_CACHE=true nyc npm test

However, that env var is only checked in an onExit() handler and cleans up the cache after tests are run. This is confusing, and a second coverage run would work fine since the cache got deleted at the end of the last run.

• https://github.com/standard-things/esm/blob/master/src/caching-compiler.js

The docs are not clear on this, or it's misbehaving.

It really seems like this var should disable caching just like the other cache option does. I'm not sure what the reasoning is here.

It is possible to run a pre-command just to clear the caches. Something that sets the env var then loads everything through esm. But this is an odd workaround to need.

ESM_OPTIONS

Two approaches that seem to work use ESM_OPTIONS. (With some escaped double quotes needed when in package.json.)

Setup a special coverage cache dir. It seems useful to put the cache next the regular esm cache dir. Regular tests and coverage tests will just use different cache dirs.

# ESM_OPTIONS='{cache:"node_modules/.cache/esm-nyc"}' nyc npm test

A less verbose version can just disable the cache if your use case doesn't really need to worry about speed for coverage testing:

# ESM_OPTIONS='{cache:false}' nyc npm test

isNyc check

• nyc v15 removed the NYC_ROOT_ID environment variable that is used in the isNyc() check in is-nyc.js
  • CHANGELOG
  • chore: remove vestigial environment variables istanbuljs/nyc#1078
  • nyc 15.x / next version of istanbul libraries istanbuljs/nyc#1104
• In the use cases I have, this didn't seem to matter, even though isNyc() is always false now.
• A fix may be to use NYC_PROCESS_ID?

Suggestions

• If isNyc() is still something that is needed, it should be fixed.
• ESM_DISABLE_CACHE=true should behave like ESM_OPTIONS='{cache:false}'.
• Ideally this could all be avoided if when isNyc() === true a different cache dir name is used like esm-nyc rather than esm.