Add property based testing based on fast-check in Jest

[dubzzz]

🚀 Feature Proposal

Property based testing can be seen as some kind of fuzzing approach. It makes users able to cover a wider range of inputs with a single test.

Motivation

fast-check proved very useful on Jest and was behind the following issues:

• toEqual not symmetric for Set #7975
• toStrictEqual fails to distinguish 0 from 5e-324 #7941
• toStrictEqual does not consider arrays with objects having undefined values correctly #7937 (for this one, the tests of fast-check failed because of the regression so I started to dig a bit and run property based on expect of Jest to see if I would be able to find other bugs like this one)

It is currently being added in the test suites of Jest itself: #8012

Example

I think the integration of fast-check within Jest could be something like the one I did for ava: https://github.com/dubzzz/ava-fast-check/ but I am very opened to discussions on this point.

My current wrapper works as follow:

• It provides testProp - and certainly itProp for Jest
• Its signature is: testProp(testName, generators, property)
• What's great with this wrapper is that it adds the seed next to the test name so that the user can have access to the seed even in case the test goes into an infinite loop (if and only if Jest can stop it in this case).

// for all a, b, c strings
// b is a substring of a + b + c
testProp('should detect the substring', [fc.string(), fc.string(), fc.string()], (a, b, c) => {
  return (a + b + c).includes(b);
});

or maybe:

// for all a, b, c strings
// b is a substring of a + b + c
test.prop('should detect the substring', [fc.string(), fc.string(), fc.string()], (a, b, c) => {
  return (a + b + c).includes(b);
});

Pitch

Jest is a very successful testing framework.
Adding property based testing directly within Jest would give the community a new way to check their code. It might help the JavaScript community to detect new bugs they even not though about before. fast-check has already been useful in many projects, see https://github.com/dubzzz/fast-check/blob/master/documentation/IssuesDiscovered.md

[dubzzz]

Something useful for the integration: fast-check follows semantic versioning.

Patch version should not:

• modify the values that would be generated by an arbitrary for a given seed - Except maybe if there is a crash or infinite loop in the generation part
• impact the shrinking tree
• impact replay - the same seed and path would lead to the same value - direct consequence of the two points above
• add new features

A minor should not:

• break existing APIs
• remove depreciated APIs

[dubzzz]

I am obviously a little biased concerning the choice of the framework as I wrote fast-check. In a nutshell when I first wanted to do property based in my company I opted for jsverify but got stuck on some of its limitations (due to initial design choices). So I started to explore how it worked and somehow started fast-check.

Why fast-check?
Maintained, used by awslabs/aws-cdk for instance, multiple unique features: replay, verbose, commands for UI tests, pre-conditions...
More on the read me.

See #7938 (comment)

[jeysal]

Having recently discovered #8032, just wanted to note that if we integrate something like this, we should definitely do it in a way that does not impact performance for users who do not use this feature (i.e. not unconditionally require('fast-check') from jest-circus or something).
In general I think it would be cleaner to have this in a standalone package, along the lines of ava-fast-check. @dubzzz do you think there would be any technical difficulties with doing that or other reasons why integrating more tightly with Jest would be beneficial?

[spontoreau]

Great idea!

[dubzzz]

Initially, I was not planning to add fast-check as an internal feature of Jest. But when @SimenB suggested it I really thought about this idea: I personally do think it would be a great thing for testing in JavaScript.

Indeed Jest has always proved ahead of time concerning test methodologies. For instance, when it came with snapshot tests, many developers tried and adopted it.

In a way, fast-check would be able to live alone without being directly integrated as a dependency of Jest. Doing a jest-fast-check would not be a big deal on my side and I can do it without any problems.

But I believe that pushing the property based testing through Jest, as one of its features, would highly increase the usage of the technic among dev communities. The more people will use such methodologies the more they will be confident about their code.

[jeysal]

I'm not totally against adding something like this and I'd like to hear what others think about this. Just wanted to note that if there's no technical reason why it should be in Jest core, we could also consider promoting it in other ways (documentation, jest-community, etc.).

[dubzzz]

Totally agree, there is no technical reason to move it inside Jest core. Promoting the approach through documentation or Jest community might also be great. I let you see what's the best option ;)

[quasicomputational]

Potentially, Jest could tell the fuzzer to do more/less work depending on how much other work there is to do, whether it's watch mode or one-shot, and probably other considerations. But that would be a nice-to-have and probably need a deeper integration than is being considered.

[dubzzz]

Jest could tell the fuzzer to do more/less work [...] watch mode or one-shot

I really like the idea suggested by @quasicomputational. Indeed by default, property based tends to take longer than classical units because it runs 100 tests for each property.

If fast-check was integrated as a dependency of Jest, such feature would be easy to build. fast-check already accepts options telling it to do only N runs (instead of 100 which is the default).

Syntax: fc.assert(/* your property */, { numRuns: 10, /* other settings */ })

If not, I don't know if there is a simple and long-term way to know whether the test is running in watch mode.

[dubzzz]

Another feature that would be more difficult to integrate if fast-check is not directly part of Jest is the retryTimes - more

By default, what you would want to retry in the case of property based is not the test itself but the execution of one run of the property.

[dubzzz]

@SimenB Any opinions on this RFC?

[quasicomputational]

Another benefit to having Jest know about the property tests: being able to isolate iterations better, including resetting mocks, timers and modules. This is particularly important to get right for shrinking, because if you've managed to contaminate the environment so that the property always fails, it'll be shrunk away to nothing and you get a useless report.