Running suites in parallel

[benhyland]

I have a few questions about test parallelisation which I don't think are answered by the docs.

As I understand it, ginkgo -r -p will run specs within each suite in parallel and runs the suites serially.

What is a good way to use ginkgo to parallelise across suites as well as within suites?

In the end, I might want to control the scheduling of specs based on their expected run time, to reduce the overall run time of the collection of suites. How might this be achieved? Does it fit into the discussion of custom test runners at #146 ?

Many thanks!

[onsi]

you understand it right. There isn't a way from the ginkgo cli to parallelize across suites and this wouldn't be solved by a custom test runner either. Each suite is its own test binary and ginkgo launches these serially

-- too much detail -->

In fact, when it runs a suite in parallel Ginkgo takes the compiled binary and launches it N times passing it flags to inform it of its node index. The binaries run independently and only run the appropriate subset of the tests (things like randomization and test focus are computed consistently across the nodes).

These individual nodes communicate back to the ginkgo CLI which acts as a server that stitches the test output together to make the parallel runs look like one seamless run.

<-- /too much detail --

To run multiple suites at once you could do some hokey orchestration around launching multiple ginkgo commands. I wouldn't recommend doing this in combination with ginkgo -p as you will just flood the CPU.

I could imagine modifying the ginkgo CLI to have more parallelization modes (e.g. run N suites at once vs shard each suite across N nodes vs some sort of mixed mode). I'm not sure how high value that would be TBH.

[benhyland]

Thanks for the confirmation, and for the extra detail :)

The value I was hoping to gain was a way to split long-running acceptance tests that require a significant external environment across several test-running agents so as to be able to target several pre-deployed environments in parallel.
I thought a custom test runner might be useful for this, and might also have been used to combine each test definition with historical data predicting its run time, which would allow more intelligent test scheduling to further reduce the overall run time.

[pohly]

I just stumbled over this. go test will run tests from different packages in parallel by default when invoked for multiple packages. I was expecting the same from ginkgo, but as explained here that doesn't work. I'm now stuck between a rock and a hard place: either run tests from multiple packages in parallel, or run specs inside a Ginkgo suite in parallel. It would be really nice if ginkgo could support both.

Regarding number of CPUs: "make -j" has solved this by connecting all processes to a shared pipe and then handing out tokens, one per allowed CPU. See http://make.mad-scientist.net/papers/jobserver-implementation/ for details.

See also #111.

[onsi]

I'm working through the backlog of old Ginkgo issues - apologies as this issue is probably stale now.

I'm investigating enabling concurrent suites in Ginkgo v2.

I'm going to close this for now, but feel free to reopen it if you'd like.

[josesa]

@onsi Was this considered for v2 ? I can't seem to find any reference to that in the migration documentation.