Qt Make/Ninja interaction

[bartoldeman]

When building Qt5 5.12.8 I observed that the QtWebEngine part was very slow. It uses Ninja inside Make and for some reason Ninja only spawned one g++ process at a time. Now I did use something non-standard, namely using a Gentoo Prefix Ninja, which is a fork with support for the GNU Make jobserver (https://github.com/Kitware/ninja), and that was what caused this interaction.

However when putting ninja 1.10.0 as a builddependency I see this:

Checking for system ninja... no

and it then proceeds to build and use an included Ninja in (qtwebengine/src/3rdparty/ninja/ninja), which is Ninja version 1.8.2 in this case.
From a pure EB perspective I see two issues here:

1. The Qt make procedure has this check (also in 5.14.1):

        contains(version, "1.[7-9].*"): return(true)
        qtLog("Ninja version too old")

Ninja 1.10 fails this test and that's why it uses the included Ninja instead of the builddep.

2. Ninja by default (ie. not the one with GNU Make jobserver support) will gobble up all cores. If you want to avoid that you need to pass it an explicit -jN flag, via prebuildopts = "NINJAFLAGS='-j8'" (see https://patchwork.ozlabs.org/project/buildroot/patch/20180629143521.9018-1-gael.portay@savoirfairelinux.com/ )

[geimer]

Regarding issue 1: Qt 5.15.0 uses the following check:

        contains(version, "1\.([7-9]|1[0-9])\..*"): return(true)
        qtLog("Ninja version too old")

So a simple patch should be sufficient here.

Regarding issue 2: Couldn't we use something like prebuildopts = "NINJAFLAGS='-j%(parallel)s'" in the easyconfig file? Or is the Ninja build overlapping with some other compilations spawned by make such that the jobserver approach would be required to effectively limit parallelism?

[jhein32]

Two days ago I was building QT5 for 2019b on our Sandybridge cluster. The build node has 32GB and it failed about 1h in, with an error “terminated”. We normally build with 16 cores for that. Telling EB to give it only 5 cores, made it fail about 2h in, again with error “terminated”. I took a different node with 96GB and it build. So did it month ago on our Haswell cluster, where the build node has 64GB. I had been looking at the build using “htop”. I had the impression during some parts of the build process more than the EB allowed 16 threads were active. Link step? Some testing?

To me the signature is consistent with "out of memory". I noticed (htop) threads can have around a GB of RSS. If you have more than 16 running, 32GB is eaten very quickly.

This might or might not be another manifestation of the issue described here.

[boegel]

Small update on this: just setting the $NINJAFLAGS environment variable to something like -j10 is enough, then you see ninja -j10 popping up in the installation log (without setting $NINJAFLAGS, so just see ninja -v, which implies it uses all the cores it can see).