Skip to content

set $NINJAFLAGS to make sure Ninja doesn't use all visible cores when building Qt5 #2338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akesandgren merged 3 commits into from
Mar 28, 2022
Merged

set $NINJAFLAGS to make sure Ninja doesn't use all visible cores when building Qt5 #2338

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c74e333
set $NINJAFLAGS to make sure Ninja doesn't use all visible cores when…
boegel Feb 18, 2021
1940c1b
Merge branch 'develop' into 20210218120915_new_pr_EWJjRhxdvd
boegel Mar 25, 2022
38d6fb5
add comment to clarify that $NINJAFLAGS is very specific to Qt5 build…
boegel Mar 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions easybuild/easyblocks/q/qt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,13 @@
import os
from distutils.version import LooseVersion

import easybuild.tools.environment as env
import easybuild.tools.toolchain as toolchain
from easybuild.easyblocks.generic.configuremake import ConfigureMake
from easybuild.framework.easyconfig import CUSTOM
from easybuild.tools.build_log import EasyBuildError
from easybuild.tools.filetools import apply_regex_substitutions
from easybuild.tools.modules import get_software_root
from easybuild.tools.run import run_cmd_qa
from easybuild.tools.systemtools import get_cpu_architecture, get_glibc_version, get_shared_lib_ext
from easybuild.tools.systemtools import AARCH64, POWER
Expand Down Expand Up @@ -154,6 +156,13 @@ def configure_step(self):
]
run_cmd_qa(cmd, qa, no_qa=no_qa, log_all=True, simple=True, maxhits=120)

# Ninja uses all visible cores by default, which can lead to lack of sufficient memory;
# so $NINJAFLAGS is set to control number of parallel processes used by Ninja;
# note that $NINJAFLAGS is not a generic thing for Ninja, it's very specific to the Qt5 build procedure
if LooseVersion(self.version) >= LooseVersion('5'):
if get_software_root('Ninja'):
env.setvar('NINJAFLAGS', '-j%s' % self.cfg['parallel'])
Copy link
Contributor

@akesandgren akesandgren May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's likely a bad idea to overwrite NINJAFLAGS like this. User could have set it for some reason. And it should check if "-j" is already in NINJAFLAGS and not override it if it is...

Copy link
Contributor

@surak surak Jul 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should one add this variable on ninja itself? Or make ninja respect parallel?

Copy link
Member Author

@boegel boegel Mar 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just figured out that $NINJAFLAGS is very specific to Qt5, so in that case I'd argue that ignoring $NINJAFLAGS if it's set already actually makes total sense (you don't want a random value for a pretty generically named environment variable like $NINJAFLAGS to mess up your Qt5 build).


def build_step(self):
"""Set $LD_LIBRARY_PATH before calling make, to ensure that all required libraries are found during linking."""
# cfr. https://elist.ornl.gov/pipermail/visit-developers/2011-September/010063.html
Expand Down