Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
134 commits
Select commit Hold shift + click to select a range
71e2e01
bump version to 4.2.1dev
boegel Apr 14, 2020
45d2897
relax regex used for PyPI URLs for easybuild packages in test_pypi_so…
boegel Apr 14, 2020
12041a1
Merge pull request #3279 from boegel/develop
Micket Apr 14, 2020
3220e99
Add CPU architecture as found by EB to test report
Flamefire Apr 15, 2020
52af67d
cast CPU arch name provided by archspec to a regular string (fixes #3…
boegel Apr 17, 2020
59d7666
add test to check whether easyconfig that includes unicode character …
boegel Apr 17, 2020
715f7c0
Introduce contextmanager to disable and restore templating
Flamefire Apr 17, 2020
91c7f0d
Don't resolve templates where it isn't required
Flamefire Apr 17, 2020
1cb09f2
Merge pull request #3286 from boegel/fix_unicode_descr
smoors Apr 17, 2020
f8844f7
add support for enhancing existing sanity check
boegel Apr 17, 2020
7fa6c42
Merge pull request #3281 from Flamefire/improve_test_report
boegel Apr 18, 2020
184791d
avoid that test_toy_build_enhanced_sanity_check causes trouble in oth…
boegel Apr 18, 2020
a4d7cad
add create_lock, check_lock and remove_lock function + use them in Ea…
boegel Apr 18, 2020
5800437
add clean_up_locks function
boegel Apr 18, 2020
5ed5b8d
make sure it's OK to call remove_lock on a lock that's no longer there
boegel Apr 18, 2020
e7f2226
add signal handler 'clean_up_locks_signal_handler' to clean up locks …
boegel Apr 19, 2020
1752c4a
register signal handlers at start of main to clean up locks on receiv…
boegel Apr 19, 2020
52acdb2
removed unused import
boegel Apr 19, 2020
7c406f5
add 'change_into_dir' named argument to 'extract_file' + print deprec…
boegel Apr 18, 2020
23167d5
take into account that sanity_check_paths values may not be lists whe…
boegel Apr 20, 2020
4ee9d25
add test for verification of sanity_check_paths
boegel Apr 20, 2020
96ec4e9
don't strictly require both files/dirs keys in sanity_check_paths in …
boegel Apr 20, 2020
4130980
ensure sorted output in sanity check step under dry run
boegel Apr 21, 2020
b89b8f4
move cleanup of toy easyblock to tearDown, to ensure it's always run,…
boegel Apr 21, 2020
8844dde
also clean up toytoy easyblock in tearDown of toy_build tests
boegel Apr 21, 2020
7c1c288
Merge pull request #3288 from boegel/enhance_sanity_check
lexming Apr 21, 2020
2a5918c
added function find_glob_pattern to filetools.py
mboisson Apr 22, 2020
6d3cc4e
appeasing hound
mboisson Apr 22, 2020
b82579d
merging upstream develop
mboisson Apr 22, 2020
4832d2d
removed extra self/
mboisson Apr 22, 2020
4908e07
get pr_title and pr_descr built_options in new_pr_from_branch instead…
migueldiascosta Apr 23, 2020
9d3e5d4
strip out 'data-yanked' from HTML page with package source URLs serve…
boegel Apr 27, 2020
4ef9c6c
appease the Hound
boegel Apr 27, 2020
e2d97f6
completely ignore releases that were yanked from PyPI
boegel Apr 27, 2020
5fa16cb
use non-greedy .*? since re.sub doesn't accept flags in Python 2.6
boegel Apr 27, 2020
8f02741
Merge pull request #3297 from ComputeCanada/find_glob_pattern
boegel Apr 27, 2020
6afc01f
fix broken test for --include-easyblocks-from-pr
boegel Apr 27, 2020
bb02431
Merge pull request #3303 from boegel/pypi_yanked_releases
lexming Apr 27, 2020
be2e192
Merge pull request #3304 from boegel/fix_broken_test
lexming Apr 27, 2020
58c15f4
Also check for module basename in module exist
Flamefire Feb 26, 2020
d7f5bc0
Constants for osdependecies.
terjekv Apr 29, 2020
d8dfe82
Appeasing the hound (length of lines)
terjekv Apr 29, 2020
8fb2273
Rename constants.
terjekv Apr 29, 2020
1def919
Further appeasement of the Hound.
terjekv Apr 29, 2020
5169fe1
add tests for ModulesTool.show and ModulesTool.run_module
boegel Apr 10, 2020
47292c9
make test_run_module a bit less strict, to take into account differen…
boegel Apr 10, 2020
a7ede60
mention full executed command in ModulesTool.run_module when command …
boegel Apr 10, 2020
a2cae66
use full module name in check for ModulesTool.run_module('avail'), to…
boegel Apr 10, 2020
a967058
Add test for available() with module_alias command
Flamefire Apr 14, 2020
0877c51
Add test with modulerc in HOME
Flamefire Apr 14, 2020
1676ae0
Add disallow_deprecated_behaviour to invert allow_deprecated_behaviour
Flamefire Apr 14, 2020
4e1f286
Deprecated module_wrapper_exists and remove its usage from exists
Flamefire Apr 14, 2020
e4566a6
Remove tests for deprecated module_wrapper_exists
Flamefire Apr 29, 2020
52feb0b
don't use distutils.dir_util in copy_dir (fixes #3306)
boegel Apr 30, 2020
3c4aa43
Fix failing exist HMNS tests on LMod and EnvironmentModulesTcl
Flamefire Apr 30, 2020
5be8530
Fix for module-alias on EnvironmentModules and Lmod 6
Flamefire Apr 29, 2020
1454dd3
Remove broken test for #368
Flamefire Apr 29, 2020
e086444
fix copying of broken symlinks with copy/copy_file
boegel Apr 30, 2020
3c80fef
always use own implementation rather than just using shutil.copytree …
boegel Apr 30, 2020
4955d8b
flesh out get_mpi_cmd_template function from Mpi.mpi_cmd_for method, …
boegel Apr 30, 2020
2af95ec
Constants may now be tuples.
terjekv Apr 30, 2020
d772e13
gitignore .mypy_cache dirs
willfurnass Apr 30, 2020
1dfd6ab
Appeasing the Hound.
terjekv Apr 30, 2020
5b191c8
Appeasing the Hound.
terjekv Apr 30, 2020
768129e
also catch shutil.Error in copy_dir
boegel Apr 30, 2020
dace69d
Improve module_wrapper_exists deprecation error
Flamefire Apr 30, 2020
20aab9f
Move mod_exists_regex_template deprecation notice
Flamefire Apr 30, 2020
8cb5517
Simplify temporary HOME dir creation
Flamefire Apr 30, 2020
9fe985a
Improve install_eb_dep.sh
Flamefire Apr 30, 2020
42aaf63
Merge pull request #3310 from boegel/no_distutils_dir_utils
lexming Apr 30, 2020
7d3b93b
print trace message for sanity check command before running it
boegel Apr 30, 2020
33d0073
Merge pull request #3216 from Flamefire/improve_module_exist
boegel Apr 30, 2020
4bb149d
raise error when original directory doesn't exist in extract_file
boegel May 1, 2020
482975a
fix silly typo: exists -> exits
boegel May 1, 2020
9090e1a
Merge pull request #3312 from boegel/get_mpi_cmd_template
May 1, 2020
4845565
Merge pull request #3316 from boegel/fix_trace_msg_sanity_check_command
akesandgren May 1, 2020
5675133
Merge branch 'develop' into lock_cleanup
boegel May 1, 2020
d879cda
add --wait-on-lock-limit and --wait-on-lock-interval configuration op…
boegel May 1, 2020
3a4bc98
fix broken test_guess_start_dir
boegel May 1, 2020
ffa87c9
modify test_new_pr_from_branch to also check that setting pr-descr works
migueldiascosta May 1, 2020
78fb8a4
Merge branch 'develop' into extract_file_chdir
boegel May 1, 2020
a594939
Merge pull request #3313 from willfurnass/gitignore-mypy-cache
boegel May 1, 2020
c0187c7
Merge pull request #3298 from migueldiascosta/fix_new_pr_from_branch_…
boegel May 1, 2020
27ab7e0
add final check for lock before giving up with an error when wait lim…
boegel May 1, 2020
270f1ed
Fix problems with CrayCCE processing when there are no actual externa…
akesandgren May 1, 2020
0627a4c
external_modules: if there is external module data with name/version …
akesandgren May 1, 2020
afa5ad6
Don't try to find software_root when MPI_MODULE_NAME is None
akesandgren May 1, 2020
f0164b5
Merge pull request #3292 from boegel/extract_file_chdir
akesandgren May 1, 2020
4b29b70
Merge pull request #3291 from boegel/lock_cleanup
akesandgren May 1, 2020
cf9fd79
Merge pull request #3287 from Flamefire/less_templating
akesandgren May 1, 2020
2f10b15
make test_find_eb_script more robust in case $EB_SCRIPT_PATH is alrea…
boegel May 1, 2020
144c28e
disable alarm signal in contextmanager used in test_toy_lock_cleanup_…
boegel May 1, 2020
b59e7a6
also reset alarm in test_toy_build_lock
boegel May 1, 2020
a6253b6
change back to original working directory before each test case in te…
boegel May 1, 2020
c7e1c9a
properly clean up after using SIGALRM signal in tests
boegel May 1, 2020
87168f8
restore original signal handler for SIGALRM rather than just setting …
boegel May 1, 2020
89c887f
enhance test for --list-software and --list-installed-software to cat…
boegel May 1, 2020
feb5b47
Merge pull request #3321 from boegel/fix_test_toy_lock_cleanup_signals
akesandgren May 1, 2020
c945760
Merge branch 'develop' into test_find_eb_script
boegel May 1, 2020
f3cafd4
Cleaner way to handle Cray toolchains lack of MPI_MODULE_NAME
akesandgren May 1, 2020
4a416f3
Merge pull request #16 from boegel/fix-crayce-problems
akesandgren May 1, 2020
af67db9
Merge branch 'develop' into fix-crayce-problems
akesandgren May 1, 2020
39ba429
Handle external modules better when short_mod_name contains name/version
akesandgren May 1, 2020
daf8659
Fix missing change, use name_version
akesandgren May 1, 2020
ed5ba4a
Merge pull request #3320 from boegel/test_find_eb_script
akesandgren May 2, 2020
1a0cbcf
undo hacky changes in _parse_dependency to inject guessed values for …
boegel May 2, 2020
ced2fb0
make HierarhicalMNS compatible with Cray toolchains (proper fix for #…
boegel May 2, 2020
837f1f3
fix special case for Cray toolchains in HierarchicalMNS.det_modpath_e…
boegel May 2, 2020
2f967ae
Refactored constant names and roles.
terjekv May 2, 2020
68db5c1
add missing HPL test easyconfig using CrayCCE as toolchain
boegel May 2, 2020
4c5892c
Testing the interpolation of the value type.
terjekv May 2, 2020
6e7fb5f
Avoid trying to print tuples as strings.
terjekv May 2, 2020
5413c1a
Merge pull request #17 from boegel/fix-crayce-problems
akesandgren May 3, 2020
dff3c2e
fix broken test_index_functions
boegel May 3, 2020
1a58c28
Merge pull request #18 from boegel/fix-crayce-problems
akesandgren May 3, 2020
0f78649
Merge pull request #3319 from akesandgren/fix-crayce-problems
boegel May 3, 2020
3b53953
Merge pull request #3314 from Flamefire/fix_install_eb_dep
akesandgren May 4, 2020
b37367c
The assert failure message, redone. Again.
terjekv May 4, 2020
3b19374
Line length control, properly done.
terjekv May 4, 2020
1064122
Merge pull request #3309 from terjekv/osdeps_constants
boegel May 4, 2020
7cb196a
New variable 'moddependpaths' to resolve dependencies at load time.
kelseymh May 5, 2020
1eb9a71
Appease the Hound, removing extraneous whitespace and tab characters.
kelseymh May 5, 2020
82e3ee3
Fixing quoting in expected Tcl output in deppaths test.
kelseymh May 5, 2020
0409300
if versionsuffix is explicitly set to None, it crashes below
mboisson May 5, 2020
85303a3
filtering out bad matches due to the absence of a label for system to…
mboisson May 5, 2020
e7bdbe0
fixed incorrect renaming of variable toolchain_candidate
mboisson May 5, 2020
1510de5
Improve phrasing of new moddependpaths descriptions.
kelseymh May 6, 2020
3f61876
Appease the Hound after improving phrasing of new moddependpaths desc…
kelseymh May 6, 2020
70b86fa
Merge pull request #3324 from kelseymh/3323_dependency_search_in_EC
May 6, 2020
f1494b0
using TC_CONSTANT_SYSTEM instead of "SYSTEM", and comparing versionsu…
mboisson May 6, 2020
7027f32
Merge pull request #3325 from ComputeCanada/fix_tweak
May 6, 2020
fa8c9a6
Make error for version suffix mapping more explicit
May 7, 2020
4e64dff
Make error for version suffix mapping more explicit
May 7, 2020
828687c
Merge branch 'develop' into fix_tweak_versionsuffixes
May 7, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ build/
dist/
*egg-info/
*.swp
.mypy_cache/

Dockerfile.*
Singularity.*
Expand Down
161 changes: 106 additions & 55 deletions easybuild/framework/easyblock.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,12 @@
from easybuild.tools.config import install_path, log_path, package_path, source_paths
from easybuild.tools.environment import restore_env, sanitize_env
from easybuild.tools.filetools import CHECKSUM_TYPE_MD5, CHECKSUM_TYPE_SHA256
from easybuild.tools.filetools import adjust_permissions, apply_patch, back_up_file
from easybuild.tools.filetools import change_dir, convert_name, compute_checksum, copy_file, derive_alt_pypi_url
from easybuild.tools.filetools import diff_files, download_file, encode_class_name, extract_file
from easybuild.tools.filetools import adjust_permissions, apply_patch, back_up_file, change_dir, convert_name
from easybuild.tools.filetools import compute_checksum, copy_file, check_lock, create_lock, derive_alt_pypi_url
from easybuild.tools.filetools import diff_files, dir_contains_files, download_file, encode_class_name, extract_file
from easybuild.tools.filetools import find_backup_name_candidate, get_source_tarball_from_git, is_alt_pypi_url
from easybuild.tools.filetools import is_binary, is_sha256_checksum, mkdir, move_file, move_logs, read_file, remove_dir
from easybuild.tools.filetools import remove_file, verify_checksum, weld_paths, write_file, dir_contains_files
from easybuild.tools.filetools import remove_file, remove_lock, verify_checksum, weld_paths, write_file
from easybuild.tools.hooks import BUILD_STEP, CLEANUP_STEP, CONFIGURE_STEP, EXTENSIONS_STEP, FETCH_STEP, INSTALL_STEP
from easybuild.tools.hooks import MODULE_STEP, PACKAGE_STEP, PATCH_STEP, PERMISSIONS_STEP, POSTITER_STEP, POSTPROC_STEP
from easybuild.tools.hooks import PREPARE_STEP, READY_STEP, SANITYCHECK_STEP, SOURCE_STEP, TEST_STEP, TESTCASES_STEP
Expand Down Expand Up @@ -1014,6 +1014,25 @@ def make_devel_module(self, create_in_builddir=False):
# cleanup: unload fake module, remove fake module dir
self.clean_up_fake_module(fake_mod_data)

def make_module_deppaths(self):
"""
Add specific 'module use' actions to module file, in order to find
dependencies outside the end user's MODULEPATH.
"""
deppaths = self.cfg['moddependpaths']
if not deppaths:
return ''
elif not isinstance(deppaths, (str, list, tuple)):
raise EasyBuildError("moddependpaths value %s (type: %s) is not a string, list or tuple",
deppaths, type(deppaths))

if isinstance(deppaths, str):
txt = self.module_generator.use([deppaths], guarded=True)
else:
txt = self.module_generator.use(deppaths, guarded=True)

return txt

def make_module_dep(self, unload_info=None):
"""
Make the dependencies for the module file.
Expand Down Expand Up @@ -1194,7 +1213,8 @@ def make_module_extra_extensions(self):
lines = [self.module_extra_extensions]

# set environment variable that specifies list of extensions
exts_list = ','.join(['%s-%s' % (ext[0], ext[1]) for ext in self.cfg['exts_list']])
# We need only name and version, so don't resolve templates
exts_list = ','.join(['-'.join(ext[:2]) for ext in self.cfg.get_ref('exts_list')])
env_var_name = convert_name(self.name, upper=True)
lines.append(self.module_generator.set_environment('EBEXTSLIST%s' % env_var_name, exts_list))

Expand All @@ -1207,7 +1227,7 @@ def make_module_footer(self):
footer = [self.module_generator.comment("Built with EasyBuild version %s" % VERBOSE_VERSION)]

# add extra stuff for extensions (if any)
if self.cfg['exts_list']:
if self.cfg.get_ref('exts_list'):
footer.append(self.make_module_extra_extensions())

# include modules footer if one is specified
Expand Down Expand Up @@ -1791,7 +1811,7 @@ def fetch_step(self, skip_checksums=False):
trace_msg(msg)

# fetch extensions
if self.cfg['exts_list']:
if self.cfg.get_ref('exts_list'):
self.exts = self.fetch_extension_sources(skip_checksums=skip_checksums)

# create parent dirs in install and modules path already
Expand Down Expand Up @@ -1911,7 +1931,9 @@ def extract_step(self):
"""
for src in self.src:
self.log.info("Unpacking source %s" % src['name'])
srcdir = extract_file(src['path'], self.builddir, cmd=src['cmd'], extra_options=self.cfg['unpack_options'])
srcdir = extract_file(src['path'], self.builddir, cmd=src['cmd'],
extra_options=self.cfg['unpack_options'], change_into_dir=False)
change_dir(srcdir)
if srcdir:
self.src[self.src.index(src)]['finalpath'] = srcdir
else:
Expand Down Expand Up @@ -2063,7 +2085,7 @@ def extensions_step(self, fetch=False):
- find source for extensions, in 'extensions' (and 'packages' for legacy reasons)
- run extra_extensions
"""
if len(self.cfg['exts_list']) == 0:
if not self.cfg.get_ref('exts_list'):
self.log.debug("No extensions in exts_list")
return

Expand Down Expand Up @@ -2409,37 +2431,71 @@ def _sanity_check_step_common(self, custom_paths, custom_commands):
SANITY_CHECK_PATHS_DIRS: ("(non-empty) directory", lambda dp: os.path.isdir(dp) and os.listdir(dp)),
}

# prepare sanity check paths
paths = self.cfg['sanity_check_paths']
if not paths:
enhance_sanity_check = self.cfg['enhance_sanity_check']
ec_commands = self.cfg['sanity_check_commands']
ec_paths = self.cfg['sanity_check_paths']

# if enhance_sanity_check is not enabled, only sanity_check_paths specified in the easyconfig file are used,
# the ones provided by the easyblock (via custom_paths) are ignored
if ec_paths and not enhance_sanity_check:
paths = ec_paths
self.log.info("Using (only) sanity check paths specified by easyconfig file: %s", paths)
else:
# if no sanity_check_paths are specified in easyconfig,
# we fall back to the ones provided by the easyblock via custom_paths
if custom_paths:
paths = custom_paths
self.log.info("Using customized sanity check paths: %s" % paths)
self.log.info("Using customized sanity check paths: %s", paths)
# if custom_paths is empty, we fall back to a generic set of paths:
# non-empty bin/ + /lib or /lib64 directories
else:
paths = {}
for key in path_keys_and_check:
paths.setdefault(key, [])
paths.update({SANITY_CHECK_PATHS_DIRS: ['bin', ('lib', 'lib64')]})
self.log.info("Using default sanity check paths: %s" % paths)
self.log.info("Using default sanity check paths: %s", paths)

# if enhance_sanity_check is enabled *and* sanity_check_paths are specified in the easyconfig,
# those paths are used to enhance the paths provided by the easyblock
if enhance_sanity_check and ec_paths:
for key in ec_paths:
val = ec_paths[key]
if isinstance(val, list):
paths[key] = paths.get(key, []) + val
else:
error_pattern = "Incorrect value type in sanity_check_paths, should be a list: "
error_pattern += "%s (type: %s)" % (val, type(val))
raise EasyBuildError(error_pattern)
self.log.info("Enhanced sanity check paths after taking into account easyconfig file: %s", paths)

sorted_keys = sorted(paths.keys())
known_keys = sorted(path_keys_and_check.keys())

# verify sanity_check_paths value: only known keys, correct value types, at least one non-empty value
only_list_values = all(isinstance(x, list) for x in paths.values())
only_empty_lists = all(not x for x in paths.values())
if sorted_keys != known_keys or not only_list_values or only_empty_lists:
error_msg = "Incorrect format for sanity_check_paths: should (only) have %s keys, "
error_msg += "values should be lists (at least one non-empty)."
raise EasyBuildError(error_msg % ', '.join("'%s'" % k for k in known_keys))

# if enhance_sanity_check is not enabled, only sanity_check_commands specified in the easyconfig file are used,
# the ones provided by the easyblock (via custom_commands) are ignored
if ec_commands and not enhance_sanity_check:
commands = ec_commands
self.log.info("Using (only) sanity check commands specified by easyconfig file: %s", commands)
else:
self.log.info("Using specified sanity check paths: %s" % paths)

ks = sorted(paths.keys())
valnottypes = [not isinstance(x, list) for x in paths.values()]
lenvals = [len(x) for x in paths.values()]
req_keys = sorted(path_keys_and_check.keys())
if not ks == req_keys or sum(valnottypes) > 0 or sum(lenvals) == 0:
raise EasyBuildError("Incorrect format for sanity_check_paths (should (only) have %s keys, "
"values should be lists (at least one non-empty)).", ','.join(req_keys))

commands = self.cfg['sanity_check_commands']
if not commands:
if custom_commands:
commands = custom_commands
self.log.info("Using customised sanity check commands: %s" % commands)
self.log.info("Using customised sanity check commands: %s", commands)
else:
commands = []
self.log.info("Using specified sanity check commands: %s" % commands)

# if enhance_sanity_check is enabled, the sanity_check_commands specified in the easyconfig file
# are combined with those provided by the easyblock via custom_commands
if enhance_sanity_check and ec_commands:
commands = commands + ec_commands
self.log.info("Enhanced sanity check commands after taking into account easyconfig file: %s", commands)

for i, command in enumerate(commands):
# set command to default. This allows for config files with
Expand Down Expand Up @@ -2475,9 +2531,17 @@ def _sanity_check_step_dry_run(self, custom_paths=None, custom_commands=None, **
"""
paths, path_keys_and_check, commands = self._sanity_check_step_common(custom_paths, custom_commands)

for key, (typ, _) in path_keys_and_check.items():
for key in [SANITY_CHECK_PATHS_FILES, SANITY_CHECK_PATHS_DIRS]:
(typ, _) = path_keys_and_check[key]
self.dry_run_msg("Sanity check paths - %s ['%s']", typ, key)
if paths[key]:
entries = paths[key]
if entries:
# some entries may be tuple values,
# we need to convert them to strings first so we can print them sorted
for idx, entry in enumerate(entries):
if isinstance(entry, tuple):
entries[idx] = ' or '.join(entry)

for path in sorted(paths[key]):
self.dry_run_msg(" * %s", str(path))
else:
Expand Down Expand Up @@ -2608,6 +2672,9 @@ def xs2str(xs):

# run sanity check commands
for command in commands:

trace_msg("running command '%s' ..." % command)

out, ec = run_cmd(command, simple=False, log_ok=False, log_all=False, trace=False)
if ec != 0:
fail_msg = "sanity check command %s exited with code %s (output: %s)" % (command, ec, out)
Expand All @@ -2616,7 +2683,7 @@ def xs2str(xs):
else:
self.log.info("sanity check command %s ran successfully! (output: %s)" % (command, out))

trace_msg("running command '%s': %s" % (command, ('FAILED', 'OK')[ec == 0]))
trace_msg("result for command '%s': %s" % (command, ('FAILED', 'OK')[ec == 0]))

# also run sanity check for extensions (unless we are an extension ourselves)
if not extension:
Expand Down Expand Up @@ -2723,6 +2790,7 @@ def make_module_step(self, fake=False):

txt += self.make_module_description()
txt += self.make_module_group_check()
txt += self.make_module_deppaths()
txt += self.make_module_dep()
txt += self.make_module_extend_modpath()
txt += self.make_module_req()
Expand Down Expand Up @@ -3049,30 +3117,14 @@ def run_all_steps(self, run_test_cases):
if ignore_locks:
self.log.info("Ignoring locks...")
else:
locks_dir = build_option('locks_dir') or os.path.join(install_path('software'), '.locks')
lock_path = os.path.join(locks_dir, '%s.lock' % self.installdir.replace('/', '_'))

# if lock already exists, either abort or wait until it disappears
if os.path.exists(lock_path):
wait_on_lock = build_option('wait_on_lock')
if wait_on_lock:
while os.path.exists(lock_path):
print_msg("lock %s exists, waiting %d seconds..." % (lock_path, wait_on_lock),
silent=self.silent)
time.sleep(wait_on_lock)
else:
raise EasyBuildError("Lock %s already exists, aborting!", lock_path)
lock_name = self.installdir.replace('/', '_')

# create lock to avoid that another installation running in parallel messes things up;
# we use a directory as a lock, since that's atomically created
try:
mkdir(lock_path, parents=True)
except EasyBuildError as err:
# clean up the error message a bit, get rid of the "Failed to create directory" part + quotes
stripped_err = str(err).split(':', 1)[1].strip().replace("'", '').replace('"', '')
raise EasyBuildError("Failed to create lock %s: %s", lock_path, stripped_err)
# check if lock already exists;
# either aborts with an error or waits until it disappears (depends on --wait-on-lock)
check_lock(lock_name)

self.log.info("Lock created: %s", lock_path)
# create lock to avoid that another installation running in parallel messes things up
create_lock(lock_name)

try:
for (step_name, descr, step_methods, skippable) in steps:
Expand All @@ -3090,8 +3142,7 @@ def run_all_steps(self, run_test_cases):
pass
finally:
if not ignore_locks:
remove_dir(lock_path)
self.log.info("Lock removed: %s", lock_path)
remove_lock(lock_name)

# return True for successfull build (or stopped build)
return True
Expand Down
9 changes: 9 additions & 0 deletions easybuild/framework/easyconfig/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,13 @@
'OS_VERSION': (get_os_version(), "System version"),
'SYS_PYTHON_VERSION': (platform.python_version(), "System Python version (platform.python_version())"),
'SYSTEM': ({'name': 'system', 'version': 'system'}, "System toolchain"),

'OS_PKG_IBVERBS_DEV': (('libibverbs-dev', 'libibverbs-devel', 'rdma-core-devel'),
"OS packages providing ibverbs/infiniband development support"),
'OS_PKG_OPENSSL_BIN': (('openssl'),
"OS packages providing the openSSL binary"),
'OS_PKG_OPENSSL_LIB': (('libssl', 'libopenssl'),
"OS packages providing openSSL libraries"),
'OS_PKG_OPENSSL_DEV': (('openssl-devel', 'libssl-dev', 'libopenssl-devel'),
"OS packages providing openSSL developement support"),
}
3 changes: 3 additions & 0 deletions easybuild/framework/easyconfig/default.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@
'easyblock': [None, "EasyBlock to use for building; if set to None, an easyblock is selected "
"based on the software name", BUILD],
'easybuild_version': [None, "EasyBuild-version this spec-file was written for", BUILD],
'enhance_sanity_check': [False, "Indicate that additional sanity check commands & paths should enhance "
"the existin sanity check, not replace it", BUILD],
'fix_perl_shebang_for': [None, "List of files for which Perl shebang should be fixed "
"to '#!/usr/bin/env perl' (glob patterns supported)", BUILD],
'fix_python_shebang_for': [None, "List of files for which Python shebang should be fixed "
Expand Down Expand Up @@ -152,6 +154,7 @@
'multi_deps': [{}, "Dict of lists of dependency versions over which to iterate", DEPENDENCIES],
'multi_deps_load_default': [True, "Load module for first version listed in multi_deps by default", DEPENDENCIES],
'osdependencies': [[], "OS dependencies that should be present on the system", DEPENDENCIES],
'moddependpaths': [None, "Absolute path(s) to prepend to MODULEPATH before loading dependencies", DEPENDENCIES],

# LICENSE easyconfig parameters
'group': [None, "Name of the user group for which the software should be available; "
Expand Down
Loading