Sanity checks break when package installs lib64 instead of lib

[olesenm]

Just building the foss-2016a toolchain failed since the sanity checks for lib/ fail (it can also be lib64/).

I experimented with easybuild a few months back, but gave up several times in total frustration. But now decided to give it another attempt. OS: opensuse-42.2 x86_64

Here there are a snippets, since I don't know the better way of doing it, and it probably hits elsewhere too, so a better overall solution is likely needed anyhow. I'll see if I hit more along the way.

--- a/easybuild/easyblocks/b/binutils.py
+++ b/easybuild/easyblocks/b/binutils.py
@@ -159,7 +159,8 @@ class EB_binutils(ConfigureMake):

         custom_paths = {
             'files': [os.path.join('bin', b) for b in binaries] +
-                     [os.path.join('lib', 'lib%s.%s' % (l, ext)) for l in libs for ext in lib_exts] +
+#                     [os.path.join('lib', 'lib%s.%s' % (l, ext)) for l in libs for ext in lib_exts] +
+                     [os.path.join('lib64', 'lib%s.%s' % (l, ext)) for l in libs for ext in lib_exts] +
                      [os.path.join('include', h) for h in headers],
             'dirs': [],
         }
--- a/easybuild/easyconfigs/b/Bison/Bison-3.0.4.eb
+++ b/easybuild/easyconfigs/b/Bison/Bison-3.0.4.eb
@@ -15,7 +15,8 @@ source_urls = [GNU_SOURCE]
 builddependencies = [('M4', '1.4.17')]

 sanity_check_paths = {
-    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+#    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib64/liby.a"],
     'dirs': [],
 }

[olesenm]

--- a/easybuild/easyconfigs/b/Bison/Bison-3.0.4-GCCcore-4.9.3.eb
+++ b/easybuild/easyconfigs/b/Bison/Bison-3.0.4-GCCcore-4.9.3.eb
@@ -20,7 +20,8 @@ builddependencies = [


 sanity_check_paths = {
-    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+#    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib64/liby.a"],
     'dirs': [],
 }

[olesenm]

--- a/easybuild/easyconfigs/n/numactl/numactl-2.0.11-GCC-4.9.3-2.25.eb
+++ b/easybuild/easyconfigs/n/numactl/numactl-2.0.11-GCC-4.9.3-2.25.eb
@@ -16,7 +16,8 @@ sources = [SOURCE_TAR_GZ]
 checksums = ['d3bc88b7ddb9f06d60898f4816ae9127']

 sanity_check_paths = {
-    'files': ['bin/numactl', 'bin/numastat', 'lib/libnuma.%s' % SHLIB_EXT, 'lib/libnuma.a'],
+#    'files': ['bin/numactl', 'bin/numastat', 'lib/libnuma.%s' % SHLIB_EXT, 'lib/libnuma.a'],
+    'files': ['bin/numactl', 'bin/numastat', 'lib64/libnuma.%s' % SHLIB_EXT, 'lib64/libnuma.a'],
     'dirs': ['share/man', 'include']
 }

[olesenm]

--- a/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.10.2-GCC-4.9.3-2.25.eb
+++ b/easybuild/easyconfigs/o/OpenMPI/OpenMPI-1.10.2-GCC-4.9.3-2.25.eb
@@ -24,7 +24,8 @@ osdependencies = [('libibverbs-dev', 'libibverbs-devel')]
 libs = ["mpi_cxx", "mpi_mpifh", "mpi", "ompitrace", "open-pal", "open-rte", "vt", "vt-hyb", "vt-mpi", "vt-mpi-unify"]
 sanity_check_paths = {
     'files': ["bin/%s" % binfile for binfile in ["ompi_info", "opal_wrapper", "orterun"]] +
-             ["lib/lib%s.%s" % (libfile, SHLIB_EXT) for libfile in libs] +
+#             ["lib/lib%s.%s" % (libfile, SHLIB_EXT) for libfile in libs] +
+             ["lib64/lib%s.%s" % (libfile, SHLIB_EXT) for libfile in libs] +
              ["include/%s.h" % x for x in ["mpi-ext", "mpif-config", "mpif", "mpi", "mpi_portable_platform"]],
     'dirs': ["include/openmpi/ompi/mpi/cxx"],
 }

[olesenm]

--- a/easybuild/easyconfigs/f/FFTW/FFTW-3.3.4-gompi-2016a.eb
+++ b/easybuild/easyconfigs/f/FFTW/FFTW-3.3.4-gompi-2016a.eb
@@ -26,9 +26,12 @@ sanity_check_paths = {
     'files': ['bin/fftw%s' % x for x in ['-wisdom', '-wisdom-to-conf', 'f-wisdom', 'l-wisdom', 'q-wisdom']] +
              ['include/fftw3%s' % x for x in ['-mpi.f03', '-mpi.h', '.f', '.f03',
                                               '.h', 'l-mpi.f03', 'l.f03', 'q.f03']] +
-             ['lib/libfftw3%s%s.a' % (x, y) for x in ['', 'f', 'l'] for y in ['', '_mpi', '_omp', '_threads']] +
-             ['lib/libfftw3q.a', 'lib/libfftw3q_omp.a'],
-    'dirs': ['lib/pkgconfig'],
+#             ['lib/libfftw3%s%s.a' % (x, y) for x in ['', 'f', 'l'] for y in ['', '_mpi', '_omp', '_threads']] +
+#             ['lib/libfftw3q.a', 'lib/libfftw3q_omp.a'],
+#    'dirs': ['lib/pkgconfig'],
+             ['lib64/libfftw3%s%s.a' % (x, y) for x in ['', 'f', 'l'] for y in ['', '_mpi', '_omp', '_threads']] +
+             ['lib64/libfftw3q.a', 'lib64/libfftw3q_omp.a'],
+    'dirs': ['lib64/pkgconfig'],
 }

[olesenm]

--- a/easybuild/easyconfigs/n/ncurses/ncurses-6.0-foss-2016a.eb
+++ b/easybuild/easyconfigs/n/ncurses/ncurses-6.0-foss-2016a.eb
@@ -27,9 +27,12 @@ libs = ["form", "menu", "ncurses", "panel"]
 sanity_check_paths = {
     'files': ['bin/%s' % x for x in ["captoinfo", "clear", "infocmp", "infotocap", "ncurses%(version_major)s-config",
                                      "reset", "tabs", "tic", "toe", "tput", "tset"]] +
-             ['lib/lib%s%s.a' % (x, y) for x in libs for y in ['', '_g', 'w', 'w_g']] +
-             ['lib/lib%s%s.so' % (x, y) for x in libs for y in ['', 'w']] +
-             ['lib/libncurses++%s.a' % x for x in ['', 'w']],
+#             ['lib/lib%s%s.a' % (x, y) for x in libs for y in ['', '_g', 'w', 'w_g']] +
+#             ['lib/lib%s%s.so' % (x, y) for x in libs for y in ['', 'w']] +
+#             ['lib/libncurses++%s.a' % x for x in ['', 'w']],
+             ['lib64/lib%s%s.a' % (x, y) for x in libs for y in ['', '_g', 'w', 'w_g']] +
+             ['lib64/lib%s%s.so' % (x, y) for x in libs for y in ['', 'w']] +
+             ['lib64/libncurses++%s.a' % x for x in ['', 'w']],
     'dirs': ['include', 'include/ncursesw'],
 }

[olesenm]

--- a/easybuild/easyconfigs/l/libreadline/libreadline-6.3-foss-2016a.eb
+++ b/easybuild/easyconfigs/l/libreadline/libreadline-6.3-foss-2016a.eb
@@ -23,7 +23,8 @@ dependencies = [('ncurses', '6.0')]
 preconfigopts = "env LDFLAGS='-lncurses'"

 sanity_check_paths = {
-    'files': ['lib/libreadline.a', 'lib/libhistory.a'] +
+#    'files': ['lib/libreadline.a', 'lib/libhistory.a'] +
+    'files': ['lib64/libreadline.a', 'lib64/libhistory.a'] +
              ['include/readline/%s' % x for x in ['chardefs.h', 'history.h', 'keymaps.h', 'readline.h', 'rlconf.h',
                                                   'rlstdc.h', 'rltypedefs.h', 'tilde.h']],
     'dirs': [],

[olesenm]

Almost the same thing as before, but in a slightly different file...

--- a/easybuild/easyconfigs/b/Bison/Bison-3.0.4-foss-2016a.eb
+++ b/easybuild/easyconfigs/b/Bison/Bison-3.0.4-foss-2016a.eb
@@ -15,7 +15,8 @@ source_urls = [GNU_SOURCE]
 builddependencies = [('M4', '1.4.17')]

 sanity_check_paths = {
-    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+#    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib/liby.a"],
+    'files': ["bin/%s" % x for x in ["bison", "yacc"]] + ["lib64/liby.a"],
     'dirs': [],
 }

[wpoely86]

@olesenm Good luck with EasyBuild. Don't hesitate to drop by in #easybuild on Freenode (IRC) if you hit issues.

We indeed assume stuff gets installed in lib instead of lib64. Most testing is done on CentOS. However, OpenSUSE has a different policy where it picks lib64 instead. You can change this for the system: https://stackoverflow.com/questions/9247769/lib-directory-on-64-bit-opensuse-distribution
I recommend you at least change it for running EasyBuild because you will hit many many failures in the sanity check otherwise.

@boegel Maybe we should document the OpenSUSE case somewhere?

[olesenm]

What about having sanity_check_paths with a libs entry that would check for the library in either lib or lib64? Seems to affect other things too (eg, CGAL, gcc, python, ...) It seems a bit heavy-handed to artificially squash these into lib.

Maybe something like this?

sanity_check_paths = {
    'files': ["bin/%s" % x for x in ["bison", "yacc"]],
    'libs':  ["liby.a"],
    'dirs': [],
}

[boegel]

@olesenm Thanks for giving EasyBuild another go, I'm sorry it got you frustrated before and made you (temporarily) give up on it. Do share any pains you suffered, so we can work towards fixing them.

Relevant change proposal that never made it in (since it's a workaround rather than a proper fix): easybuilders/easybuild-framework#1405 .

The general idea of listing libraries separately has existed for a while, and can also be extended to binaries and header_files, see easybuilders/easybuild-framework#196.

An implementation of this feature was worked on by @torbjoernk, but it didn't get finished/merged, see easybuilders/easybuild-framework#1424 .

Currently, we try to consider both lib/ and lib64/ in the sanity check (finding one is sufficient), but we're definitely not consistent there, for example:

'files': [('lib/liby.a', 'lib64/liby.a')],

In the long run, listing libraries separately is probably better, but also a lot more effort (since it requires updating a lot of sanity checks).

[boegel]

I hunted down the relevant post from the EasyBuild mailing list from @berndmohr who was hitting this problem as well for a while: https://lists.ugent.be/wws/arc/easybuild/2015-05/msg00001.html

[olesenm]

This really needs to either be documented, configured as a part of EasyBuild, or the sanity checks need to be generalized. I just tried building with foss-2016b and hit exactly the same problems yet again.

[ocaisa]

I guess the solution would be to unset CONFIG_SITE in the Autotools easyconfig that you use. Autotools is just a bundle though so there is not so much flexibility, the only solution I can see being to add

modtclfooter = "unsetenv CONFIG_SITE"
modluafooter = 'unsetenv ("CONFIG_SITE")'

(untested)

[boegel]

This change request that never made it through (yet) is very relevant: easybuilders/easybuild-framework#1405

We can also consider adding support for specific changes to be made to the environment every time eb is run, for example by making eb (which is a bash script) source a particular startup script (where you could include unsetenv CONFIG_SITE, maybe alongside other site-specific changes to the environment).

[boegel]

fixed for a while now, since we symlink lib64 to lib if it doesn't exist since EasyBuild v4.3.3, see easybuilders/easybuild-framework#3549