failing installation of GCCcore/10.3.0 on top of EESSI 2023.04 compat layer

[boegel]

The installation of GCCcore/10.3.0 is failing with EasyBuild v4.7.1 on top of the EESSI pilot 2023.04 compat layer, with:

/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/gcc-10.3.0/stage1_obj/./gcc/xgcc -B/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/gcc-10.3.0/stage1_obj/./gcc/ -B/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/GCC_stage1_eb/x86_64-pc-linux-gnu/bin/ -B/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/GCC_stage1_eb/x86_64-pc-linux-gnu/lib/ -isystem /tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/GCC_stage1_eb/x86_64-pc-linux-gnu/include -isystem /tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/GCC_stage1_eb/x86_64-pc-linux-gnu/sys-include    -O2  -g -O2 -DIN_GCC    -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include  -fpic -mlong-double-80 -DUSE_ELF_SYMVER  -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -shared -nodefaultlibs -Wl,--soname=libgcc_s.so.1 -Wl,--version-script=libgcc.map -o ./libgcc_s.so.1.tmp -g -O2 -B./ _muldi3_s.o _negdi2_s.o _lshrdi3_s.o _ashldi3_s.o _ashrdi3_s.o _cmpdi2_s.o _ucmpdi2_s.o _clear_cache_s.o _trampoline_s.o __main_s.o _absvsi2_s.o _absvdi2_s.o _addvsi3_s.o _addvdi3_s.o _subvsi3_s.o _subvdi3_s.o _mulvsi3_s.o _mulvdi3_s.o _negvsi2_s.o _negvdi2_s.o _ctors_s.o _ffssi2_s.o _ffsdi2_s.o _clz_s.o _clzsi2_s.o _clzdi2_s.o _ctzsi2_s.o _ctzdi2_s.o _popcount_tab_s.o _popcountsi2_s.o _popcountdi2_s.o _paritysi2_s.o _paritydi2_s.o _powisf2_s.o _powidf2_s.o _powixf2_s.o _mulhc3_s.o _mulsc3_s.o _muldc3_s.o _mulxc3_s.o _divhc3_s.o _divsc3_s.o _divdc3_s.o _divxc3_s.o _bswapsi2_s.o _bswapdi2_s.o _clrsbsi2_s.o _clrsbdi2_s.o _fixunssfsi_s.o _fixunsdfsi_s.o _fixunsxfsi_s.o _fixsfdi_s.o _fixdfdi_s.o _fixxfdi_s.o _fixunssfdi_s.o _fixunsdfdi_s.o _fixunsxfdi_s.o _floatdisf_s.o _floatdidf_s.o _floatdixf_s.o _floatundisf_s.o _floatundidf_s.o _floatundixf_s.o _divdi3_s.o _moddi3_s.o _divmoddi4_s.o _udivdi3_s.o _umoddi3_s.o _udivmoddi4_s.o _udiv_w_sdiv_s.o cpuinfo_s.o sfp-exceptions_s.o addtf3_s.o divtf3_s.o multf3_s.o negtf2_s.o subtf3_s.o unordtf2_s.o fixtfsi_s.o fixunstfsi_s.o floatsitf_s.o floatunsitf_s.o fixtfdi_s.o fixunstfdi_s.o floatditf_s.o floatunditf_s.o fixtfti_s.o fixunstfti_s.o floattitf_s.o floatuntitf_s.o extendsftf2_s.o extenddftf2_s.o extendxftf2_s.o trunctfsf2_s.o trunctfdf2_s.o trunctfxf2_s.o getf2_s.o letf2_s.o eqtf2_s.o _divtc3_s.o _multc3_s.o _powitf2_s.o enable-execute-stack_s.o unwind-dw2_s.o unwind-dw2-fde-dip_s.o unwind-sjlj_s.o unwind-c_s.o emutls_s.o libgcc.a -lc && rm -f ./libgcc_s.so && if [ -f ./libgcc_s.so.1 ]; then mv -f ./libgcc_s.so.1 ./libgcc_s.so.1.backup; else true; fi && mv ./libgcc_s.so.1.tmp ./libgcc_s.so.1 && (echo "/* GNU ld script"; echo "   Use the shared library, but some functions are only in"; echo "   the static library.  */"; echo "GROUP ( libgcc_s.so.1 -lgcc )" ) > ./libgcc_s.so
/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/bin/ld.gold: error: cannot open /lib64/libc.so.6: No such file or directory
/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/bin/ld.gold: error: cannot open /usr/lib64/libc_nonshared.a: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:994: libgcc_s.so] Error 1
make[2]: Leaving directory '/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/gcc-10.3.0/stage1_obj/x86_64-pc-linux-gnu/libgcc'
make[1]: *** [Makefile:14332: all-target-libgcc] Error 2
make[1]: Leaving directory '/tmp/boegel/easybuild/build/GCCcore/10.3.0/system-system/gcc-10.3.0/stage1_obj'
make: *** [Makefile:967: all] Error 2

see also NorESSI#96 (cc @trz42)

[boegel]

This problem is fixed if the patched GCC easyblock from easybuilders/easybuild-easyblocks#2921 is used.

[boegel]

Same problem occurs with GCCcore/11.3.0.

Although easybuilders/easybuild-easyblocks#2921 helps with fixing this build problem that happens just a couple of minutes into stage 1 of the GCC bootstrap procedure, it doesn't lead to a fully working installation in that case, since the RPATH part of sanity check for GCCcore/11.3.0 fails:

== FAILED: Installation ended unsuccessfully (build directory: /tmp/vsc40023/easybuild/build/GCCcore/11.3.0/system-system): build failed (first 300 chars): Sanity check failed: No '(RPATH)' found in
'readelf -d' output for /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/bin/x86_64-pc-linux-gnu-gcc-11.3.0

Indeed, there's no RPATH section in binaries/libraries, although there should be:

Details

No '(RPATH)' found in 'readelf -d' output for /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib64/libgcc_s.so.1:
Dynamic section at offset 0x17dc0 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libgcc_s.so.1]
 0x000000000000000c (INIT)               0x3000
 0x000000000000000d (FINI)               0x14638
 0x0000000000000019 (INIT_ARRAY)         0x18da8
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x18db8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x270
 0x0000000000000005 (STRTAB)             0x1788
 0x0000000000000006 (SYMTAB)             0x768
 0x000000000000000a (STRSZ)              2227 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x18fe8
 0x0000000000000002 (PLTRELSZ)           1080 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x24a8
 0x0000000000000007 (RELA)               0x23a0
 0x0000000000000008 (RELASZ)             264 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffc (VERDEF)             0x2198
 0x000000006ffffffd (VERDEFNUM)          13
 0x000000006ffffffe (VERNEED)            0x2360
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x203c
 0x000000006ffffff9 (RELACOUNT)          4
 0x0000000000000000 (NULL)               0x0

In comparison, the successful GCCcore/10.3.0 install does have an RPATH section, as expected:

Details

$ readelf -d /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/10.3.0/lib64/libgcc_s.so.1

Dynamic section at offset 0x16da8 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000003 (PLTGOT)             0x17fe8
 0x0000000000000002 (PLTRELSZ)           1128 (bytes)
 0x0000000000000017 (JMPREL)             0x2688
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000007 (RELA)               0x2598
 0x0000000000000008 (RELASZ)             240 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000006 (SYMTAB)             0x220
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000005 (STRTAB)             0x1240
 0x000000000000000a (STRSZ)              2810 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x1d40
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libgcc_s.so.1]
 0x000000000000000c (INIT)               0x2af0
 0x000000000000000d (FINI)               0x13690
 0x000000000000001a (FINI_ARRAY)         0x17d90
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000019 (INIT_ARRAY)         0x17d98
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000000f (RPATH)              Library rpath: [/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/10.3.0/lib:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/10.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/lib/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/lib/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/lib:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/lib]
 0x000000006ffffff0 (VERSYM)             0x2238
 0x000000006ffffffc (VERDEF)             0x2390
 0x000000006ffffffd (VERDEFNUM)          13
 0x000000006ffffffe (VERNEED)            0x2554
 0x000000006fffffff (VERNEEDNUM)         1
 0x0000000000000000 (NULL)               0x0

Same problem was observed with GCCcore/12.2.0 on top of EESSI 2023.04 compat layer.

[boegel]

The installation of GCCcore/11.3.0 outside of EESSI while using eb --rpath works fine, it leads to binaries/libraries that have an RPATH section as expected:

$ readelf -d /home/boegel/software/GCCcore/11.3.0/bin/gcc | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/home/boegel/software/GCCcore/11.3.0/lib:/home/boegel/software/GCCcore/11.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/home/boegel/software/binutils/2.38/lib/../lib64]

$ readelf -d /home/boegel/software/GCCcore/11.3.0/lib64/libgcc_s.so.1 | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/home/boegel/software/GCCcore/11.3.0/lib:/home/boegel/software/GCCcore/11.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/home/boegel/software/binutils/2.38/lib/../lib64]

So the problem with the failing sanity check seems to only occur when eb --sysroot is also used to install GCCcore/11.3.0 on top of the EESSI pilot 2023.04 compat layer, with the patched GCC easyblock from easybuilders/easybuild-easyblocks#2921 (or the enhanced version from trz42/easybuild-easyblocks#1 - commit 37ffbf6d001b82e1c3b117ddfc4f53a7b406bcdd).

@bartoldeman: Any ideas here?

[bartoldeman]

@boegel what's the RPATH for GCCcore/10.3.0 on top of EESSI 2023.04 ?

[trz42]

For NESSI/2023.04 (should be identical to EESSI/2023.04 compat layer), I have

$ readelf -d gcc | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/cvmfs/pilot.nessi.no/versions/2023.04/software/linux/x86_64/generic/software/GCCcore/10.3.0/lib:/cvmfs/pilot.nessi.no/versions/2023.04/software/linux/x86_64/generic/software/GCCcore/10.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/lib/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/usr/lib/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/lib:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/usr/lib]
$ readelf -d ../lib64/libgcc_s.so.1 | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/cvmfs/pilot.nessi.no/versions/2023.04/software/linux/x86_64/generic/software/GCCcore/10.3.0/lib:/cvmfs/pilot.nessi.no/versions/2023.04/software/linux/x86_64/generic/software/GCCcore/10.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/lib/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/usr/lib/../lib64:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/lib:/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64/usr/lib]

And similar for NESSI/2022.11 which has a compat layer built in November 2022.

[boegel]

@boegel what's the RPATH for GCCcore/10.3.0 on top of EESSI 2023.04 ?

see 2nd details block in #250 (comment)

[boegel]

The installation of GCCcore/11.3.0 is also failing to pass the RPATH part of the sanity check when installing on top of EESSI 2021.12 (using EasyBuild v4.7.1), so it seems like this problem is due to a change in GCC 11.x vs 10.x rather than a change in EESSI 2023.04 or the changes being made to the GCC easyblock...

Details

No '(RPATH)' found in 'readelf -d' output for /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/amd/zen3/software/GCCcore/11.3.0/lib64/libgcc_s.so.1:
Dynamic section at offset 0x17dd8 contains 27 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [libgcc_s.so.1]
0x000000000000000c (INIT) 0x3000
0x000000000000000d (FINI) 0x14638
0x0000000000000019 (INIT_ARRAY) 0x18dc0
0x000000000000001b (INIT_ARRAYSZ) 16 (bytes)
0x000000000000001a (FINI_ARRAY) 0x18dd0
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x270
0x0000000000000005 (STRTAB) 0x1788
0x0000000000000006 (SYMTAB) 0x768
0x000000000000000a (STRSZ) 2227 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000003 (PLTGOT) 0x19000
0x0000000000000002 (PLTRELSZ) 1080 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x24a8
0x0000000000000007 (RELA) 0x23a0
0x0000000000000008 (RELASZ) 264 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffffc (VERDEF) 0x2198
0x000000006ffffffd (VERDEFNUM) 13
0x000000006ffffffe (VERNEED) 0x2360
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0x203c
0x000000006ffffff9 (RELACOUNT) 4
0x0000000000000000 (NULL) 0x0

[trz42]

I compared the logs for building GCC/10.3.0 and for building GCC/11.3.0 (on top of 2023.04), and the only difference I can spot are in options of two configure commands.

== Running pre-configure hook...
10.3.0
../configure --with-sysroot=/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64 [snip] --enable-gold=default --enable-ld --with-plugin-ld=ld.gold [snip]
11.3.0
../configure --with-sysroot=/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64 [snip] --enable-gold --enable-ld=default [snip]

and

== building...
10.3.0
../configure --with-sysroot=/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64 [snip] --enable-gold=default --enable-ld --with-plugin-ld=ld.gold [snip]
11.3.0
../configure --with-sysroot=/cvmfs/pilot.nessi.no/versions/2023.04/compat/linux/x86_64 [snip] --enable-gold --enable-ld=default [snip]

So difference between 10.3.0 and 11.3.0 is --enable-gold=default --enable-ld --with-plugin-ld=ld.gold and --enable-gold --enable-ld=default. 11.3.0 lacks the --with-plugin-ld=ld.gold.

These differences are because of changed settings in the gcc easyblock (lines 532-535)

        elif self.cfg['use_gold_linker']:
            self.configopts += " --enable-gold=default --enable-ld --with-plugin-ld=ld.gold"
        else:
            self.configopts += " --enable-gold --enable-ld=default"

and gold shall not be used from 11.3.0 because of bitrot (lines 133-135)

            # Disable the Gold linker by default for GCC 11.3.0 and newer, as it suffers from bitrot
            if self.cfg['use_gold_linker'] is None:
                self.cfg['use_gold_linker'] = version < LooseVersion('11.3.0')

[boegel]

Updates:

• GCCcore/11.2.0 on top of EESSI 2023.04 with patched GCC easyblock: 🆗 - so could indeed be because we stopped using ld.gold from GCC 11.3.0 onwards, as @trz42 pointed out);
• GCCcore/11.3.0 on top of EESSI 2021.12 with EasyBuild 4.5.1: 🆗 - makes sense, because changed to not use ld.gold was made in EasyBuild v4.5.5 (see don't use gold linker by default for GCC >= 11.3 easybuilders/easybuild-easyblocks#2711);
• rebuild of GCCcore/10.3.0 on top of EESSI 2021.12 with EasyBuild v4.7.1 + patched GCC: 🆗 (makes sense)

So we need to figure out why the changes in easybuilders/easybuild-easyblocks#2711 are causing trouble when eb --rpath --sysroot=${EPREFIX} is used...

[bartoldeman]

I tried GCC 13.1 on our compatibility layer but with --rpath and without our own ld wrapper. It was ok.
Note that gcc doesn't actually need an RPATH as it only depends on system libraries just libm and libc but if I read the wrapper correctly it'll always put at least $ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64 in there, so the mystery is now, why not here?

[boegel]

@bartoldeman Which EasyBuild version were you using there? Does that GCC easyblock that was used include the changes made in easybuilders/easybuild-easyblocks#2711?

[boegel]

One more clue: some libraries in the GCCcore/11.3.0 installation for which the sanity check fails actually do have a proper RPATH section, for example libcc1.so.0.0.0:

$ readelf -d /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib64/libcc1.so.0.0.0 | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib64:$ORIGIN:$ORIGIN/../lib:$ORIGIN/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib/gcc/x86_64-pc-linux-gnu/11.3.0:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib/gcc/x86_64-pc-linux-gnu/11.3.0/../../../../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/lib/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/lib/../lib64:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib/gcc/x86_64-pc-linux-gnu/11.3.0/../../..:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/lib:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/lib:/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib/../lib64]
$
$ readelf -d /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/intel/cascadelake/software/GCCcore/11.3.0/lib64/libgcc_s.so.1 | grep RPATH
$ 

If found libcc1.so.0.0.0 by looking at the rpath_wrapper_ld.log that is produced by the EasyBuild RPATH wrapper script that wraps around ld (which resolves to /cvmfs/pilot.eessi-hpc.org/versions/2023.04/compat/linux/x86_64/usr/bin/ld).

That tells me that for some reason libgcc_s.so.1 is not getting linked with ld, and hence is escaping the RPATH wrapper script?
There's indeed no trace of libgcc_s in rpath_wrapper_ld.log...