Easybuild with rpath support enabled is really slow when building extensions. Easybuild launches many processes

[pescobar]

When compiling my old Perl easyconfig which includes more than 400 extensions enabling the rpath support I have noticed that the build takes really long.

While monitoring the build with htop I have noticed that easybuild is launching a lot of processes when building the extensions. See the attached screenshot. Is this expected?

[boegel]

@pescobar For every compilation or linking command, the RPATH wrapper will kick in which means you don't get a single process (e.g. the compiler), but 3 processes (the wrapper, a Python script called by the wrapper, and the compiler)
The wrapper and the Python script are very light weight though, and some benchmarking I did didn't show huge overhead (e.g. for OpenFOAM the whole build slows down with less than 10%).

The output you showed seems to suggest something is very wrong though, since they're not all running the same wrapper script, but different ones (the tmp* subdir is different every time)...

This looks weird to me, since only a single gcc wrapper script should be generated/used for every extension being installed (+ one for Perl itself), and the extensions are installed sequentially, not in parallel.
I think you are hitting a bug that only pops up when building extensions.

One likely possibility is that the gcc wrapper script used for building Perl itself is being wrapped with an additional wrapper script when the 1st extension is being built, which then gets wrapped again when the 2nd extension is being installed, and so on.

So, when you're building the 400th extension, there's actually 400 wrapper scripts being called before the actual compiler gets called... That's of course not the intention. ;-)

This probably happens because the prepare_step is called again for every extension, which installs an additional wrapper script that wraps around the current gcc 'command' that is first in $PATH (which may already be an RPATH wrapper script).

There is a check in place to avoid that the wrapper script calls itself (since that would make it a fork bomb), but it doesn't check that it's wrapping the real compiler and not a wrapper script...

[boegel]

fixed in #2022

[boegel]

follow-up in #2022