respect start dir defined in extensions

[lexming]

Currently extensions ignore start_dir because ExtensionEasyBlock._set_start_dir forcefully overwrites self.cfg['start_dir'] of the extension.

This PR enables extensions to define a start_dir. The resulting start dir will be on of the following by preference from top to bottom:

1. if self.cfg['start_dir'] is an absolute path and exists it will be used as is
2. combine self.start_dir with self.cfg['start_dir']
3. combine self.ext_dir with self.cfg['start_dir']

Motivation: Some Julia packages are distributed in a subfolder of other Julia packages. For instance, SnoopPrecompile and SnoopCompileCore are found inside SnoopCompile

[boegel]

We have a bunch of easyconfigs that specify start_dir for extensions, so I'm a bit puzzled how those can have worked.
Maybe a particular easyblock does correctly pick up on start_dir if specified for an extension?

In any case, we should check whether these still work correctly on top of the changes proposed in this PR:

isoCirc-1.0.4-foss-2020b.eb => ext isoCirc specifies 'start_dir': isoCirc_pipeline
immunedeconv-2.0.2-foss-2020a-R-4.0.0.eb => ext MCPcounter specifies 'start_dir': Source
NiBabel-2.1.0-intel-2017a-Python-2.7.13.eb => ext pydicom specifies 'start_dir': source
NiBabel-2.2.1-intel-2018a-Python-3.6.4.eb => ext pydicom specifies 'start_dir': source
GenomeWorks-2021.02.2-fosscuda-2020b.eb => ext genomeworks specifies 'start_dir': pygenomeworks
TensorRT-4.0.1.6-fosscuda-2018b-Python-2.7.15.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.12.0-foss-2018b-Python-3.6.6.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.13.1-foss-2019a-Python-3.7.2.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.14.0-foss-2019a-Python-3.7.2.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.12.0-fosscuda-2018b-Python-2.7.15.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.14.0-fosscuda-2019a-Python-3.7.2.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.12.0-fosscuda-2018b-Python-3.6.6.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.13.1-fosscuda-2019a-Python-3.7.2.eb => ext protobuf-python specifies 'start_dir': python
TensorFlow-1.13.1-foss-2018b-Python-3.6.6.eb => ext protobuf-python specifies 'start_dir': python
Spark-2.2.0-intel-2017b-Hadoop-2.6-Java-1.8.0_152-Python-3.6.3.eb => ext pyspark specifies 'start_dir': python
Spark-2.4.0-intel-2018b-Hadoop-2.7-Java-1.8-Python-3.6.6.eb => ext pyspark specifies 'start_dir': python
Arrow-0.7.1-intel-2017b-Python-3.6.3.eb => ext pyarrow specifies 'start_dir': python
AutoDock-Vina-1.2.3-foss-2021a.eb => ext vina specifies 'start_dir': build/python
AutoDock-Vina-1.2.3-foss-2021b.eb => ext vina specifies 'start_dir': build/python
QCG-PilotJob-0.12.3-foss-2021a.eb => ext qcg specifies 'start_dir': components/core
QCG-PilotJob-0.12.3-foss-2021a.eb => ext qcg specifies 'start_dir': components/cmds
QCG-PilotJob-0.12.3-foss-2021a.eb => ext qcg specifies 'start_dir': components/executor_api
QCG-PilotJob-0.13.1-foss-2022a.eb => ext qcg specifies 'start_dir': components/core
QCG-PilotJob-0.13.1-foss-2022a.eb => ext qcg specifies 'start_dir': components/cmds
QCG-PilotJob-0.13.1-foss-2022a.eb => ext qcg specifies 'start_dir': components/executor_api

Python code used to compose this list:

import glob
import os

from easybuild.framework.easyconfig.tools import parse_easyconfigs
from easybuild.tools.options import set_up_configuration

os.environ['EASYBUILD_IGNORE_OSDEPS'] = '1'
set_up_configuration()

ec_files = glob.glob('easybuild/easyconfigs/*/*/*.eb')
print("Found %d easyconfigs!" % len(ec_files))

ecs, _ = parse_easyconfigs([(x, False) for x in ec_files])
print("Parsed %d easyconfigs!" % len(ecs))
for ec in ecs:
    ec = ec['ec']
    ec_fn = os.path.basename(ec.path)
    exts_list = ec['exts_list']
    if exts_list:
        for ext in exts_list:
            if len(ext) >= 3:
                ext_options = ext[2]
                if 'start_dir' in ext_options:
                    print("%s => ext %s specifies 'start_dir': %s" % (ec_fn, ext[0], ext_options['start_dir']))

[lexming]

@boegel thanks for listing those packages, they work with start_dir in a different manner than what I imagined. The derivative easyblocks do not use start_dir directly. The only involvement of start_dir is via ExtensionEasyBlock.run, which will change directory to start_dir after _set_start_dir. Then all following commands to install the extension are executed from there.

In that case, my problem is not that self.cfg['start_dir'] is overwritten, but that self.start_dir can stay as a relative path after ExtensionEasyBlock._set_start_dir. Because then it cannot be reused at later stages of the install.

Closing this PR as it is not a good solution, I'll make a new one if needed.

[lexming]

Follow-up in #4196