also try and resolve dependencies using subtoolchains

[boegel]

Feature proposal by @berndmohr:

For resolving dependencies specified in a goolf toolchain build, we could also try and resolve them using a gompi toolchain and maybe even a GCC toolchain.

This would allow that tools like CMake are built with a minimal toolchain (e.g. CMake-2.8.4-GCC-4.7.2), and still can be used in a goolf easyconfig as build dependency.

This is less strict than what is being done now (i.e. only use the specified toolchain, either the one in toolchain or in the custom toolchain in the dependency specification).

I'm confident @besserox would support this feature proposal too, let's see what @stdweird thinks about it. :-)

[besserox]

I support this request!

@boegel, I believe you meant CMake-2.8.4-GCC-4.7.2 instead of CMake-2.8.4-goolf-1.4.10.

[boegel]

@besserox: Thanks for the correction, fixed in the description.

[fgeorgatos]

If things go into this direction, eventually there may be a need to define classes of toolchains such as:

• compc
• compc_compf
• compc_compf_mpi
• compc_compf_mpi_la_fft
  and so on (for simplicity, I collapsed c++ to c above; this could well be represented via a Python list, too).

The reason why certain build alternatives (--try-*) are very simple right now is, that most easyconfigs are assumed against the later category, therefor little precaution is needed to try most alternative toolchains and, dependencies are handled in a straight-forward manner and opening request #450 is feasible, at all:
"the -try-X command line options should cooperate with --robot" # ie. recursively resolve deps for new toolchains.

Once/if subtoolchains become the norm though, spotting the correct replacement toolchains won't be as automated piece of cake work as it is now (remember the quick transitions from goalf to goolf and ictce/4* to ictce/5*?).

your offers?

[boegel]

@fgeorgatos: Some points you mentioned are relevant for boegel/easybuild-wiki#11, please get them included there.
Making EasyBuild look for dependencies for various (sub)toolchains is separate from making --try-toolchain --robot work correctly when minimal toolchains are used imho.

I'm not sure if we even need dedicated classes for subtoolchains, we can probably tell what we need to know from the toolchain definitions in the framework (e.g. whether an MPI library is included, wheter a Fortran compiler is available, etc.). Let's not try and encode the same information in two places in different ways...

[boegel]

From the top of my head, without trying it at all: the toolchain definitions in the framework need to be made aware of which subtoolchains 'apply' to them, and then some magic needs to happen to figure out the particular version to look for. #422 maybe also be relevant (but I don't think it's needed, it would probably even complicate things)

[geimer]

I support this request, too -- and would even go as far as trying the dummy toolchain as a last resort if everything else failed (e.g. for statically linked debuggers or helper packages which solely consist of shell scripts, etc.).

[boegel]

Using a module installed with dummy as a last resort makes a lot of sense yes, since that's always a subtoolchain. In fact, I see no reason why not to consider dummy.

[ocaisa]

I struggle to see how you can get this to work without having some kind of equivalency classes defined between toolchains (somewhat like what Fotis was describing but not so fine-grained) and well defined inheritance in a toolchain (gmpolf inherits gompi inherits GCC) so that searching back down the toolchain hierarchy is straightforward.

An added plus would be that you could warn people when --try-toolchain invokes a compiler or MPI switch (GCC -> icc for example) since this can have a big, but silent, impact on many applications.

Not to worry, this is what I plan to work on next week (along with a new HMNS that would leverage hierarchically defined toolchains)!

[boegel]

@ocaisa: that's indeed indispensable, but it could be as simple as adding something like this in the toolchain definitions that are now already in the framework:

# in goolf
subtoolchains = ['gompi', 'GCC']

dummy would then be an implied subtoolchains for all others

[ocaisa]

That might be enough...but then which version of gompi matches the current toolchain? If it's not listed somehow in the easyconfig, wouldn't it be a struggle to figure out uniquely?

[boegel]

The versioning can be tricky, yes. But we can do that best effort...

For several toolchains (e.g. goolf), subtoolchains (e.g. gompi) that are not individual toolchain components (e.g. GCC) have the same version as their parent toolchain.

So, if a subtoolchain is included as an actual toolchain component, use the component version, if not, use the toolchain version itself.

For example, when using goolf/1.4.10, consider gompi/1.4.10 (use toolchain version), and GCC/4.7.2 (use GCC component version), and eventually also dummy.

That way, several toolchains we're using already will work with this procedure, and others can be made compatible with this in future versions of the toolchain.