diff --git a/docs/easystack-files.md b/docs/easystack-files.md index 3a00dbccf5..38db5d26f3 100644 --- a/docs/easystack-files.md +++ b/docs/easystack-files.md @@ -7,6 +7,14 @@ install with Easybuild with *easystack files*. [experimental feature][experimental_features]. **Some of the mentioned functionality may be subject to change or be prone to errors.** +!!! note + Some breaking changes were made to the experimental easystack support in EasyBuild v4.7.0. + + Easystack files must now use the `easyconfigs` top-level key to list *easyconfig filenames*, + as opposed to the `software` top-level key and separate subkeys like `version`, `versionsuffix` and + `toolchain` to specify aspects of an easyconfig file that were supported before. + + ## The basics {: #easystack_basics } *Easystack files* describe an entire software stack, and can be used to @@ -23,97 +31,154 @@ eb --easystack example.yaml --experimental where `example.yaml` is the file with specifications that you just created (more about this in the next section). + ## Structure of an easystack file {: #easystack_structure } Easystack files are written in [YAML syntax](https://learnxinyminutes.com/docs/yaml). -General options, which should be applied to each software (for example -`robot`), are defined at the top of the file. +Essentially, an easystack file lists the easyconfig files you want to install, which are specified under the +`easyconfigs` key. -Afterwards, particular software specifications follow. +For example: -It is mandatory to specify basic software-related keywords: *software -name*, *toolchains* and *versions*. +``` yaml +easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb + - OpenFOAM-v2206-foss-2022a.eb +``` -- *software name*: Name of the software. -- *toolchains*: Names and versions of compiler [toolchains][toolchains]. -- *versions*: Versions of software. If multiple entries are provided, EasyBuild will install all of - them. Can be in form of a list or consecutive line entries (see example). +!!! note + You must use '`-`' to list the easyconfigs in an easystack file. -**General structure of YAML-formatted easystack:** +In addition, you can specify additional configuration options via the `options` subkey +which will only apply to the installation of a particular easyconfig file. + +For example: ``` yaml -software: - : - toolchains: - : - : - versionsuffix: '<-example>' +easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb: + options: + from-pr: 15924 + debug: True + - Hypre-2.25.0-foss-2022a.eb: + - OpenFOAM-v2206-foss-2022a.eb: + options: + installpath: /my/custom/installpath +``` + +!!! note + You need to take care with some values in YAML, especially integers, booleans, etc. + + If the specified value definitely must be a string value, you should use quotes (`'...'`) to avoid + that the YAML parser automatically converts the value to be of a specific type. + + In other cases, the automatic conversion is useful, like for the `True` used above, + since `debug` is a boolean configuration option. + +The configuration options that are valid for the `eb` command can be used (see `eb --help`), +but the `-` or `--` prefixes that are commonly used on the command line are omitted in easystack files. + +Using the example easystack file above would be equivalent to running: + +``` shell +eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --from-pr 15924 --debug +eb Hypre-2.25.0-foss-2022a.eb +eb OpenFOAM-v2206-foss-2022a.eb --installpath /my/custom/installpath ``` -**Example of YAML-formatted easystack:** +!!! note + Whenever configuration options are *not* specified (as is the case for `Hypre` in the example easystack file above), + you are still allowed to use '`:`' after the easyconfig filename: there is no difference in behaviour in ending with or without '`:`'. + +Specifying short options in an easystack file is allowed, for example: ``` yaml -software: - Bioconductor: - toolchains: - foss-2020a: - versions: - 3.11: - EasyBuild: - toolchains: - SYSTEM: - versions: [4.3.1] - GROMACS: - toolchains: - foss-2020a: - versions: - 2020.1: - 2020.3: - fosscuda-2020a: - versions: [2020.1] - OpenFOAM: - toolchains: - foss-2020a: - versions: [8, v2006] - R: - toolchains: - foss-2020a: - versions: [4.0.0] +easyconfigs: + - OpenFOAM-v2206-foss-2022a.eb: + options: + D: True ``` -To install the software specified in this *easystack file* named -`myeasystack.yaml`, run: +This is not recommended however, as short options are more difficult to interpret by humans. + + +## Combining command line options with options in an easystack file {: #easystack_combining_options } + +When building software with an easystack file, you can still add additional options on the command line as well. +These apply to *all* items in the easystack file. For example, if you have an easystack file named +`my_easystack.yaml` + +``` yaml +easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb: + options: + from-pr: 15924 + debug: True + - OpenFOAM-v2206-foss-2022a.eb: +``` + +and you run with + +``` shell +eb --experimental --easystack my_easystack.yaml --dry-run +``` + +this will have the same effect as running ``` shell -eb --easystack myeasystack.yaml +eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --from-pr 15924 --debug +eb OpenFOAM-v2206-foss-2022a.eb --dry-run --installpath /my/custom/installpath ``` -This is equivalent to running: +Note that options specified on the command line are placed *before* the easyconfig-specific options in the easystack file. +EasyBuild will always respect the argument that was put *last*. + +For example: ``` shell -eb Bioconductor-3.11-foss-2020a.eb EasyBuild-4.3.1-SYSTEM.eb GROMACS-2020.1-foss-2020a.eb GROMACS-2020.3-foss-2020a.eb GROMACS-2020.1-fosscuda-2020a.eb OpenFOAM-8-foss-2020a.eb OpenFOAM-v2006-foss-2020a.eb R-4.0.0-foss-2020a.eb +eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --disable-dry-run ``` +will effectively run *without* enabling dry run mode, since `--disable-dry-run` is specified after `--dry-run`. + +Since easyconfig-specific configuration options specified in the easystack file are put *last*, +they take priority over the the ones on the command line, if the same configuration option is specified in both. + +For example, running: + +``` shell +eb --experimental --easystack my_easystack.yaml --disable-debug +``` + +will effectively cause `PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb` to be installed with debug logging enabled, +while `OpenFOAM-v2206-foss-2022a.eb` will be effectively installed *without* debug logging. + ## To be developed -Optionally, more advanced keywords can be specified: -*easybuild_version*, *robot*, *from_pr*, *versionsuffix*, -*include-labels*, *exclude-labels*. - -- *easybuild_version:* if present, EasyBuild will check if the - easystack file was intended for the current version of EasyBuild. -- *robot:* enables dependency resolution; see - [Using the EasyBuild command line][using_the_easybuild_command_line] for - more details. -- *from_pr:* easyconfig files that are added or modified by a - particular pull request to the easybuild-easyconfigs GitHub - repository can be used (regardless of whether the pull request is - merged or not). (see [Integration with GitHub][integration_with_github] for more details). -- *versionsuffix:* additional suffix for software version (placed - after toolchain name) -- *include-labels:* only include this software when EasyBuild is - configured with one of the specified labels -- *exclude-labels:* **do not** include this software when EasyBuild is - configured with one of the specified labels +In the future, we are planning to support additional also global options specified in the easystack file. For example: + +``` yaml +options: + robot: True +easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb + - OpenFOAM-v2206-foss-2022a.eb +``` + +would installed both `PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb` and `OpenFOAM-v2206-foss-2022a.eb` using `--robot` +(see [issue #4105](https://github.com/easybuilders/easybuild-framework/issues/4105)). + +Additionally, we plan to support specifying for which EasyBuild version an easystack file was intended, +which can be helpful in more accurately recreating a certain software stack +(see [issue #4106](https://github.com/easybuilders/easybuild-framework/issues/4106)). + +In the future, the `--easystack` option will probably be dropped, and EasyBuild will automatically detect +the use of easystack files (see [issue #4104](https://github.com/easybuilders/easybuild-framework/issues/4104)). + +Finally, we plan to support specifying *labels*, which would make it more easy to install only a certain subset of +the items listed in an easystack file. For example, by labelling all GPU-capable software with a `gpu` label, +one could easily choose to *not* build anything labeled `gpu` on a CPU node +(see [issue #3512](https://github.com/easybuilders/easybuild-framework/issues/3512)). +