diff --git a/docs/Easystack-files.rst b/docs/Easystack-files.rst index 9c46e611..fc1a30e8 100644 --- a/docs/Easystack-files.rst +++ b/docs/Easystack-files.rst @@ -7,6 +7,13 @@ This documentation covers aspects of specifying a software stack to install with **Note: this is an** :ref:`experimental feature `. **Some of the mentioned functionality may be subject to change or be prone to errors.** +.. note:: + Some breaking changes were made to the experimental easystack support in EasyBuild v4.7.0. + + Easystack files must now use the ``easyconfigs`` top-level key to list *easyconfig filenames*, + as opposed to the ``software`` top-level key and separate subkeys like ``version``, ``versionsuffix`` and + ``toolchain`` to specify aspects of an easyconfig file that were supported before. + .. contents:: :depth: 3 :backlinks: none @@ -32,6 +39,7 @@ To build software with *Easystack*, type: where ``example.yaml`` is the file with specifications that you just created (more about this in the next section). + .. _easystack_structure: Structure of an easystack file @@ -39,82 +47,153 @@ Structure of an easystack file Easystack files are written in `YAML syntax `_. -General options, which should be applied to each software (for example ``robot``), are defined at the top of the file. +Essentially, an easystack file lists the easyconfig files you want to install, which are specified under the +`easyconfigs` key. + +For example: + +.. code:: yaml -Afterwards, particular software specifications follow. + easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb + - OpenFOAM-v2206-foss-2022a.eb -It is mandatory to specify basic software-related keywords: *software name*, *toolchains* and *versions*. +.. note:: + You must use '``-``' to list the easyconfigs in an easystack file. -- *software name*: Name of the software. -- *toolchains*: Names and versions of compiler :ref:`toolchains`. -- *versions*: Versions of software. If multiple entries are provided, EasyBuild will install all of them. -Can be in form of a list or consecutive line entries (see example). +In addition, you can specify additional configuration options via the ``options`` subkey +which will only apply to the installation of a particular easyconfig file. -**General structure of YAML-formatted easystack:** +For example: .. code:: yaml - software: - : - toolchains: - : - : - versionsuffix: '<-example>' + easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb: + options: + from-pr: 15924 + debug: True + - Hypre-2.25.0-foss-2022a.eb: + - OpenFOAM-v2206-foss-2022a.eb: + options: + installpath: /my/custom/installpath + +.. note:: + You need to take care with some values in YAML, especially integers, booleans, etc. + + If the specified value definitely must be a string value, you should use quotes (``'...'``) to avoid + that the YAML parser automatically converts the value to be of a specific type. + + In other cases, the automatic conversion is useful, like for the ``True`` used above, + since ``debug`` is a boolean configuration option. + +The configuration options that are valid for the ``eb`` command can be used (see ``eb --help``), +but the ``-`` or ``--`` prefixes that are commonly used on the command line are omitted in easystack files. + +Using the example easystack file above would be equivalent to running: + +.. code:: + + eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --from-pr 15924 --debug + eb Hypre-2.25.0-foss-2022a.eb + eb OpenFOAM-v2206-foss-2022a.eb --installpath /my/custom/installpath + +.. note:: + Whenever configuration options are *not* specified (as is the case for ``Hypre`` in the example easystack file above), + you are still allowed to use '``:``' after the easyconfig filename: there is no difference in behaviour in ending with or without '``:``'. + +Specifying short options in an easystack file is allowed, for example: + +.. code:: yaml + + easyconfigs: + - OpenFOAM-v2206-foss-2022a.eb: + options: + D: True + +This is not recommended however, as short options are more difficult to interpret by humans. + + +.. _easystack_combining_options: + +Combining command line options with options in an easystack file +---------------------------------------------------------------- -**Example of YAML-formatted easystack:** +When building software with an easystack file, you can still add additional options on the command line as well. +These apply to *all* items in the easystack file. For example, if you have an easystack file named +``my_easystack.yaml`` .. code:: yaml - software: - Bioconductor: - toolchains: - foss-2020a: - versions: - 3.11: - EasyBuild: - toolchains: - SYSTEM: - versions: [4.3.1] - GROMACS: - toolchains: - foss-2020a: - versions: - 2020.1: - 2020.3: - fosscuda-2020a: - versions: [2020.1] - OpenFOAM: - toolchains: - foss-2020a: - versions: [8, v2006] - R: - toolchains: - foss-2020a: - versions: [4.0.0] - -To install the software specified in this *easystack file* named '``myeasystack.yaml``', run: + easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb: + options: + from-pr: 15924 + debug: True + - OpenFOAM-v2206-foss-2022a.eb: + +and you run with + +.. code:: + + eb --experimental --easystack my_easystack.yaml --dry-run + +this will have the same effect as running .. code:: - eb --easystack myeasystack.yaml + eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --from-pr 15924 --debug + eb OpenFOAM-v2206-foss-2022a.eb --dry-run --installpath /my/custom/installpath + +Note that options specified on the command line are placed *before* the easyconfig-specific options in the easystack file. +EasyBuild will always respect the argument that was put *last*. + +For example: + +.. code:: + + eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --disable-dry-run + +will effectively run *without* enabling dry run mode, since ``--disable-dry-run`` is specified after ``--dry-run``. -This is equivalent to running: +Since easyconfig-specific configuration options specified in the easystack file are put *last*, +they take priority over the the ones on the command line, if the same configuration option is specified in both. + +For example, running: .. code:: - eb Bioconductor-3.11-foss-2020a.eb EasyBuild-4.3.1-SYSTEM.eb GROMACS-2020.1-foss-2020a.eb GROMACS-2020.3-foss-2020a.eb GROMACS-2020.1-fosscuda-2020a.eb OpenFOAM-8-foss-2020a.eb OpenFOAM-v2006-foss-2020a.eb R-4.0.0-foss-2020a.eb + eb --experimental --easystack my_easystack.yaml --disable-debug + +will effectively cause ``PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb`` to be installed with debug logging enabled, +while ``OpenFOAM-v2206-foss-2022a.eb`` will be effectively installed *without* debug logging. + To be developed --------------- -Optionally, more advanced keywords can be specified: *easybuild_version*, *robot*, *from_pr*, *versionsuffix*, *include-labels*, *exclude-labels*. - -- *easybuild_version:* if present, EasyBuild will check if the easystack file was intended for the current version of EasyBuild. -- *robot:* enables dependency resolution; see :ref:`Using_the_EasyBuild_command_line` for more details. -- *from_pr:* easyconfig files that are added or modified by a - particular pull request to the easybuild-easyconfigs GitHub repository - can be used (regardless of whether the pull request is merged or not). - (see :ref:`Integration_with_GitHub` for more details). -- *versionsuffix:* additional suffix for software version (placed after toolchain name) -- *include-labels:* only include this software when EasyBuild is configured with one of the specified labels -- *exclude-labels:* **do not** include this software when EasyBuild is configured with one of the specified labels +In the future, we are planning to support additional also global options specified in the easystack file. For example: + +.. code:: yaml + + options: + robot: True + + easyconfigs: + - PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb + - OpenFOAM-v2206-foss-2022a.eb + +would installed both ``PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb`` and ``OpenFOAM-v2206-foss-2022a.eb`` using ``--robot`` +(see `issue #4105 `_). + +Additionally, we plan to support specifying for which EasyBuild version an easystack file was intended, +which can be helpful in more accurately recreating a certain software stack +(see `issue #4106 `_). + +In the future, the ``--easystack`` option will probably be dropped, and EasyBuild will automatically detect +the use of easystack files (see `issue #4104 `_). + +Finally, we plan to support specifying *labels*, which would make it more easy to install only a certain subset of +the items listed in an easystack file. For example, by labelling all GPU-capable software with a ``gpu`` label, +one could easily choose to *not* build anything labeled ``gpu`` on a CPU node +(see `issue #3512 `_).