Skip to content

Port over recent changes to easystack files page #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
orbsmiv merged 1 commit into from
Nov 24, 2022
Merged

Port over recent changes to easystack files page #67

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
199 changes: 132 additions & 67 deletions docs/easystack-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ install with Easybuild with *easystack files*.
[experimental feature][experimental_features]. **Some of the mentioned functionality may be subject to
change or be prone to errors.**

!!! note
Some breaking changes were made to the experimental easystack support in EasyBuild v4.7.0.

Easystack files must now use the `easyconfigs` top-level key to list *easyconfig filenames*,
as opposed to the `software` top-level key and separate subkeys like `version`, `versionsuffix` and
`toolchain` to specify aspects of an easyconfig file that were supported before.


## The basics {: #easystack_basics }

*Easystack files* describe an entire software stack, and can be used to
Expand All @@ -23,97 +31,154 @@ eb --easystack example.yaml --experimental
where `example.yaml` is the file with specifications that you just
created (more about this in the next section).


## Structure of an easystack file {: #easystack_structure }

Easystack files are written in [YAML
syntax](https://learnxinyminutes.com/docs/yaml).

General options, which should be applied to each software (for example
`robot`), are defined at the top of the file.
Essentially, an easystack file lists the easyconfig files you want to install, which are specified under the
`easyconfigs` key.

Afterwards, particular software specifications follow.
For example:

It is mandatory to specify basic software-related keywords: *software
name*, *toolchains* and *versions*.
``` yaml
easyconfigs:
- PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb
- OpenFOAM-v2206-foss-2022a.eb
```

- *software name*: Name of the software.
- *toolchains*: Names and versions of compiler [toolchains][toolchains].
- *versions*: Versions of software. If multiple entries are provided, EasyBuild will install all of
them. Can be in form of a list or consecutive line entries (see example).
!!! note
You must use '`-`' to list the easyconfigs in an easystack file.

**General structure of YAML-formatted easystack:**
In addition, you can specify additional configuration options via the `options` subkey
which will only apply to the installation of a particular easyconfig file.

For example:

``` yaml
software:
<software_name>:
toolchains:
<toolchain name and version>:
<software_version>:
versionsuffix: '<-example>'
easyconfigs:
- PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb:
options:
from-pr: 15924
debug: True
- Hypre-2.25.0-foss-2022a.eb:
- OpenFOAM-v2206-foss-2022a.eb:
options:
installpath: /my/custom/installpath
```

!!! note
You need to take care with some values in YAML, especially integers, booleans, etc.

If the specified value definitely must be a string value, you should use quotes (`'...'`) to avoid
that the YAML parser automatically converts the value to be of a specific type.

In other cases, the automatic conversion is useful, like for the `True` used above,
since `debug` is a boolean configuration option.

The configuration options that are valid for the `eb` command can be used (see `eb --help`),
but the `-` or `--` prefixes that are commonly used on the command line are omitted in easystack files.

Using the example easystack file above would be equivalent to running:

``` shell
eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --from-pr 15924 --debug
eb Hypre-2.25.0-foss-2022a.eb
eb OpenFOAM-v2206-foss-2022a.eb --installpath /my/custom/installpath
```

**Example of YAML-formatted easystack:**
!!! note
Whenever configuration options are *not* specified (as is the case for `Hypre` in the example easystack file above),
you are still allowed to use '`:`' after the easyconfig filename: there is no difference in behaviour in ending with or without '`:`'.

Specifying short options in an easystack file is allowed, for example:

``` yaml
software:
Bioconductor:
toolchains:
foss-2020a:
versions:
3.11:
EasyBuild:
toolchains:
SYSTEM:
versions: [4.3.1]
GROMACS:
toolchains:
foss-2020a:
versions:
2020.1:
2020.3:
fosscuda-2020a:
versions: [2020.1]
OpenFOAM:
toolchains:
foss-2020a:
versions: [8, v2006]
R:
toolchains:
foss-2020a:
versions: [4.0.0]
easyconfigs:
- OpenFOAM-v2206-foss-2022a.eb:
options:
D: True
```

To install the software specified in this *easystack file* named
`myeasystack.yaml`, run:
This is not recommended however, as short options are more difficult to interpret by humans.


## Combining command line options with options in an easystack file {: #easystack_combining_options }

When building software with an easystack file, you can still add additional options on the command line as well.
These apply to *all* items in the easystack file. For example, if you have an easystack file named
`my_easystack.yaml`

``` yaml
easyconfigs:
- PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb:
options:
from-pr: 15924
debug: True
- OpenFOAM-v2206-foss-2022a.eb:
```

and you run with

``` shell
eb --experimental --easystack my_easystack.yaml --dry-run
```

this will have the same effect as running

``` shell
eb --easystack myeasystack.yaml
eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --from-pr 15924 --debug
eb OpenFOAM-v2206-foss-2022a.eb --dry-run --installpath /my/custom/installpath
```

This is equivalent to running:
Note that options specified on the command line are placed *before* the easyconfig-specific options in the easystack file.
EasyBuild will always respect the argument that was put *last*.

For example:

``` shell
eb Bioconductor-3.11-foss-2020a.eb EasyBuild-4.3.1-SYSTEM.eb GROMACS-2020.1-foss-2020a.eb GROMACS-2020.3-foss-2020a.eb GROMACS-2020.1-fosscuda-2020a.eb OpenFOAM-8-foss-2020a.eb OpenFOAM-v2006-foss-2020a.eb R-4.0.0-foss-2020a.eb
eb PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb --dry-run --disable-dry-run
```

will effectively run *without* enabling dry run mode, since `--disable-dry-run` is specified after `--dry-run`.

Since easyconfig-specific configuration options specified in the easystack file are put *last*,
they take priority over the the ones on the command line, if the same configuration option is specified in both.

For example, running:

``` shell
eb --experimental --easystack my_easystack.yaml --disable-debug
```

will effectively cause `PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb` to be installed with debug logging enabled,
while `OpenFOAM-v2206-foss-2022a.eb` will be effectively installed *without* debug logging.

## To be developed

Optionally, more advanced keywords can be specified:
*easybuild_version*, *robot*, *from_pr*, *versionsuffix*,
*include-labels*, *exclude-labels*.

- *easybuild_version:* if present, EasyBuild will check if the
easystack file was intended for the current version of EasyBuild.
- *robot:* enables dependency resolution; see
[Using the EasyBuild command line][using_the_easybuild_command_line] for
more details.
- *from_pr:* easyconfig files that are added or modified by a
particular pull request to the easybuild-easyconfigs GitHub
repository can be used (regardless of whether the pull request is
merged or not). (see [Integration with GitHub][integration_with_github] for more details).
- *versionsuffix:* additional suffix for software version (placed
after toolchain name)
- *include-labels:* only include this software when EasyBuild is
configured with one of the specified labels
- *exclude-labels:* **do not** include this software when EasyBuild is
configured with one of the specified labels
In the future, we are planning to support additional also global options specified in the easystack file. For example:

``` yaml
options:
robot: True
easyconfigs:
- PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb
- OpenFOAM-v2206-foss-2022a.eb
```

would installed both `PyTorch-1.12.0-foss-2022a-CUDA-11.7.0.eb` and `OpenFOAM-v2206-foss-2022a.eb` using `--robot`
(see [issue #4105](https://github.com/easybuilders/easybuild-framework/issues/4105)).

Additionally, we plan to support specifying for which EasyBuild version an easystack file was intended,
which can be helpful in more accurately recreating a certain software stack
(see [issue #4106](https://github.com/easybuilders/easybuild-framework/issues/4106)).

In the future, the `--easystack` option will probably be dropped, and EasyBuild will automatically detect
the use of easystack files (see [issue #4104](https://github.com/easybuilders/easybuild-framework/issues/4104)).

Finally, we plan to support specifying *labels*, which would make it more easy to install only a certain subset of
the items listed in an easystack file. For example, by labelling all GPU-capable software with a `gpu` label,
one could easily choose to *not* build anything labeled `gpu` on a CPU node
(see [issue #3512](https://github.com/easybuilders/easybuild-framework/issues/3512)).