Skip to content

{bio}[foss/2022a] AlphaFold v2.3.1, HH-suite v3.3.0, Kalign v3.3.5, OpenMM 8.0.0 w/ Python 3.10.4 #17604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
boegel merged 13 commits into from
Apr 14, 2023
Merged

{bio}[foss/2022a] AlphaFold v2.3.1, HH-suite v3.3.0, Kalign v3.3.5, OpenMM 8.0.0 w/ Python 3.10.4 #17604

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
5ca7aa3
Add AlphaFold and deps
maxim-masterov Mar 27, 2023
ceb5300
Update to AlphaFold-2.3.1
maxim-masterov Mar 27, 2023
3e57144
Add missing binutils
maxim-masterov Mar 27, 2023
3e6e9bb
Add OpenMM-7.7.0
maxim-masterov Mar 28, 2023
25466dc
Switch to OpenMM-8.0.0
maxim-masterov Mar 28, 2023
d86a6c9
Add CUDA to OpenMM
maxim-masterov Apr 3, 2023
956625c
Fix comment
maxim-masterov Apr 3, 2023
b42386e
Add patch to resolve tree-vectorize issue
maxim-masterov Apr 4, 2023
2b69ad1
Fix style
maxim-masterov Apr 4, 2023
b7c176b
Add CUDA versionsuffix to OpenMM
maxim-masterov Apr 11, 2023
02e6688
Merge branch 'develop' of https://github.com/easybuilders/easybuild-e…
maxim-masterov Apr 11, 2023
8af4143
Fix style
maxim-masterov Apr 11, 2023
a1f307a
update jax dependency of AlphaFold 2.3.1 to 0.3.25
boegel Apr 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 169 additions & 0 deletions easybuild/easyconfigs/a/AlphaFold/AlphaFold-2.3.1-foss-2022a-CUDA-11.7.0.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
easyblock = 'PythonBundle'

name = 'AlphaFold'
version = '2.3.1'
versionsuffix = '-CUDA-%(cudaver)s'

homepage = 'https://deepmind.com/research/case-studies/alphafold'
description = "AlphaFold can predict protein structures with atomic accuracy even where no similar structure is known"

toolchain = {'name': 'foss', 'version': '2022a'}

builddependencies = [
# required for installing dm-tree
('Bazel', '5.1.1'),
('CMake', '3.23.1'),
]

dependencies = [
('Python', '3.10.4'),
('CUDA', '11.7.0', '', SYSTEM),
('SciPy-bundle', '2022.05'),
('PyYAML', '6.0'),
('TensorFlow', '2.11.0', versionsuffix),
('Biopython', '1.79'),
('HH-suite', '3.3.0'),
('HMMER', '3.3.2'),
('Kalign', '3.3.5'),
('jax', '0.3.14', versionsuffix), # also provides absl-py
('UCX-CUDA', '1.12.1', versionsuffix),
('cuDNN', '8.4.1.50', versionsuffix, SYSTEM),
('NCCL', '2.12.12', versionsuffix),
('OpenMM', '8.0.0'),
Copy link
Member

@boegel boegel Mar 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maxim-masterov Doesn't OpenMM need to be GPU-capable for AlphaFold?

Copy link
Collaborator Author

@maxim-masterov maxim-masterov Mar 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel I'm checking it. I didn't include OpenMM compiled with CUDA support as previous easyconfigs didn't use it. I'm trying to build it now, but as you mentioned earlier, I hit some errors. At the moment it looks like CMake picks nvcc from /usr/bin, instead of $EBCUDAROOT/bin

Copy link
Collaborator Author

@maxim-masterov maxim-masterov Apr 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@boegel I've added CUDA to OpenMM, It builds fine and passes all tests after specifying the OPENMM_CUDA_COMPILER variable. Without this variable, OpenMM tries to use /usr/local/cuda/bin/nvcc, instead of ${EBROOTCUDA}/bin/nvcc. The only test that fails with this variable is CudaCompiler, which requires OPENMM_CUDA_COMPILER not to be set, so I've excluded it.

Copy link
Contributor

@bedroge bedroge Apr 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave this a try, as we also had some requests for this new AlphaFold version, but I do get an internal compiler error as well:

/dev/shm/f115372/OpenMM/8.0.0/foss-2022a/openmm-8.0.0/platforms/common/src/CommonKernels.cpp: In member function void OpenMM::CommonCalcGayBerneForceKernel::sortAtoms():
/dev/shm/f115372/OpenMM/8.0.0/foss-2022a/openmm-8.0.0/platforms/common/src/CommonKernels.cpp:5055:6: internal compiler error: in vect_get_vec_defs_for_operand, at tree-vect-stmts.c:1450
 5055 | void CommonCalcGayBerneForceKernel::sortAtoms() {
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0x69b359 vect_get_vec_defs_for_operand(vec_info*, _stmt_vec_info*, unsigned int, tree_node*, vec<tree_node*, va_heap, vl_ptr>*, tree_node*)
        ../../gcc/tree-vect-stmts.c:1450
0xf42df4 vect_build_gather_load_calls
        ../../gcc/tree-vect-stmts.c:2728
0xf42df4 vectorizable_load
        ../../gcc/tree-vect-stmts.c:8718
0xf4bca0 vect_transform_stmt(vec_info*, _stmt_vec_info*, gimple_stmt_iterator*, _slp_tree*, _slp_instance*)
        ../../gcc/tree-vect-stmts.c:10922
0xf4fa41 vect_transform_loop_stmt
        ../../gcc/tree-vect-loop.c:9254
0xf6740d vect_transform_loop(_loop_vec_info*, gimple*)
        ../../gcc/tree-vect-loop.c:9690
0xf9059c try_vectorize_loop_1
        ../../gcc/tree-vectorizer.c:1104
0xf91181 vectorize_loops()
        ../../gcc/tree-vectorizer.c:1243
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.

Copy link
Collaborator Author

@maxim-masterov maxim-masterov Apr 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

odd, what optarch flags do you use? and what machine are you building on?

Copy link
Contributor

@bedroge bedroge Apr 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't set any optarch, so it was using -march=native, and this was on an AMD EPYC 7763. Also tried OpenMM 7.7.0 with CUDA and foss 2022a, but that resulted in the same error.

Copy link
Collaborator Author

@maxim-masterov maxim-masterov Apr 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built succesfully on Intel Platinum 8360Y with optarch=Intel:O2 -march=core-avx2;GCC:O2 -mavx2 -mfma and NVIDIA A100. And commenting out CUDA allowed me to build on AMD EPYC 7H12 without GPUs on board.

Copy link
Collaborator Author

@maxim-masterov maxim-masterov Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was experimenting with compiler flags and managed to get the same "internal compiler" error. It appears when -march is used, which switches on the tree vectorization. Surprisingly, with -march=znver2 on our AMD CPUs all works fine, but with, e.g., -march=skylake-avx512 on our Intel CPUs I get this error.

There are two ways to solve it, except patching the compiler, which will require shrinking down the code to a small reproducible example and posting an issue on Bugzilla. The first is to compile OpenMM with -fno-tree-vectorize flag. The second is to patch platforms/common/src/CommonKernels.cpp file and add __attribute__((optimize("no-tree-vectorize"))) in front of the CommonCalcGayBerneForceKernel::sortAtoms() function definition. IMO, the second is better, as it affects only one function, instead of the whole source code.

Copy link
Contributor

@lexming lexming Apr 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirm that just removing -ftree-vectorize is enough to workaround this ICE. Using -march=native will only be an issue on those architectures affected by this bug.

On my side,I hit this ICE on our old Intel Broadwells with just AVX2. However, the build on our AMD EPYC 7282 (znver2) worked fine and those also just support AVX2.

So maybe we can collect a few known systems where this ICE triggers and update the comment OpenMM-8.0.0_add_no_tree_vectorize.patch to inform that this patch is not always needed.

]

# commit to use for downloading stereo_chemical_props.txt and copy to alphafold/common,
# see docker/Dockerfile in AlphaFold repository
local_scp_commit = '7102c6'

components = [
(name, version, {
'easyblock': 'PythonPackage',
'source_urls': [
'https://github.com/deepmind/alphafold/archive/refs/tags/',
'https://git.scicore.unibas.ch/schwede/openstructure/-/raw/%s/modules/mol/alg/src/' % local_scp_commit,
],
'sources': [
{
'download_filename': 'v%(version)s.tar.gz',
'filename': SOURCE_TAR_GZ,
},
{
'download_filename': 'stereo_chemical_props.txt',
'filename': 'stereo_chemical_props-%s.txt' % local_scp_commit,
'extract_cmd': "cp %s .",
},
],
'patches': [
'AlphaFold-2.0.0_fix-packages.patch',
'AlphaFold-2.3.0_data-dep-paths.patch',
'AlphaFold-2.0.0_n-cpu.patch',
'AlphaFold-2.1.0_fix-scp-path.patch',
'AlphaFold-2.0.1_setup_rm_tfcpu.patch',
'AlphaFold-2.3.1_use_openmm_7.7.0.patch',
],
'checksums': [
'1161b2609fa896b16399b900ec2b813e5a0b363fe4e2b26bd826953ba234736a', # v2.3.1.tar.gz
'24510899eeb49167cffedec8fa45363a4d08279c0c637a403b452f7d0ac09451', # stereo_chemical_props-7102c6.txt
'826d2d1a5d6ac52c51a60ba210e1947d5631a1e2d76f8815305b5d23f74458db', # AlphaFold-2.0.0_fix-packages.patch
'5cff3fc7104e020ef546d23cb4fb1b8d6517562783f055cc55fc65fe2b0248d0', # AlphaFold-2.3.0_data-dep-paths.patch
'dfda4dd5f9aba19fe2b6eb9a0ec583d12dcefdfee8ab8803fc57ad48d582db04', # AlphaFold-2.0.0_n-cpu.patch
'5363d403baf5ab73f4d3ddd72e19af9ff832de4b1d7ba25a5fbcc5846c1c890f', # AlphaFold-2.1.0_fix-scp-path.patch
'1a2e4e843bd9a4d15ee39e6c37cc63ba281311cc7a0a5610f0e43b52ef93faac', # AlphaFold-2.0.1_setup_rm_tfcpu.patch
# AlphaFold-2.3.1_use_openmm_7.7.0.patch
'd800bb085deac7edbe2d72916c1194043964aaf51b88a3b5a5016ab68a1090ec',

],
'start_dir': 'alphafold-%(version)s',
'use_pip': True,
}),
]

use_pip = True

exts_list = [
('PDBFixer', '1.8.1', {
'source_urls': ['https://github.com/openmm/pdbfixer/archive/refs/tags/'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['d50551abfe9dbaefc066f4d9d400cdebe57f1fefd9de9d01e12beb87efd99595'],
}),
('toolz', '0.12.0', {
'checksums': ['88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194'],
}),
('chex', '0.1.6', {
'checksums': ['adb5d2352b5f0d248ccf594be1b1bf9ee7a2bee2a57f0eac78547538d479b0e7'],
}),
('tabulate', '0.9.0', {
'checksums': ['0095b12bf5966de529c0feb1fa08671671b3368eec77d7ef7ab114be2c068b3c'],
}),
('jmp', '0.0.4', {
'checksums': ['5dfeb0fd7c7a9f72a70fff0aab9d0cbfae32a809c02f4037ff3485ceb33e1730'],
}),
('dm-haiku', '0.0.9', {
'modulename': 'haiku',
'source_urls': ['https://github.com/deepmind/dm-haiku/archive/refs/tags/'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['d550f07f5891ede30ada5faafde98f549ed1b8ceadb7a601cca3d81db7d82414'],
}),
('dm-tree', '0.1.8', {
'modulename': 'tree',
'checksums': ['0fcaabbb14e7980377439e7140bd05552739ca5e515ecb3119f234acee4b9430'],
}),
('websocket-client', '1.5.1', {
'modulename': 'websocket',
'checksums': ['3f09e6d8230892547132177f575a4e3e73cfdf06526e20cc02aa1c3b47184d40'],
}),
('docker', '6.0.1', {
'checksums': ['896c4282e5c7af5c45e8b683b0b0c33932974fe6e50fc6906a0a83616ab3da97'],
}),
('immutabledict', '2.2.3', {
'checksums': ['0e1e8a3f2b3ff062daa19795f947e9ec7a58add269d44e34d3ab4319e1343853'],
}),
('contextlib2', '21.6.0', {
'checksums': ['ab1e2bfe1d01d968e1b7e8d9023bc51ef3509bba217bb730cee3827e1ee82869'],
}),
('ml_collections', '0.1.1', {
'preinstallopts': "touch requirements.txt && touch requirements-test.txt && ",
'checksums': ['3fefcc72ec433aa1e5d32307a3e474bbb67f405be814ea52a2166bfc9dbe68cc'],
}),
]

postinstallcmds = [
"mkdir -p %(installdir)s/bin",
# run_alphafold.py script is missing a shebang...
"echo '#!/usr/bin/env python' > %(installdir)s/bin/run_alphafold.py",
"cat %(builddir)s/alphafold-%(version)s/run_alphafold.py >> %(installdir)s/bin/run_alphafold.py",
"chmod a+x %(installdir)s/bin/run_alphafold*.py",
"cd %(installdir)s/bin && ln -s run_alphafold.py alphafold",
"cp -a %(builddir)s/alphafold-%(version)s/scripts %(installdir)s/",
"cp %%(builddir)s/stereo_chemical_props-%s.txt %%(installdir)s/stereo_chemical_props.txt" % local_scp_commit,
# run tests for run_alphafold.py script;
# shouldn't do this in sanity check to avoid breaking use of --module-only
"PYTHONPATH=%(installdir)s/lib/python%(pyshortver)s/site-packages:$PYTHONPATH "
"python %(builddir)s/alphafold-%(version)s/run_alphafold_test.py",
]

sanity_check_paths = {
'files': ['bin/alphafold', 'bin/pdbfixer', 'bin/run_alphafold.py', 'stereo_chemical_props.txt'],
'dirs': ['lib/python%(pyshortver)s/site-packages', 'scripts'],
}

sanity_check_commands = [
"pdbfixer --help",
"python -m openmm.testInstallation",
"python -c 'import alphafold'",
"alphafold --help 2>&1 | grep 'Full AlphaFold protein structure prediction script'",
]

sanity_pip_check = True

# these allow to make predictions on proteins that would typically be too long to fit into GPU memory;
# see https://github.com/deepmind/alphafold/blob/main/docker/run_docker.py
modextravars = {
'TF_FORCE_UNIFIED_MEMORY': '1',
'XLA_PYTHON_CLIENT_MEM_FRACTION': '3',
# 'ALPHAFOLD_DATA_DIR': '/path/to/AlphaFold_DBs', # please adapt
'OPENMM_RELAX': 'CUDA' # unset or set to 'CPU' in order not to run the energy minimization on GPU; PR#189
}

moduleclass = 'bio'
Loading