Skip to content

{ai}[foss/2022a] PyTorch-bundle v1.12.1 #17540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lexming merged 6 commits into from
Apr 20, 2023
Merged

{ai}[foss/2022a] PyTorch-bundle v1.12.1 #17540

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
170a69f
adding easyconfigs: PyTorch-bundle-1.12.1-foss-2022a-CUDA-11.7.0.eb
Micket Mar 15, 2023
d81414e
Fix version template in download filename
Micket Mar 15, 2023
b3e0cca
fix typo
Micket Mar 16, 2023
d42c60d
Switch to torchdata 0.5.1
Micket Mar 17, 2023
bdc3b27
Include Viktors fixes
Micket Apr 17, 2023
dfcc41c
Fix patch descriptions
Micket Apr 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions easybuild/easyconfigs/p/PyTorch-bundle/PyTorch-bundle-1.12.1-foss-2022a-CUDA-11.7.0.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
easyblock = 'PythonBundle'

name = 'PyTorch-bundle'
Copy link
Member

@boegel boegel Mar 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't PyTorch-extensions be a better name?

I think PyTorch-bundle was chosen in analogy to SciPy-bundle, but that's a collection of Python packages from different contexts, while here we're only bundling "official" PyTorch extensions (since they're all obtained via https://github.com/pytorch)

Copy link
Member

@boegel boegel Apr 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lexming @Micket Can we make up our minds here, so we can get this merged?

Copy link
Contributor

@VRehnberg VRehnberg Apr 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As Micket didn't have a strong opinion (see below) I'd say that you can probably decide @boegel. To add some nuance I'd like to note that it is a PythonBundle of PyTorch domain libraries under https://github.com/pytorch, though excluding PyTorch it self, and as it is a PythonBundle these libraries are added in an extension list. So, I think PyTorch-extensions and PyTorch-bundle can both make sense, or even PyTorch-extensions-bundle if you'd like to be even more specific.

I looked through the names of other bundles and these where the only other relevant ones I found (i.e. they didn't just have names corresponding to one of the dependencies or extensions):

  • ESL-Bundle
  • R-bundle-Bioconductor
  • SciPy-bundle

As this is the only thing remaining I'd appreciate if it could be resolved soon.

Copy link
Contributor

@lexming lexming Apr 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My vote goes for PyTorch-bundle

version = '1.12.1'
versionsuffix = '-CUDA-%(cudaver)s'

homepage = 'https://pytorch.org/'
description = """PyTorch with compatible versions of official Torch extensions."""

toolchain = {'name': 'foss', 'version': '2022a'}

builddependencies = [
('RE2', '2022-06-01'), # for torchtext
]

dependencies = [
('CUDA', '11.7.0', '', SYSTEM),
('PyTorch', version, versionsuffix),
('Pillow-SIMD', '9.2.0'), # for torchvision
('SentencePiece', '0.1.97'), # for torchtext
('tqdm', '4.64.0'), # for torchtext
('double-conversion', '3.2.0'), # for torchtext
]

use_pip = True

exts_list = [
('torchvision', '0.14.1', {
'source_urls': ['https://github.com/pytorch/vision/archive'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['ced67e1cf1f97e168cdf271851a4d0b6d382ab7936e7bcbb39aaa87239c324b6'],
}),
('torchtext', '0.14.1', {
Copy link
Member

@boegel boegel Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work without additional effort I think, at least not if we want to be sure that the provided SentencePiece is used.

@PetrKralCZ has been fighting with this, and is about to open a PR for a torchtext easyconfig...

Copy link
Contributor Author

@Micket Micket Mar 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes, i forgot about that sed thing while i was fighting the download filenames

Copy link
Contributor Author

@Micket Micket Mar 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the old sed command that was used in earlier versions, completely untested. @PetrKralCZ if you have any feedback on this part i would appreciate it.

# Disable bundled libraries to use those from EB: RE2, SentencePiece
'preinstallopts': "sed -i '/third_party/d;/BuildExtension/d' setup.py &&",
'source_urls': ['https://github.com/pytorch/text/archive'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['fd1ef3da7d9c20408c740f7dc7d02ad52a6048b46368355a1a7326d3bc4f2e63'],
}),
('torchaudio', '0.13.1', {
'source_urls': ['https://github.com/pytorch/audio/archive'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['3a5c108a9ea31d0e6533a2cd6cbaf133efaefe4bdcb4b689183a33b991c95d12'],
}),
('pytorch-ignite', '0.4.11', {
'source_urls': ['https://github.com/pytorch/ignite/archive'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['33e7485cea3cac08d7a49a3a01d3013e1156128867602bff0a8e32b2c5ebd4e9'],
}),
('torchdata', '0.5.1', {
'source_urls': ['https://github.com/pytorch/data/archive'],
'sources': [{'download_filename': 'v%(version)s.tar.gz', 'filename': '%(name)s-%(version)s.tar.gz'}],
'checksums': ['69d80bd33ce8f08e7cfeeb71cefddfc29cede25a85881e33dbae47576b96ed29'],
}),
]

sanity_pip_check = True

moduleclass = 'ai'