Skip to content

make Rust a build dependency for maturin 1.1.0 (2023a generation) #22983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Micket merged 11 commits into from
Jun 11, 2025
Merged

make Rust a build dependency for maturin 1.1.0 (2023a generation) #22983

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
cc78ff6
make Rust a build dependency for maturin 1.1.0 (2023a generation)
boegel May 31, 2025
1d968e9
use CargoPythonBundle or CargoPythonPackage easyblock when Rust is a …
boegel May 31, 2025
70242e7
use CargoPythonBundle easyblock for jupyter-server 2.7.2 + add list o…
boegel May 31, 2025
600597f
add list of crates to easyconfig for torch-em 0.7.1
boegel May 31, 2025
20fd2f8
add list of crates to easyconfig for vLLM 0.4.0
boegel May 31, 2025
2d44fc3
add additional crates required by torch-em 0.7.1 extensions
boegel May 31, 2025
90b6618
fix order of checksums in torcm-em easyconfig
boegel May 31, 2025
ed6e8f8
remove maturin + Rust build dependencies for nbclassic & pymatgen (no…
boegel Jun 2, 2025
293e133
add back maturin as build dependency for pymatgen
boegel Jun 3, 2025
2e4f299
add Rust build dependency + use CargoPythonBundle for pymatgen v2023.…
boegel Jun 6, 2025
f21d6b6
add comment to clarify that orjson 3.9.10 vendors the Rust crates it …
boegel Jun 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 77 additions & 1 deletion easybuild/easyconfigs/j/jupyter-server/jupyter-server-2.7.2-GCCcore-12.3.0.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
easyblock = 'PythonBundle'
easyblock = 'CargoPythonBundle'

name = 'jupyter-server'
version = "2.7.2"
Expand All @@ -12,6 +12,7 @@ toolchain = {'name': 'GCCcore', 'version': '12.3.0'}

builddependencies = [
('binutils', '2.40'),
('Rust', '1.70.0'),
('maturin', '1.1.0'),
]
dependencies = [
Expand All @@ -24,6 +25,81 @@ dependencies = [
('tornado', '6.3.2'),
]

crates = [
('archery', '0.5.0'),
('autocfg', '1.1.0'),
('bitflags', '1.3.2'),
('cfg-if', '1.0.0'),
('indoc', '1.0.9'),
('libc', '0.2.147'),
('lock_api', '0.4.10'),
('memoffset', '0.9.0'),
('once_cell', '1.18.0'),
('parking_lot', '0.12.1'),
('parking_lot_core', '0.9.8'),
('proc-macro2', '1.0.66'),
('pyo3', '0.19.1'),
('pyo3-build-config', '0.19.1'),
('pyo3-ffi', '0.19.1'),
('pyo3-macros', '0.19.1'),
('pyo3-macros-backend', '0.19.1'),
('quote', '1.0.31'),
('redox_syscall', '0.3.5'),
('rpds', '0.13.0'),
('scopeguard', '1.1.0'),
('smallvec', '1.11.0'),
('static_assertions', '1.1.0'),
('syn', '1.0.109'),
('target-lexicon', '0.12.9'),
('unicode-ident', '1.0.11'),
('unindent', '0.1.11'),
('windows-targets', '0.48.1'),
('windows_aarch64_gnullvm', '0.48.0'),
('windows_aarch64_msvc', '0.48.0'),
('windows_i686_gnu', '0.48.0'),
('windows_i686_msvc', '0.48.0'),
('windows_x86_64_gnu', '0.48.0'),
('windows_x86_64_gnullvm', '0.48.0'),
('windows_x86_64_msvc', '0.48.0'),
]
checksums = [
{'archery-0.5.0.tar.gz': 'b6cd774058b1b415c4855d8b86436c04bf050c003156fe24bc326fb3fe75c343'},
{'autocfg-1.1.0.tar.gz': 'd468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa'},
{'bitflags-1.3.2.tar.gz': 'bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a'},
{'cfg-if-1.0.0.tar.gz': 'baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd'},
{'indoc-1.0.9.tar.gz': 'bfa799dd5ed20a7e349f3b4639aa80d74549c81716d9ec4f994c9b5815598306'},
{'libc-0.2.147.tar.gz': 'b4668fb0ea861c1df094127ac5f1da3409a82116a4ba74fca2e58ef927159bb3'},
{'lock_api-0.4.10.tar.gz': 'c1cc9717a20b1bb222f333e6a92fd32f7d8a18ddc5a3191a11af45dcbf4dcd16'},
{'memoffset-0.9.0.tar.gz': '5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c'},
{'once_cell-1.18.0.tar.gz': 'dd8b5dd2ae5ed71462c540258bedcb51965123ad7e7ccf4b9a8cafaa4a63576d'},
{'parking_lot-0.12.1.tar.gz': '3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f'},
{'parking_lot_core-0.9.8.tar.gz': '93f00c865fe7cabf650081affecd3871070f26767e7b2070a3ffae14c654b447'},
{'proc-macro2-1.0.66.tar.gz': '18fb31db3f9bddb2ea821cde30a9f70117e3f119938b5ee630b7403aa6e2ead9'},
{'pyo3-0.19.1.tar.gz': 'ffb88ae05f306b4bfcde40ac4a51dc0b05936a9207a4b75b798c7729c4258a59'},
{'pyo3-build-config-0.19.1.tar.gz': '554db24f0b3c180a9c0b1268f91287ab3f17c162e15b54caaae5a6b3773396b0'},
{'pyo3-ffi-0.19.1.tar.gz': '922ede8759e8600ad4da3195ae41259654b9c55da4f7eec84a0ccc7d067a70a4'},
{'pyo3-macros-0.19.1.tar.gz': '8a5caec6a1dd355964a841fcbeeb1b89fe4146c87295573f94228911af3cc5a2'},
{'pyo3-macros-backend-0.19.1.tar.gz': 'e0b78ccbb160db1556cdb6fd96c50334c5d4ec44dc5e0a968d0a1208fa0efa8b'},
{'quote-1.0.31.tar.gz': '5fe8a65d69dd0808184ebb5f836ab526bb259db23c657efa38711b1072ee47f0'},
{'redox_syscall-0.3.5.tar.gz': '567664f262709473930a4bf9e51bf2ebf3348f2e748ccc50dea20646858f8f29'},
{'rpds-0.13.0.tar.gz': '9bd6ce569b15c331b1e5fd8cf6adb0bf240678b5f0cdc4d0f41e11683f6feba9'},
{'scopeguard-1.1.0.tar.gz': 'd29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd'},
{'smallvec-1.11.0.tar.gz': '62bb4feee49fdd9f707ef802e22365a35de4b7b299de4763d44bfea899442ff9'},
{'static_assertions-1.1.0.tar.gz': 'a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f'},
{'syn-1.0.109.tar.gz': '72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237'},
{'target-lexicon-0.12.9.tar.gz': 'df8e77cb757a61f51b947ec4a7e3646efd825b73561db1c232a8ccb639e611a0'},
{'unicode-ident-1.0.11.tar.gz': '301abaae475aa91687eb82514b328ab47a211a533026cb25fc3e519b86adfc3c'},
{'unindent-0.1.11.tar.gz': 'e1766d682d402817b5ac4490b3c3002d91dfa0d22812f341609f97b08757359c'},
{'windows-targets-0.48.1.tar.gz': '05d4b17490f70499f20b9e791dcf6a299785ce8af4d709018206dc5b4953e95f'},
{'windows_aarch64_gnullvm-0.48.0.tar.gz': '91ae572e1b79dba883e0d315474df7305d12f569b400fcf90581b06062f7e1bc'},
{'windows_aarch64_msvc-0.48.0.tar.gz': 'b2ef27e0d7bdfcfc7b868b317c1d32c641a6fe4629c171b8928c7b08d98d7cf3'},
{'windows_i686_gnu-0.48.0.tar.gz': '622a1962a7db830d6fd0a69683c80a18fda201879f0f447f065a3b7467daa241'},
{'windows_i686_msvc-0.48.0.tar.gz': '4542c6e364ce21bf45d69fdd2a8e455fa38d316158cfd43b3ac1c5b1b19f8e00'},
{'windows_x86_64_gnu-0.48.0.tar.gz': 'ca2b8a661f7628cbd23440e50b05d705db3686f894fc9580820623656af974b1'},
{'windows_x86_64_gnullvm-0.48.0.tar.gz': '7896dbc1f41e08872e9d5e8f8baa8fdd2677f29468c4e156210174edc7f7b953'},
{'windows_x86_64_msvc-0.48.0.tar.gz': '1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a'},
]

# WARNING: the versions of ipywidgets, widgetsnbextension and jupyterlab_widgets are tied between them
# use the versions published in a single release commit instead of blindly pushing to last available version,
# see for instance https://github.com/jupyter-widgets/ipywidgets/commit/b728926f58ed3ffef08f716998ac6c226dafc1aa
Expand Down
2 changes: 1 addition & 1 deletion easybuild/easyconfigs/m/maturin/maturin-1.1.0-GCCcore-12.3.0.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,11 +303,11 @@ checksums = [

builddependencies = [
('binutils', '2.40'),
('Rust', '1.70.0'),
('setuptools-rust', '1.6.0'),
]
dependencies = [
('Python', '3.11.3'),
('Rust', '1.70.0'),
]

crates = [
Expand Down
1 change: 0 additions & 1 deletion easybuild/easyconfigs/n/nbclassic/nbclassic-1.0.0-GCCcore-12.3.0.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ toolchain = {'name': 'GCCcore', 'version': '12.3.0'}

builddependencies = [
('binutils', '2.40'),
('maturin', '1.1.0'),
]
dependencies = [
('Python', '3.11.3'),
Expand Down
3 changes: 2 additions & 1 deletion easybuild/easyconfigs/p/pymatgen/pymatgen-2023.12.18-foss-2023a.eb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
easyblock = 'PythonBundle'
easyblock = 'CargoPythonBundle'
Copy link
Contributor

@Micket Micket Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orjson is a rust package

crates = [
    ('ahash', '0.8.6'),
    ('arrayvec', '0.7.4'),
    ('associative-cache', '2.0.0'),
    ('autocfg', '1.1.0'),
    ('beef', '0.5.2'),
    ('bytecount', '0.6.7'),
    ('castaway', '0.2.2'),
    ('cc', '1.0.83'),
    ('cfg-if', '1.0.0'),
    ('chrono', '0.4.31'),
    ('compact_str', '0.7.1'),
    ('encoding_rs', '0.8.33'),
    ('itoa', '1.0.9'),
    ('itoap', '1.0.1'),
    ('libc', '0.2.149'),
    ('libm', '0.2.8'),
    ('no-panic', '0.1.26'),
    ('num-traits', '0.2.17'),
    ('once_cell', '1.18.0'),
    ('packed_simd', '0.3.9'),
    ('proc-macro2', '1.0.69'),
    ('pyo3-build-config', '0.20.0'),
    ('pyo3-ffi', '0.20.0'),
    ('quote', '1.0.33'),
    ('rustversion', '1.0.14'),
    ('ryu', '1.0.15'),
    ('serde', '1.0.190'),
    ('serde_derive', '1.0.190'),
    ('serde_json', '1.0.107'),
    ('simdutf8', '0.1.4'),
    ('smallvec', '1.11.1'),
    ('static_assertions', '1.1.0'),
    ('syn', '2.0.38'),
    ('target-lexicon', '0.12.12'),
    ('unicode-ident', '1.0.12'),
    ('version_check', '0.9.4'),
    ('zerocopy', '0.7.15'),
    ('zerocopy-derive', '0.7.15'),
]

I.. really don't understand how this could pass a build without this. We should be setting cargo to offline mode, but maybe the maturin/pip wrapper thing overrides those environment variables, allowing it to go online?

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was also a bit puzzled about how this could work without providing a list of crates...

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this explains it?

== 2025-05-31 23:13:07,252 easyblock.py:3091 DEBUG Installing extension orjson with default class PythonPackage (from easybuild.easyblocks.generic.pythonpackage)

In pymatgen-2024.5.1-foss-2023b.eb, orjson has been lifted in a proper dependency, so situation is different there.

Copy link
Member Author

@boegel boegel Jun 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Micket CargoPythonBundle should make sure that CargoPythonPackage is used for all extensions, so Cargo.extract_step (which ends up setting $CARGO_NET_OFFLINE to true is used for every extension?

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, no, should be fine, the log tells me that $CARGO_NET_OFFLINE is being set to true right before the installation of every extension, including orjson

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it's being set, but then later being unset again, before the actual pip install happens:

egrep 'installing.*extension|CARGO_NET_OFFLINE|Running shell command.*pip install.*orjson' /software/pymatgen/2023.12.18-foss-2023a/easybuild/*log*
...
== 2025-05-31 23:15:56,812 environment.py:93 INFO Environment variable CARGO_NET_OFFLINE set to true (previously undefined)
== 2025-05-31 23:15:56,852 build_log.py:322 INFO installing extension orjson 3.9.10 (13/25)...
== 2025-05-31 23:15:57,920 environment.py:172 DEBUG Key in old environment found that is not in new one: CARGO_NET_OFFLINE (true)
== 2025-05-31 23:16:05,368 run.py:500 INFO Running shell command '/user/gent/400/vsc40023/eb_arcaninescratch/RHEL9/zen2-ib/software/Python/3.11.3-GCCcore-12.3.0/bin/python -m pip install --prefix=/user/gent/400/vsc40023/eb_arcaninescratch/RHEL9/zen2-ib/software/pymatgen/2023.12.18-foss-2023a  --verbose --no-deps --ignore-installed --no-build-isolation .' in /tmp/vsc40023/easybuild_build/pymatgen/2023.12.18/foss-2023a/orjson/orjson-3.9.10

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the problem is that $CARGO_NET_OFFLINE is not being set by Cargo.set_cargo_vars, which results in it not being set anymore when the actual installation of extensions is done.

That's probably an unexpected side effect from the changes implemented in easybuilders/easybuild-framework#4868

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the specific version of orjson being used here (3.9.10), it seems like cargo doesn't need to download anything at all, because the sources of the requires crates are included in the source tarball or orjson:

eb-shell> pwd
/tmp/vsc40023/easybuild_build/pymatgen/2023.12.18/foss-2023a/orjson/orjson-3.9.10
eb-shell> ls include/cargo/
ahash-0.8.6              bytecount-0.6.7  compact_str-0.7.1   libm-0.2.8         proc-macro2-1.0.69        ryu-1.0.15            smallvec-1.11.1          version_check-0.9.4
arrayvec-0.7.4           castaway-0.2.2   encoding_rs-0.8.33  no-panic-0.1.26    pyo3-build-config-0.20.0  serde-1.0.190         static_assertions-1.1.0  zerocopy-0.7.15
associative-cache-2.0.0  cc-1.0.83        itoa-1.0.9          num-traits-0.2.17  pyo3-ffi-0.20.0           serde_derive-1.0.190  syn-2.0.38               zerocopy-derive-0.7.15
autocfg-1.1.0            cfg-if-1.0.0     itoap-1.0.1         once_cell-1.18.0   quote-1.0.33              serde_json-1.0.107    target-lexicon-0.12.12
beef-0.5.2               chrono-0.4.31    libc-0.2.149        packed_simd-0.3.9  rustversion-1.0.14        simdutf8-0.1.4        unicode-ident-1.0.12

So although easybuilders/easybuild-easyblocks#3764 is a correct fix, we don't actually need to provide a list of crates at all for this pymatgen easyconfig, since all crates required by the orjson extension included in it are shipped/vendored with it...

Copy link
Contributor

@Micket Micket Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets make a comment that explains that this specific older version of orjson used vendored crates.
Updating to newer versions will require adding the crates.

Copy link
Member Author

@boegel boegel Jun 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Micket OK, makes sense, done in f21d6b6


name = 'pymatgen'
version = '2023.12.18'
Expand All @@ -12,6 +12,7 @@ toolchain = {'name': 'foss', 'version': '2023a'}
builddependencies = [
('hatchling', '1.18.0'),
('hypothesis', '6.82.0'), # required for tests
('Rust', '1.70.0'),
('maturin', '1.1.0'),
('poetry', '1.5.1'),
]
Expand Down
Loading