don't build GO tests by default for LLVM versions < 16 #3690

Crivella · 2025-04-07T10:12:35Z

Should be tested with some LLVM ECs < 16 and atleast one full build with LLVM > 19

Crivella · 2025-04-07T10:18:58Z

As a note there is also the option to disable LLVM_ENABLE_BINDINGS directly but that controls not only the GO bindings (removed in 16) but also the OCaml bindings that are still present in 20.1.2, so i would probably just see if disabling the test works

Crivella · 2025-04-15T09:32:16Z

Added a fix for the problem described in #3680 (comment)

In general i do not see a problem with installing the python bindings, creating the config files and symlinks for the runtime libraries/headers, together with the install step (as this is necessary for the compilers to work).

Should be tested with:

An EasyConfig with LLVM >=18.1.8 and GCCcore with python bindings (as they are in general disabled for the system version)
An EasyConfig for LLVM ~=19.x
An EasyConfig for LLVM ~=20.x

Crivella · 2025-04-30T16:04:15Z

@boegelbot please test @ jsc-zen3
EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb --installpath /tmp/$USER/ebpr-3690"

Crivella · 2025-04-30T16:05:34Z

Beside #3690 (comment) , also a local test with a system GO should be performed in order to check for #3689

boegelbot · 2025-04-30T16:13:08Z

@Crivella: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3690 EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb --installpath /tmp/$USER/ebpr-3690" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3690 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 6269

Test results coming soon (I hope)...

- notification for comment with ID 2842491970 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-04-30T16:57:22Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS LLVM-11.1.0-GCCcore-10.3.0.eb
SUCCESS LLVM-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/067b7cdbe36315d6bb4d547ba3dcd5f5 for a full test report.

Crivella · 2025-05-01T01:11:35Z

Test report by @Crivella

Overview of tested easyconfigs (in order)

SUCCESS LLVM-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
crivella-desktop - Linux Ubuntu 22.04.5 LTS (Jammy Jellyfish), x86_64, 13th Gen Intel(R) Core(TM) i9-13900K, Python 3.11.12
See https://gist.github.com/Crivella/e32c0f8c153e8ea7bbc73bfb1d4f0d56 for a full test report.

NOTES

Built with a system GO-LANG present (simulated by having an executable /usr/bin/go)

crivella@crivella-desktop:~/.local/easybuild/software/LLVM/12.0.1-GCCcore-10.3.0$ grep "Go bindings" easybuild/easybuild-LLVM-12.0.1-20250501.031134.log
-- Go bindings enabled.

The test is properly not executed

crivella@crivella-desktop:~/.local/easybuild/software/LLVM/12.0.1-GCCcore-10.3.0$ grep "Go/" easybuild/easybuild-LLVM-12.0.1-20250501.031134.log
UNSUPPORTED: LLVM :: Bindings/Go/go.test (7050 of 42113)
UNSUPPORTED: LLVM :: Bindings/Go/go.test (7050 of 42113)

Also tested without --include-easyblocks-from-pr which lead to the same error as reported in easybuilders/easybuild-easyconfigs#22720 (comment)

https://gist.github.com/Crivella/8ddaa42ef4cef71930cdd373f8b8846a

Crivella · 2025-05-01T09:58:43Z

I've rebased this to be up to date with develop

@bedroge Would you mind testing one of the Bundle build you are using with this to see if it works?

I am also redoing the test for the GO issue.

After both passes i think this would probably be good to go

Crivella · 2025-05-01T10:40:06Z

Test report by @Crivella

Overview of tested easyconfigs (in order)

SUCCESS LLVM-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
crivella-desktop - Linux Ubuntu 22.04.5 LTS (Jammy Jellyfish), x86_64, 13th Gen Intel(R) Core(TM) i9-13900K, Python 3.11.12
See https://gist.github.com/Crivella/05b4a282e0498b24dac050f322dd3047 for a full test report.

Crivella · 2025-05-07T10:10:41Z

Ran a test for #3680 using a Bundle EC wrapping around LLVM only

Without the fix the build fails by improperly setting the internal variable llvm_src_dir to .../easybuild/build/llvmproject/20.1.1/...

$ eb LLVMBundleTest-20.1.1.eb -d

...
== installing bundle component llvm-project v20.1.1 (1/1)...                                                                                                                                                                                             
  >> applying patch LLVM-18.1.8_envintest.patch                                                                                                                                                                                                          
  >> applying patch LLVM-19.1.7_libomptarget_tests.patch                                                                                                                                                                                                 
  >> applying patch LLVM-19.1.7_clang_rpathwrap_test.patch                                                                                                                                                                                               
== ... (took < 1 sec)                                                                                                                                                                                                                                    
== FAILED: Installation ended unsuccessfully: Failed to patch /home/crivella/.local/easybuild/build/llvmproject/20.1.1/system-system/llvm-project-20.1.1.src/clang/CMakeLists.txt: [Errno 2] No such file or directory: '/home/crivella/.local/easybuild/
build/llvmproject/20.1.1/system-system/llvm-project-20.1.1.src/clang/CMakeLists.txt' (took 10 secs)              
...

With the PR fix the internal variable is properly set to .../easybuild/build/LLVMBundleTest/20.1.1/...

$ eb LLVMBundleTest-20.1.1.eb -d --include-easyblocks-from-pr 3690

...
== installing bundle component llvm-project v20.1.1 (1/1)...
  >> applying patch LLVM-18.1.8_envintest.patch
  >> applying patch LLVM-19.1.7_libomptarget_tests.patch
  >> applying patch LLVM-19.1.7_clang_rpathwrap_test.patch
  >> running shell command:
        cmake  -DCMAKE_INSTALL_LIBDIR:PATH=lib -DCM
...

Crivella · 2025-05-08T12:11:02Z

@bedroge I think with #3690 (comment) this is ready to go, we can wait for #3706 to be merged or viceversa to rebase this PR on top and than have it go in (maybe with another quick test, but i think the 2 PRs are completely orthogonal, so there should be no problem)

bedroge · 2025-05-09T08:48:39Z

I also tried this with my ROCm-LLVM draft easyconfig, which makes use of the easyblock in a Bundle. It worked like a charm.

Crivella · 2025-05-20T14:03:34Z

@boegelbot please test @ jsc-zen3
EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb --installpath /tmp/$USER/ebpr-3690"

Crivella · 2025-05-20T14:07:25Z

Rebased this PR on top of develop after merging #3706 which also included the fixes for running the EBlock in a Bundle

The test with LLVM inside a bundle still works for me (the crashes without the fix would happen before getting to the actual build

(easybuild-dev) crivella@crivella-desktop:~$ eb --max-parallel 1 LLVMBundleTest-20.1.1.eb 
== Temporary log file in case of crash /home/crivella/.local/easybuild/logs/easybuild-qm6lbqs1.log

WARNING: Index for /home/crivella/Documents/GIT/easybuild-easyconfigs/easybuild/easyconfigs is no longer valid (too old), so ignoring it...

== processing EasyBuild easyconfig /home/crivella/test/easyconfigs/LLVMBundleTest-20.1.1.eb
== building and installing LLVMBundleTest/20.1.1...
  >> installation prefix: /home/crivella/.local/easybuild/software/LLVMBundleTest/20.1.1
== fetching files and verifying checksums...
  >> sources:
  >> /home/crivella/.local/easybuild/sources/l/LLVMBundleTest/llvm-project-20.1.1.src.tar.xz [SHA256: 4d5ebbd40ce1e984a650818a4bb5ae86fc70644dec2e6d54e78b4176db3332e0]
  >> patches:
  >> /home/crivella/test/easyconfigs/LLVM-18.1.8_envintest.patch [SHA256: 8e25dfab8a29a860717b4bd2d8cdd0e795433766d7fffbda32d06a2bde47058d]
  >> /home/crivella/test/easyconfigs/LLVM-19.1.7_libomptarget_tests.patch [SHA256: 79a67c118d034cfb74e255696369150c73432d2b422f4834efacb26f7904edbf]
  >> /home/crivella/test/easyconfigs/LLVM-19.1.7_clang_rpathwrap_test.patch [SHA256: 5ee6a87ec8ff1c8b736ffe0513aa2098bd2b83a1ffc647a1ad2cf966f567e8a1]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /home/crivella/.local/easybuild/build/LLVMBundleTest/20.1.1/system-system
== ... (took < 1 sec)
== unpacking...
  >> running shell command:
	unset TAPE; unxz /home/crivella/.local/easybuild/sources/l/LLVMBundleTest/llvm-project-20.1.1.src.tar.xz --stdout | tar x
	[started at: 2025-05-20 16:02:15]
	[working dir: /home/crivella/.local/easybuild/build/LLVMBundleTest/20.1.1/system-system]
	[output and state saved to /tmp/eb-3mimv3rc/run-shell-cmd-output/unset-bjir2q7h]
  >> command completed: exit 0, ran in 00h00m09s
== ... (took 9 secs)
== patching...
== ... (took < 1 sec)
== preparing...
  >> loading modules for build dependencies:
  >>  * GCCcore/13.3.0
  >>  * CMake/3.29.3-GCCcore-13.3.0
  >>  * psutil/6.0.0-GCCcore-13.3.0
  >>  * Python/3.12.3-GCCcore-13.3.0
  >>  * lit/18.1.8-GCCcore-13.3.0
  >>  * git/2.45.1-GCCcore-13.3.0
  >>  * zlib/1.3.1
  >> (no (runtime) dependencies specified)
== ... (took < 1 sec)
== configuring...
== ... (took < 1 sec)
== building...
== ... (took < 1 sec)
== testing...
== ... (took < 1 sec)
== installing...
== installing bundle component llvm-project v20.1.1 (1/1)...
  >> applying patch LLVM-18.1.8_envintest.patch
  >> applying patch LLVM-19.1.7_libomptarget_tests.patch
  >> applying patch LLVM-19.1.7_clang_rpathwrap_test.patch
  >> running shell command:
	cmake  -DCMAKE_INSTALL_LIBDIR:PATH=lib -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DLLVM_ENABLE_PROJECTS="llvm;clang;lld" -DLLVM_ENABLE_RUNTIMES="compiler-rt;libunwind;libcxx;libcxxabi" -DCMAKE_VERBOSE_MAKEFILE=ON -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_INSTALL_UTILS=ON -DPython3_FIND_VIRTUALENV=STANDARD -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DLIBCXX_ENABLE_SHARED=ON -DLIBCXXABI_ENABLE_SHARED=ON -DLIBUNWIND_ENABLE_SHARED=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_REQUIRES_RTTI=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCLANG_DEFAULT_LINKER=lld -DFLANG_DEFAULT_LINKER=lld -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="all" -DCMAKE_INSTALL_PREFIX=/home/crivella/.local/easybuild/software/LLVMBundleTest/20.1.1 -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_ENABLE_ZLIB=ON -DLLVM_ENABLE_ZSTD=OFF -DLLDB_ENABLE_SWIG=OFF -DLLVM_ENABLE_Z3_SOLVER=OFF -DPYTHON_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DPython3_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python -DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_FLAGS=--gcc-install-dir=/home/crivella/.local/easybuild/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DCMAKE_CXX_FLAGS=--gcc-install-dir=/home/crivella/.local/easybuild/software/GCCcore/13.3.0/lib/gcc/x86_64-pc-linux-gnu/13.3.0;-DPYTHON_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python;-DPython3_EXECUTABLE=/home/crivella/.local/easybuild/software/Python/3.12.3-GCCcore-13.3.0/bin/python" /home/crivella/.local/easybuild/build/LLVMBundleTest/20.1.1/system-system/llvm-project-20.1.1.src/llvm
	[started at: 2025-05-20 16:02:25]
	[working dir: /home/crivella/.local/easybuild/build/LLVMBundleTest/20.1.1/system-system/llvm.obj.1]
	[output and state saved to /tmp/eb-3mimv3rc/run-shell-cmd-output/cmake-qu328mo_]
  >> command completed: exit 0, ran in 00h00m29s
== Building stage 1/3
  >> running shell command:
	make
	[started at: 2025-05-20 16:02:55]
	[working dir: /home/crivella/.local/easybuild/build/LLVMBundleTest/20.1.1/system-system/llvm.obj.1]
	[output and state saved to /tmp/eb-3mimv3rc/run-shell-cmd-output/make-9yk6jhu8]

I am also rerunning the tests with a system /usr/bin/go present again and the bot than I think this can be merged

boegelbot · 2025-05-20T14:13:09Z

@Crivella: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3690 EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb --installpath /tmp/$USER/ebpr-3690" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3690 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 6401

Test results coming soon (I hope)...

- notification for comment with ID 2894546205 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Crivella · 2025-05-20T14:45:40Z

Test report by @Crivella

Overview of tested easyconfigs (in order)

SUCCESS LLVM-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
crivella-desktop - Linux Ubuntu 22.04.5 LTS (Jammy Jellyfish), x86_64, 13th Gen Intel(R) Core(TM) i9-13900K (skylake), Python 3.11.12
See https://gist.github.com/Crivella/05811031a1d74c27f694da95d161f14d for a full test report.

Results based on having GO installed at the system level

Similarly to the previous test

GO bindings are enabled

$ grep "Go bindings" easybuild/easybuild-LLVM-12.0.1-20250520.164539.log 
-- Go bindings enabled.

But tests are skipped since GO is not coming from an EB module

$ grep "Go/" easybuild/easybuild-LLVM-12.0.1-20250520.164539.log 
UNSUPPORTED: LLVM :: Bindings/Go/go.test (7050 of 42113)
UNSUPPORTED: LLVM :: Bindings/Go/go.test (7050 of 42113)

boegelbot · 2025-05-20T14:58:54Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS LLVM-11.1.0-GCCcore-10.3.0.eb
SUCCESS LLVM-12.0.1-GCCcore-10.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/907f7e799958f9c2b1200e4cd4b0bf07 for a full test report.

boegel

lgtm

Crivella mentioned this pull request Apr 7, 2025

Re-add build of shared libs for LLVM [GCCcore 10.3.0] easybuilders/easybuild-easyconfigs#22720

Merged

boegel changed the title ~~Fixes #3689~~ fix using LLM easyblock for bundle component Apr 23, 2025

boegel added the bug fix label Apr 23, 2025

boegel added this to the 5.0.1 milestone Apr 23, 2025

boegel changed the title ~~fix using LLM easyblock for bundle component~~ fix using LLVM easyblock for bundle component Apr 23, 2025

boegel changed the title ~~fix using LLVM easyblock for bundle component~~ fix using LLVM easyblock for bundle component + don't build GO tests by default for LLVM versions < 16 Apr 23, 2025

Crivella requested a review from bedroge May 1, 2025 09:50

Crivella force-pushed the fix-go_tests branch from 0fda827 to d60e130 Compare May 1, 2025 09:56

Crivella force-pushed the fix-go_tests branch from d60e130 to 639ba77 Compare May 19, 2025 08:22

Crivella added 2 commits May 20, 2025 15:51

Fixes easybuilders#3689

e5550f7

Fixes easybuilders#3680

9c8ad42

Crivella force-pushed the fix-go_tests branch from 639ba77 to 9c8ad42 Compare May 20, 2025 13:55

Revert empty line

02703f8

Crivella mentioned this pull request May 20, 2025

LLVM EasyBlock fails when used as a Bundle component #3680

Closed

boegel changed the title ~~fix using LLVM easyblock for bundle component + don't build GO tests by default for LLVM versions < 16~~ don't build GO tests by default for LLVM versions < 16 May 21, 2025

boegel approved these changes May 21, 2025

View reviewed changes

boegel merged commit d49fff7 into easybuilders:develop May 21, 2025
17 checks passed

Crivella deleted the fix-go_tests branch May 21, 2025 12:31

don't build GO tests by default for LLVM versions < 16 #3690

don't build GO tests by default for LLVM versions < 16 #3690

Uh oh!

Conversation

Crivella commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Crivella commented Apr 7, 2025

Uh oh!

Crivella commented Apr 15, 2025

Uh oh!

Crivella commented Apr 30, 2025

Uh oh!

Crivella commented Apr 30, 2025

Uh oh!

boegelbot commented Apr 30, 2025

Uh oh!

boegelbot commented Apr 30, 2025

Overview of tested easyconfigs (in order)

Uh oh!

Crivella commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

NOTES

Uh oh!

Crivella commented May 1, 2025

Uh oh!

Crivella commented May 1, 2025

Overview of tested easyconfigs (in order)

Uh oh!

Crivella commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Crivella commented May 8, 2025

Uh oh!

bedroge commented May 9, 2025

Uh oh!

Crivella commented May 20, 2025

Uh oh!

Crivella commented May 20, 2025

Uh oh!

boegelbot commented May 20, 2025

Uh oh!

Crivella commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview of tested easyconfigs (in order)

Results based on having GO installed at the system level

Uh oh!

boegelbot commented May 20, 2025

Overview of tested easyconfigs (in order)

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Crivella commented Apr 7, 2025 •

edited

Loading

Crivella commented May 1, 2025 •

edited

Loading

Crivella commented May 7, 2025 •

edited

Loading

Crivella commented May 20, 2025 •

edited

Loading