make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) #2581

ocaisa · 2021-09-23T11:00:46Z

Fixes easybuilders/easybuild-framework#3846

This is necessary because if the EB python is, e.g., python3.6 but python3.8 is in the PATH, then 3.8 will be used for the actual installation, causing the sanity check to fail (since it will check for libs in the 3.6 path).

ocaisa · 2021-09-23T11:35:45Z

With this fix

[EESSI pilot 2021.03] $ export PATH=/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin:$PATH
[EESSI pilot 2021.03] $ which python
/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python

[EESSI pilot 2021.03] $ which python3
/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python3

[EESSI pilot 2021.03] $ /usr/bin/python3 --version
Python 3.6.8

[EESSI pilot 2021.03] $ EB_VERBOSE=1 EB_PYTHON=/usr/bin/python3 eb --install-latest
>> Considering '/usr/bin/python3'...
>> '/usr/bin/python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5)
>> '/usr/bin/python3' is able to import 'easybuild.main', so retaining it
>> Selected Python command: /usr/bin/python3 (/usr/bin/python3)
>> /usr/bin/python3 -m easybuild.main --install-latest
== Temporary log file in case of crash /tmp/eb-1hk_ls7r/easybuild-ho_3iyls.log

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* GitPython/3.1.0

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

  >> running command:
	[started at: 2021-09-23 11:28:22]
	[working dir: /tmp/eb-1hk_ls7r/tmp_l26gwcg/easybuilders]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-spuxg8yn.log]
	tar xzf /tmp/eb-1hk_ls7r/tmp_l26gwcg/easybuilders/develop.tar.gz
  >> command completed: exit 0, ran in 00h00m01s

WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information

== processing EasyBuild easyconfig /tmp/eb-1hk_ls7r/tmp_l26gwcg/easybuilders/easybuild-easyconfigs-develop/easybuild/easyconfigs/e/EasyBuild/EasyBuild-4.4.2.eb
== building and installing EasyBuild/4.4.2...
  >> installation prefix: /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
== fetching files...
  >> sources:
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-framework-4.4.2.tar.gz [SHA256: 1e6580f210bc742ee1518d0a85389b32199b692de80548c80e93fa2a1a32dd01]
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyblocks-4.4.2.tar.gz [SHA256: 4918f13ffcd97c8b9eb0a10517bdc5b28f67c3602882d4be0bee3afc401fd78c]
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyconfigs-4.4.2.tar.gz [SHA256: 6ec2fa4d9116d45298318f31292726b6f6d1257580cff82138904c69ab38bc46]
== ... (took 2 secs)
== creating build dir, resetting environment...
  >> build dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy
== ... (took < 1 sec)
== unpacking...
  >> running command:
	[started at: 2021-09-23 11:28:27]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-prgyh28i.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-framework-4.4.2.tar.gz
  >> command completed: exit 0, ran in < 1s
  >> running command:
	[started at: 2021-09-23 11:28:28]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-11c6zkyt.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyblocks-4.4.2.tar.gz
  >> command completed: exit 0, ran in < 1s
  >> running command:
	[started at: 2021-09-23 11:28:28]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-6xc0s2mb.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyconfigs-4.4.2.tar.gz
  >> command completed: exit 0, ran in 00h00m19s
== ... (took 20 secs)
== patching...
== ... (took < 1 sec)
== preparing...
== ... (took < 1 sec)
== configuring...
== ... (took < 1 sec)
== building...
== ... (took < 1 sec)
== testing...
== ... (took < 1 sec)
== installing...
  >> running command:
	[started at: 2021-09-23 11:29:05]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-framework-4.4.2]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-tfk_os77.log]
	/usr/bin/python3 setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m06s
  >> running command:
	[started at: 2021-09-23 11:29:12]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-easyblocks-4.4.2]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-6oxd1liz.log]
	/usr/bin/python3 setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m03s
  >> running command:
	[started at: 2021-09-23 11:29:15]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-easyconfigs-4.4.2]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-jos04ikx.log]
	/usr/bin/python3 setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m55s
== ... (took 1 min 23 secs)
== taking care of extensions...
== ... (took < 1 sec)
== restore after iterating...
== ... (took < 1 sec)
== postprocessing...
== ... (took < 1 sec)
== sanity checking...
  >> running command:
	[started at: 2021-09-23 11:30:12]
	[working dir: /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2]
	[output logged in /tmp/eb-1hk_ls7r/easybuild-run_cmd-__wqx2fz.log]
	/usr/bin/python3 -c "import easybuild"
  >> command completed: exit 0, ran in < 1s
  >> file 'bin/eb' found: OK
  >> (non-empty) directory 'lib/python3.6/site-packages' found: OK
  >> loading modules: EasyBuild/4.4.2...
  >> running command 'eb --version' ...
  >> result for command 'eb --version': OK
  >> running command 'eb -a' ...
  >> result for command 'eb -a': OK
  >> running command 'eb -e ConfigureMake -a' ...
  >> result for command 'eb -e ConfigureMake -a': OK
== ... (took 4 secs)
== cleaning up...
== ... (took 8 secs)
== creating module...
  >> generating module file @ /home/ocaisa/.local/easybuild/modules/all/EasyBuild/4.4.2.lua
== ... (took < 1 sec)
== permissions...
== ... (took 16 secs)
== packaging...
== ... (took < 1 sec)
== COMPLETED: Installation ended successfully (took 2 mins 17 secs)
== Results of the build can be found in the log file(s) /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2/easybuild/easybuild-EasyBuild-4.4.2-20210923.113043.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/eb-1hk_ls7r/easybuild-ho_3iyls.log* have been removed.
== Temporary directory /tmp/eb-1hk_ls7r has been removed.

[EESSI pilot 2021.03] $ module show EasyBuild/4.4.2 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/ocaisa/.local/easybuild/modules/all/EasyBuild/4.4.2.lua:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
help([[
Description
===========
EasyBuild is a software build and installation framework
 written in Python that allows you to install software in a structured,
 repeatable and robust way.


More information
================
 - Homepage: https://easybuilders.github.io/easybuild
]])
whatis("Description: EasyBuild is a software build and installation framework
 written in Python that allows you to install software in a structured,
 repeatable and robust way.")
whatis("Homepage: https://easybuilders.github.io/easybuild")
whatis("URL: https://easybuilders.github.io/easybuild")
conflict("EasyBuild")
prepend_path("CMAKE_PREFIX_PATH","/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2")
prepend_path("PATH","/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2/bin")
setenv("EBROOTEASYBUILD","/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2")
setenv("EBVERSIONEASYBUILD","4.4.2")
setenv("EBDEVELEASYBUILD","/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2/easybuild/EasyBuild-4.4.2-easybuild-devel")
prepend_path("PYTHONPATH","/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2/lib/python3.6/site-packages")
setenv("EB_INSTALLPYTHON","/usr/bin/python3")

without the fix we get a failing installation:

[EESSI pilot 2021.03] $ EB_VERBOSE=1 EB_PYTHON=/usr/bin/python3 eb --install-latest
>> Considering '/usr/bin/python3'...
>> '/usr/bin/python3' version: 3.6.8, which matches Python 3 version requirement (>= 3.5)
>> '/usr/bin/python3' is able to import 'easybuild.main', so retaining it
>> Selected Python command: /usr/bin/python3 (/usr/bin/python3)
>> /usr/bin/python3 -m easybuild.main --install-latest
== Temporary log file in case of crash /tmp/eb-66p8t26a/easybuild-5jcmhprv.log

WARNING: Found one or more non-allowed loaded (EasyBuild-generated) modules in current environment:
* GitPython/3.1.0

This is not recommended since it may affect the installation procedure(s) performed by EasyBuild.

To make EasyBuild allow particular loaded modules, use the --allow-loaded-modules configuration option.
To specify action to take when loaded modules are detected, use --detect-loaded-modules={error,ignore,purge,unload,warn}.

See http://easybuild.readthedocs.io/en/latest/Detecting_loaded_modules.html for more information.

  >> running command:
	[started at: 2021-09-23 11:32:59]
	[working dir: /tmp/eb-66p8t26a/tmp6ogxkwno/easybuilders]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-bvzi6ir2.log]
	tar xzf /tmp/eb-66p8t26a/tmp6ogxkwno/easybuilders/develop.tar.gz
  >> command completed: exit 0, ran in < 1s

WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information

== processing EasyBuild easyconfig /tmp/eb-66p8t26a/tmp6ogxkwno/easybuilders/easybuild-easyconfigs-develop/easybuild/easyconfigs/e/EasyBuild/EasyBuild-4.4.2.eb
== building and installing EasyBuild/4.4.2...
  >> installation prefix: /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
== fetching files...
  >> sources:
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-framework-4.4.2.tar.gz [SHA256: 1e6580f210bc742ee1518d0a85389b32199b692de80548c80e93fa2a1a32dd01]
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyblocks-4.4.2.tar.gz [SHA256: 4918f13ffcd97c8b9eb0a10517bdc5b28f67c3602882d4be0bee3afc401fd78c]
  >> /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyconfigs-4.4.2.tar.gz [SHA256: 6ec2fa4d9116d45298318f31292726b6f6d1257580cff82138904c69ab38bc46]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy
== ... (took < 1 sec)
== unpacking...
  >> running command:
	[started at: 2021-09-23 11:33:02]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-ex8d_ac7.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-framework-4.4.2.tar.gz
  >> command completed: exit 0, ran in < 1s
  >> running command:
	[started at: 2021-09-23 11:33:02]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-1cdcvkrk.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyblocks-4.4.2.tar.gz
  >> command completed: exit 0, ran in < 1s
  >> running command:
	[started at: 2021-09-23 11:33:03]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-r9d9pi2m.log]
	tar xzf /home/ocaisa/.local/easybuild/sources/e/EasyBuild/easybuild-easyconfigs-4.4.2.tar.gz
  >> command completed: exit 0, ran in 00h00m19s
== ... (took 20 secs)
== patching...
== ... (took < 1 sec)
== preparing...
== ... (took 3 secs)
== configuring...
== ... (took < 1 sec)
== building...
== ... (took < 1 sec)
== testing...
== ... (took < 1 sec)
== installing...
  >> running command:
	[started at: 2021-09-23 11:33:48]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-framework-4.4.2]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-g2qklbld.log]
	/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m08s
  >> running command:
	[started at: 2021-09-23 11:33:57]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-easyblocks-4.4.2]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-w_0qlo0w.log]
	/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m03s
  >> running command:
	[started at: 2021-09-23 11:34:00]
	[working dir: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy/easybuild-easyconfigs-4.4.2]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-363ugyzo.log]
	/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python setup.py install --prefix=/home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2
  >> command completed: exit 0, ran in 00h00m57s
== ... (took 1 min 32 secs)
== taking care of extensions...
== ... (took < 1 sec)
== restore after iterating...
== ... (took < 1 sec)
== postprocessing...
== ... (took < 1 sec)
== sanity checking...
  >> running command:
	[started at: 2021-09-23 11:34:59]
	[working dir: /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2]
	[output logged in /tmp/eb-66p8t26a/easybuild-run_cmd-8r2aleqj.log]
	/cvmfs/pilot.eessi-hpc.org/2021.03/software/linux/x86_64/intel/haswell/software/Python/3.8.2-GCCcore-9.3.0/bin/python -c "import easybuild"
  >> command completed: exit 0, ran in < 1s
  >> file 'bin/eb' found: OK
  >> (non-empty) directory 'lib/python3.6/site-packages' found: FAILED
  >> loading modules: EasyBuild/4.4.2...
  >> running command 'eb --version' ...
  >> result for command 'eb --version': OK
  >> running command 'eb -a' ...
  >> result for command 'eb -a': OK
  >> running command 'eb -e ConfigureMake -a' ...
  >> result for command 'eb -e ConfigureMake -a': OK
== ... (took 5 secs)
== FAILED: Installation ended unsuccessfully (build directory: /home/ocaisa/.local/easybuild/build/EasyBuild/4.4.2/dummy-dummy): build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib/python3.6/site-packages' in /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2 (took 2 mins 1 secs)
== Results of the build can be found in the log file(s) /tmp/eb-66p8t26a/easybuild-EasyBuild-4.4.2-20210923.113301.POeiO.log
ERROR: Build of /tmp/eb-66p8t26a/tmp6ogxkwno/easybuilders/easybuild-easyconfigs-develop/easybuild/easyconfigs/e/EasyBuild/EasyBuild-4.4.2.eb failed (err: "build failed (first 300 chars): Sanity check failed: no (non-empty) directory found at 'lib/python3.6/site-packages' in /home/ocaisa/.local/easybuild/software/EasyBuild/4.4.2")

easybuild/easyblocks/e/easybuildmeta.py

boegel · 2021-10-27T20:54:48Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS EasyBuild-4.4.2.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node2602.swalot.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (haswell), Python 3.6.8
See https://gist.github.com/5ae57579dea5ca51499c49fd00584432 for a full test report.

Make sure EasyBuild easyblock respects requested version of python

773e101

ocaisa mentioned this pull request Sep 23, 2021

Migrating from EasyBuild on python2 to python3 easybuilders/easybuild-framework#3846

Closed

boegel requested changes Sep 23, 2021

View reviewed changes

easybuild/easyblocks/e/easybuildmeta.py Outdated Show resolved Hide resolved

boegel added the bug fix label Sep 23, 2021

boegel added this to the next release (4.5.0?) milestone Sep 23, 2021

ocaisa added 3 commits September 23, 2021 14:10

Don't use unnecessary conditional

d8c7d4e

Update easybuildmeta.py

f4828be

Update easybuildmeta.py

daebf61

boegel changed the title ~~Make sure EasyBuild easyblock respects requested version of python~~ make sure EasyBuild easyblock respects requested version of Python Oct 27, 2021

boegel changed the title ~~make sure EasyBuild easyblock respects requested version of Python~~ make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) Oct 27, 2021

boegel approved these changes Oct 27, 2021

View reviewed changes

boegel merged commit cf80c7e into easybuilders:develop Oct 27, 2021

ocaisa deleted the respect_python_eb_install branch October 27, 2021 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) #2581

make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) #2581

Uh oh!

ocaisa commented Sep 23, 2021 •

edited

Loading

Uh oh!

ocaisa commented Sep 23, 2021

Uh oh!

Uh oh!

boegel commented Oct 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) #2581

make sure custom easyblock for installing EasyBuild respects requested version of Python (like $EB_PYTHON) #2581

Uh oh!

Conversation

ocaisa commented Sep 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocaisa commented Sep 23, 2021

Uh oh!

Uh oh!

boegel commented Oct 27, 2021

Overview of tested easyconfigs (in order)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ocaisa commented Sep 23, 2021 •

edited

Loading