Skip to content

add support for detecting auto-downloaded dependencies in PythonPackage easyblock #1377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vanzod merged 3 commits into from
May 17, 2018
Merged

add support for detecting auto-downloaded dependencies in PythonPackage easyblock #1377

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7a9f8f1
add support for detecting auto-downloaded dependencies in PythonPacka…
boegel Mar 6, 2018
ac2f9ff
move checking for downloaded dependencies into sanity check of Python…
boegel Apr 29, 2018
985a4ae
Merge branch 'develop' into pythonpackage_download_dep_fail
boegel May 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions easybuild/easyblocks/generic/pythonpackage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,17 +178,18 @@ def extra_options(extra_vars=None):
if extra_vars is None:
extra_vars = {}
extra_vars.update({
'unpack_sources': [True, "Unpack sources prior to build/install", CUSTOM],
'buildcmd': ['build', "Command to pass to setup.py to build the extension", CUSTOM],
'download_dep_fail': [False, "Fail if downloaded dependencies are detected", CUSTOM],
Copy link
Member Author

@boegel boegel Mar 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set to False for backward-compatibility, but we should consider switching this to True by default for EasyBuild v4.0.0.

Problem is that we'll need to fix quite a bit of easyconfigs...

We could try to enforce a policy where new easyconfigs for Python packages include download_dep_fail = True?

'install_target': ['install', "Option to pass to setup.py", CUSTOM],
'req_py_majver': [2, "Required major Python version (only relevant when using system Python)", CUSTOM],
'req_py_minver': [6, "Required minor Python version (only relevant when using system Python)", CUSTOM],
'runtest': [True, "Run unit tests.", CUSTOM], # overrides default
'unpack_sources': [True, "Unpack sources prior to build/install", CUSTOM],
'use_easy_install': [False, "Install using '%s' (deprecated)" % EASY_INSTALL_INSTALL_CMD, CUSTOM],
'use_pip': [False, "Install using '%s'" % PIP_INSTALL_CMD, CUSTOM],
'use_pip_for_deps': [False, "Install dependencies using '%s'" % PIP_INSTALL_CMD, CUSTOM],
'use_setup_py_develop': [False, "Install using '%s' (deprecated)" % SETUP_PY_DEVELOP_CMD, CUSTOM],
'zipped_egg': [False, "Install as a zipped eggs (requires use_easy_install)", CUSTOM],
'buildcmd': ['build', "Command to pass to setup.py to build the extension", CUSTOM],
'install_target': ['install', "Option to pass to setup.py", CUSTOM],
})
return ExtensionEasyBlock.extra_options(extra_vars=extra_vars)

Expand Down Expand Up @@ -478,7 +479,18 @@ def install_step(self):

# actually install Python package
cmd = self.compose_install_command(self.installdir)
run_cmd(cmd, log_all=True, simple=True)
(out, ec) = run_cmd(cmd, log_all=True, simple=False)

if self.cfg['download_dep_fail']:
self.log.info("Detection of downloaded depenencies enabled, checking output of '%s'...", cmd)
patterns = [
'Downloading.*pypi.*packages.*', # setuptools
'Collecting .* \(from.*', # pip
]
for pattern in patterns:
res = re.compile(pattern, re.M).findall(out)
if res:
raise EasyBuildError("Found one or more downloaded dependencies: %s", res)

# restore PYTHONPATH if it was set
if pythonpath is not None:
Expand Down