Skip to content

Fix dist_utils.py so it doesn't rely on internal pip API #4750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 29, 2019
Merged

Fix dist_utils.py so it doesn't rely on internal pip API #4750

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a2744a2
Update dist_utils.py so it doesn't depend on internal pip methods.
Kami Jul 28, 2019
7f7cdab
Add tests for dist_utils.fetch_requirements.
Kami Jul 28, 2019
4fb6842
Copy over latest dist_utils.py.
Kami Jul 28, 2019
a3aa063
Add changelog entry.
Kami Jul 28, 2019
82bb59e
Fix links handling, add support for non-git VCS requirements, add
Kami Jul 28, 2019
d713f38
Copy over latest dist_utils.py.
Kami Jul 28, 2019
1011b6b
Add additional test case with top-level requirements.txt.
Kami Jul 28, 2019
6d6384a
Add support for editable links and lines with &subdirectory=foo pragma.
Kami Jul 28, 2019
893b305
Copy over latest dist_utils.py.
Kami Jul 28, 2019
9d63f85
Include auto-generated warning in the copied over files.
Kami Jul 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ Fixed
completed. (bug fix) #4735
* Added better error handling to `contrib/linux/actions/dig.py` to inform if dig is not installed.
Contributed by JP Bourget (@punkrokk Syncurity) #4732
* Update ``dist_utils`` module which is bundled with ``st2client`` and other Python packages so it
doesn't depend on internal pip API and so it works with latest pip version. (bug fix) #4750

3.1.0 - June 27, 2019
---------------------
Expand Down
1 change: 1 addition & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,6 +847,7 @@ debs:
# Copy over shared dist utils module which is needed by setup.py
@for component in $(COMPONENTS_WITH_RUNNERS); do\
cp -f ./scripts/dist_utils.py $$component/dist_utils.py;\
sed -i -e '1s;^;# NOTE: This file is auto-generated - DO NOT EDIT MANUALLY\n;' $$component/dist_utils.py;\
done

# Copy over CHANGELOG.RST, CONTRIBUTING.RST and LICENSE file to each component directory
Expand Down
59 changes: 39 additions & 20 deletions contrib/runners/action_chain_runner/dist_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# NOTE: This file is auto-generated - DO NOT EDIT MANUALLY
# -*- coding: utf-8 -*-
# Copyright 2019 Extreme Networks, Inc.
#
Expand Down Expand Up @@ -27,31 +28,18 @@
if PY3:
text_type = str
else:
text_type = unicode
text_type = unicode # NOQA

GET_PIP = 'curl https://bootstrap.pypa.io/get-pip.py | python'

try:
import pip
from pip import __version__ as pip_version
except ImportError as e:
print('Failed to import pip: %s' % (text_type(e)))
print('')
print('Download pip:\n%s' % (GET_PIP))
sys.exit(1)

try:
# pip < 10.0
from pip.req import parse_requirements
except ImportError:
# pip >= 10.0

try:
from pip._internal.req.req_file import parse_requirements
except ImportError as e:
print('Failed to import parse_requirements from pip: %s' % (text_type(e)))
print('Using pip: %s' % (str(pip_version)))
sys.exit(1)

__all__ = [
'check_pip_version',
Expand Down Expand Up @@ -80,12 +68,43 @@ def fetch_requirements(requirements_file_path):
"""
links = []
reqs = []
for req in parse_requirements(requirements_file_path, session=False):
# Note: req.url was used before 9.0.0 and req.link is used in all the recent versions
link = getattr(req, 'link', getattr(req, 'url', None))
if link:
links.append(str(link))
reqs.append(str(req.req))

def _get_link(line):
Copy link
Member

@arm4b arm4b Jul 29, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering how error-prone is that code block?
Did you take the rules from pip upstream or it's something else you consulted for code logic?

Copy link
Member Author

@Kami Kami Jul 29, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't check pip code itself, but I compared the results and consulted high level documentation on supported requirements format (which is the only thing I care about at this point - the function output).

If you check the tests, you will see we have tests which compare the result of our version and pip version to ensure they are the same.

It's likely there are still some edge cases somewhere, but it's not a big deal at this point since for our current requirements.txt, the end result is the same.

Copy link
Contributor

@blag blag Mar 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code did not support environment markers from pip, so I had to add (hacky) support for them in #4895.

@Kami Why can the dist_utils.py module not use any third party dependencies?

# NOTE: This script can't rely on any 3rd party dependency so we need to use this code here

Instead of importing from pip directly (which we should never have been doing in the first place, IMO), if we can use third party dependencies, we can utilize the packaging package. It has an entire section devoted to reading and parsing requirements.txt files, and that would be kept up-to-date with what pip and setuptools support, and would lower our maintenance burden of this code.

vcs_prefixes = ['git+', 'svn+', 'hg+', 'bzr+']

for vcs_prefix in vcs_prefixes:
if line.startswith(vcs_prefix) or line.startswith('-e %s' % (vcs_prefix)):
req_name = re.findall('.*#egg=(.+)([&|@]).*$', line)

if not req_name:
req_name = re.findall('.*#egg=(.+?)$', line)
else:
req_name = req_name[0]

if not req_name:
raise ValueError('Line "%s" is missing "#egg=<package name>"' % (line))

link = line.replace('-e ', '').strip()
return link, req_name[0]

return None, None

with open(requirements_file_path, 'r') as fp:
for line in fp.readlines():
line = line.strip()

if line.startswith('#') or not line:
continue

link, req_name = _get_link(line=line)

if link:
links.append(link)
else:
req_name = line

reqs.append(req_name)

return (reqs, links)


Expand Down
59 changes: 39 additions & 20 deletions contrib/runners/announcement_runner/dist_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# NOTE: This file is auto-generated - DO NOT EDIT MANUALLY
# -*- coding: utf-8 -*-
# Copyright 2019 Extreme Networks, Inc.
#
Expand Down Expand Up @@ -27,31 +28,18 @@
if PY3:
text_type = str
else:
text_type = unicode
text_type = unicode # NOQA

GET_PIP = 'curl https://bootstrap.pypa.io/get-pip.py | python'

try:
import pip
from pip import __version__ as pip_version
except ImportError as e:
print('Failed to import pip: %s' % (text_type(e)))
print('')
print('Download pip:\n%s' % (GET_PIP))
sys.exit(1)

try:
# pip < 10.0
from pip.req import parse_requirements
except ImportError:
# pip >= 10.0

try:
from pip._internal.req.req_file import parse_requirements
except ImportError as e:
print('Failed to import parse_requirements from pip: %s' % (text_type(e)))
print('Using pip: %s' % (str(pip_version)))
sys.exit(1)

__all__ = [
'check_pip_version',
Expand Down Expand Up @@ -80,12 +68,43 @@ def fetch_requirements(requirements_file_path):
"""
links = []
reqs = []
for req in parse_requirements(requirements_file_path, session=False):
# Note: req.url was used before 9.0.0 and req.link is used in all the recent versions
link = getattr(req, 'link', getattr(req, 'url', None))
if link:
links.append(str(link))
reqs.append(str(req.req))

def _get_link(line):
vcs_prefixes = ['git+', 'svn+', 'hg+', 'bzr+']

for vcs_prefix in vcs_prefixes:
if line.startswith(vcs_prefix) or line.startswith('-e %s' % (vcs_prefix)):
req_name = re.findall('.*#egg=(.+)([&|@]).*$', line)

if not req_name:
req_name = re.findall('.*#egg=(.+?)$', line)
else:
req_name = req_name[0]

if not req_name:
raise ValueError('Line "%s" is missing "#egg=<package name>"' % (line))

link = line.replace('-e ', '').strip()
return link, req_name[0]

return None, None

with open(requirements_file_path, 'r') as fp:
for line in fp.readlines():
line = line.strip()

if line.startswith('#') or not line:
continue

link, req_name = _get_link(line=line)

if link:
links.append(link)
else:
req_name = line

reqs.append(req_name)

return (reqs, links)


Expand Down
59 changes: 39 additions & 20 deletions contrib/runners/http_runner/dist_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# NOTE: This file is auto-generated - DO NOT EDIT MANUALLY
# -*- coding: utf-8 -*-
# Copyright 2019 Extreme Networks, Inc.
#
Expand Down Expand Up @@ -27,31 +28,18 @@
if PY3:
text_type = str
else:
text_type = unicode
text_type = unicode # NOQA

GET_PIP = 'curl https://bootstrap.pypa.io/get-pip.py | python'

try:
import pip
from pip import __version__ as pip_version
except ImportError as e:
print('Failed to import pip: %s' % (text_type(e)))
print('')
print('Download pip:\n%s' % (GET_PIP))
sys.exit(1)

try:
# pip < 10.0
from pip.req import parse_requirements
except ImportError:
# pip >= 10.0

try:
from pip._internal.req.req_file import parse_requirements
except ImportError as e:
print('Failed to import parse_requirements from pip: %s' % (text_type(e)))
print('Using pip: %s' % (str(pip_version)))
sys.exit(1)

__all__ = [
'check_pip_version',
Expand Down Expand Up @@ -80,12 +68,43 @@ def fetch_requirements(requirements_file_path):
"""
links = []
reqs = []
for req in parse_requirements(requirements_file_path, session=False):
# Note: req.url was used before 9.0.0 and req.link is used in all the recent versions
link = getattr(req, 'link', getattr(req, 'url', None))
if link:
links.append(str(link))
reqs.append(str(req.req))

def _get_link(line):
vcs_prefixes = ['git+', 'svn+', 'hg+', 'bzr+']

for vcs_prefix in vcs_prefixes:
if line.startswith(vcs_prefix) or line.startswith('-e %s' % (vcs_prefix)):
req_name = re.findall('.*#egg=(.+)([&|@]).*$', line)

if not req_name:
req_name = re.findall('.*#egg=(.+?)$', line)
else:
req_name = req_name[0]

if not req_name:
raise ValueError('Line "%s" is missing "#egg=<package name>"' % (line))

link = line.replace('-e ', '').strip()
return link, req_name[0]

return None, None

with open(requirements_file_path, 'r') as fp:
for line in fp.readlines():
line = line.strip()

if line.startswith('#') or not line:
continue

link, req_name = _get_link(line=line)

if link:
links.append(link)
else:
req_name = line

reqs.append(req_name)

return (reqs, links)


Expand Down
59 changes: 39 additions & 20 deletions contrib/runners/inquirer_runner/dist_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# NOTE: This file is auto-generated - DO NOT EDIT MANUALLY
# -*- coding: utf-8 -*-
# Copyright 2019 Extreme Networks, Inc.
#
Expand Down Expand Up @@ -27,31 +28,18 @@
if PY3:
text_type = str
else:
text_type = unicode
text_type = unicode # NOQA

GET_PIP = 'curl https://bootstrap.pypa.io/get-pip.py | python'

try:
import pip
from pip import __version__ as pip_version
except ImportError as e:
print('Failed to import pip: %s' % (text_type(e)))
print('')
print('Download pip:\n%s' % (GET_PIP))
sys.exit(1)

try:
# pip < 10.0
from pip.req import parse_requirements
except ImportError:
# pip >= 10.0

try:
from pip._internal.req.req_file import parse_requirements
except ImportError as e:
print('Failed to import parse_requirements from pip: %s' % (text_type(e)))
print('Using pip: %s' % (str(pip_version)))
sys.exit(1)

__all__ = [
'check_pip_version',
Expand Down Expand Up @@ -80,12 +68,43 @@ def fetch_requirements(requirements_file_path):
"""
links = []
reqs = []
for req in parse_requirements(requirements_file_path, session=False):
# Note: req.url was used before 9.0.0 and req.link is used in all the recent versions
link = getattr(req, 'link', getattr(req, 'url', None))
if link:
links.append(str(link))
reqs.append(str(req.req))

def _get_link(line):
vcs_prefixes = ['git+', 'svn+', 'hg+', 'bzr+']

for vcs_prefix in vcs_prefixes:
if line.startswith(vcs_prefix) or line.startswith('-e %s' % (vcs_prefix)):
req_name = re.findall('.*#egg=(.+)([&|@]).*$', line)

if not req_name:
req_name = re.findall('.*#egg=(.+?)$', line)
else:
req_name = req_name[0]

if not req_name:
raise ValueError('Line "%s" is missing "#egg=<package name>"' % (line))

link = line.replace('-e ', '').strip()
return link, req_name[0]

return None, None

with open(requirements_file_path, 'r') as fp:
for line in fp.readlines():
line = line.strip()

if line.startswith('#') or not line:
continue

link, req_name = _get_link(line=line)

if link:
links.append(link)
else:
req_name = line

reqs.append(req_name)

return (reqs, links)


Expand Down
Loading