Upgraded dev env pip version

[zenmonkeykstop]

Status

Ready for review

Description of Changes

Fixes #5000.

Upgrades pip in the dev env before installing dependencies.

Testing

• check out this branch
• (optional) clear existing docker images with docker system prune
• run make dev - confirm that no errors relating to setup.py bdist_wheel are displayed.

Deployment

Dev-only change, will be updated on developer machines when make dev executed.

[emkll]

Thanks @zenmonkeykstop it seems like we do pin an old version of pip in https://github.com/freedomofpress/securedrop/blob/develop/securedrop/requirements/python3/develop-requirements.in#L15-L16

Would it make sense to update the requirements file(s) instead by pinning a new version or unpinning the pip version from develop-requirements.in ? This would ensure these benefits are applied to CI (see the related #4902) and that we pin specific versions and hashes of pip and it's dependencies.

[redshiftzero]

For background we pinned pip due to incompatibility for (at the time) the most recent version of pip with pipenv (#3853) (similar compatibility issues can happen with pip-tools also), pinning to a newer version of pip that we've confirmed make update-pip-requirements works with will cause the least breakage

[zenmonkeykstop]

Pinned version is 19.1, that would probably work fine (the version on dev envs without upgrade is ancient). Will add it to test-requirements.in and update hashes accordingly.

(Actually that means that test-requirements.in's deps are installed with system pip (8.1.1) so there are still some bdist_wheel errors.)

[zenmonkeykstop]

Ended up adding a separate docker-requirements.in here instead specifically to update pip and setuptools.

[redshiftzero]

hmm update-python3-requirements is failing for me due to the following error (I wanted to test this to verify no pip/pip-tools incompatibility) - did this work for you?

$ make update-python3-requirements
███ Updating Python 3 requirements files...
Complete output from command /opt/venvs/securedrop-app-code/bin/python3 /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpjqx_9gce:
running dist_info
creating /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info
writing /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/PKG-INFO
writing top-level names to /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/top_level.txt
writing entry points to /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/entry_points.txt
writing requirements to /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/requires.txt
writing dependency_links to /tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/dependency_links.txt
writing manifest file '/tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/SOURCES.txt'
reading manifest file '/tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/SOURCES.txt'
writing manifest file '/tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.egg-info/SOURCES.txt'
creating '/tmp/tmpzxgoug_ybuild/pytest-xdist/pip-wheel-metadata/pytest_xdist.dist-info'
error: invalid command 'bdist_wheel'
----------------------------------------
Traceback (most recent call last):
  File "/opt/venvs/securedrop-app-code/bin/pip-compile", line 10, in <module>
    sys.exit(cli())
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/scripts/compile.py", line 345, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/resolver.py", line 164, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/resolver.py", line 259, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/resolver.py", line 354, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/repositories/local.py", line 70, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 254, in get_dependencies
    download_dir, ireq, wheel_cache
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 213, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 368, in prepare_linked_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/operations/prepare.py", line 177, in prep_for_dist
    self.req.prepare_metadata()
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 537, in prepare_metadata
    self.prepare_pep517_metadata()
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 582, in prepare_pep517_metadata
    metadata_dir
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_vendor/pep517/wrappers.py", line 86, in prepare_metadata_for_build_wheel
    'config_settings': config_settings,
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_vendor/pep517/wrappers.py", line 155, in _call_hook
    extra_environ={'PEP517_BUILD_BACKEND': build_backend}
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 520, in runner
    spinner=spinner
  File "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "/opt/venvs/securedrop-app-code/bin/python3 /opt/venvs/securedrop-app-code/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpjqx_9gce" failed with error code 1 in /tmp/tmpzxgoug_ybuild/pytest-xdist
make: *** [update-python3-requirements] Error 1

[zenmonkeykstop]

What's the pip version in your virtualenv immediately before running said command?

[redshiftzero]

just checked, it's pip==19.1 - you don't see the same error I guess? let me know if not and I can do some more digging

[zenmonkeykstop]

Actually never mind, I see it too. I'll take a look.

[zenmonkeykstop]

Added wheel to installed docker packages, which allows make update-python3-requirements to pass again. Apologies @redshiftzero, I could have sworn that was working for me without it.

[redshiftzero]

no worries, lgtm now, thanks!