setuptools <49.0.0 does not handle Python 3.10 correctly in `python_version` environment markers

[sileht]

The newly added python 3.10 doesn't work with pip 20 and setuptools 47

old pip/setuptools version thinks python 3.10 is python 3.1:

• https://bugs.python.org/issue42744
• python3.10: environment markers assume python 3.1 pypa/pip#6730
• bpo-43077: Update bundled pip to 21.0.1 and setuptools to 52.0.0 python/cpython#24386

Python 3.10 requires pip>=21 and setuptools>=52 (https://github.com/python/cpython/blob/3.10/Lib/ensurepip/__init__.py)

[edmorley]

@sileht Hi! Thank you for opening an issue :-)

The Python buildpack doesn't use ensurepip to install the versions of Pip or setuptools in the build for a few reasons, and instead installs the version of Pip/setuptools defined here:

heroku-buildpack-python/bin/steps/python

Lines 153 to 167 in ec2fd78

	PIP_VERSION='20.2.4'
	# Must use setuptools <47.2.0 until we fix:
	# https://github.com/heroku/heroku-buildpack-python/issues/1006
	SETUPTOOLS_VERSION='47.1.1'
	WHEEL_VERSION='0.36.2'
	if [[ "${PYTHON_VERSION}" == ${PY34}* ]]; then
	# Python 3.4 support was dropped in pip 19.2+, setuptools 44+ and wheel 0.34+.
	PIP_VERSION='19.1.1'
	SETUPTOOLS_VERSION='43.0.0'
	WHEEL_VERSION='0.33.6'
	elif [[ "${PYTHON_VERSION}" == ${PY27}* || "${PYTHON_VERSION}" == ${PYPY27}* ]]; then
	# Python 2.7 support was dropped in setuptools 45+.
	SETUPTOOLS_VERSION='44.1.1'
	fi

We're currently using Pip 20.2.4 since it's the last release before the new resolver became the default, and there are still several compatibility issues that make upgrading to newer pip painful (more in #1109). Thankfully the upcoming Pip 21.3 release (due within days, though we will want to wait for any dust to settle) resolves the most significant of those, so we should be good to update soon.

For setuptools, we're currently using 47.1.1, since 47.2.0 and newer are affected by #1006. The long term fix for that is changing the way the build system works, so that the directory in which builds occur is the same as that which is used at runtime. However doing that breaks some other buildpacks, so is not a quick change (and that work has had to be put on hold).

Re Python 3.10 specifically, do you have some examples of things that are failing currently using the above pip/setuptools versions?

Our CI is currently passing, and the pip issue linked in the issue description was fixed in pip 19.3, so shouldn't affect Python 3.10 usage on Heroku.

Many thanks :-)

[sileht]

Yeah, I'm aware the buildpack doesn't use ensurepip, I just put the link as a reference for the recommended version.

Here is an example of a backtrace, you have some others in bug reports I posted in the description.

Traceback (most recent call last):
  File "/app/.heroku/python/bin/mergifyio-import-check", line 33, in <module>
    sys.exit(load_entry_point('mergifyio', 'console_scripts', 'mergifyio-import-check')())
  File "/app/mergifyio/import_check.py", line 6, in check
    from mergifyio.web import app as application  # noqa
  File "/app/mergifyio/web.py", line 18, in <module>
    from mergifyio.blueprints import badges
  File "/app/mergifyio/blueprints/badges.py", line 5, in <module>
    import pkg_resources
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3262, in <module>
    def _initialize_master_working_set():
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pep562; python_version < "3.7"' distribution was not found and is required by ddtrace

[sileht]

Our CI is currently passing, and the pip issue linked in the issue description was fixed in pip 19.3, so shouldn't affect Python 3.10 usage on Heroku.

Does your CI install any libraries that use python environment markers in requirements? if not that may explain why you didn't trigger the issue.

[sileht]

Hi @edmorley, maybe you can run the tests on the pull request I send, it should trigger the issue.

[edmorley]

@sileht Hi! Sorry for the delay and thank you for the PR to try and reproduce.

I have triggered CI on that PR, however it doesn't show the issue - the pep562 package is correctly ignored:

remote: -----> Building on the Heroku-20 stack
remote: -----> Using buildpack: https://github.com/heroku/heroku-buildpack-python.git#pr/1249
remote: -----> Python app detected
remote: -----> Using Python version specified in runtime.txt
remote: -----> Installing python-3.10.0
remote: -----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Ignoring pep562: markers 'python_version < "3.7"' don't match your environment
remote:        Collecting urllib3
remote:          Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
remote:        Installing collected packages: urllib3
remote:        Successfully installed urllib3-1.26.7
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)

See:
https://app.circleci.com/pipelines/github/heroku/heroku-buildpack-python/218/workflows/a7eaaeb1-9fb1-420b-bb50-bbb6b1e2f97a/jobs/933

Looking at the traceback in the earlier comment, I see it's originating via pkg_resources, which is part of setuptools - which may suggest an issue with setuptools rather than pip, and so why the requirements.txt file based testcase wasn't affected.

Could you open a support ticket (and give app access permission, and link to the ticket from here) so I can take a closer look at your app to try and work out a reduced testcase that demonstrates the issue?

[edmorley]

I was able to reproduce when using:

• Python 3.10
• A requirements.txt file containing an editable local package specifier (eg -e ./local_package/)
• The local package containing a setup.py / setup.cfg with an install_requires of pep562; python_version < "3.7" and an entrypoint set
• A bin/post_compile that invokes said entrypoint

Example failure log:

$ make compile FIXTURE=spec/fixtures/requirements_editable/
Running compile using: STACK=heroku-20 FIXTURE=spec/fixtures/requirements_editable/

-----> Using Python version specified in runtime.txt
-----> Installing python-3.10.0
-----> Installing pip 20.2.4, setuptools 47.1.1 and wheel 0.36.2
-----> Installing SQLite3
-----> Installing requirements with pip
       Obtaining file:///build/local_package (from -r /build/requirements.txt (line 1))
       Obtaining gunicorn from git+https://github.com/benoitc/gunicorn@20.1.0#egg=gunicorn (from -r /build/requirements.txt (line 2))
         Cloning https://github.com/benoitc/gunicorn (to revision 20.1.0) to /app/.heroku/src/gunicorn
       Installing collected packages: gunicorn, local-package
         Running setup.py develop for gunicorn
         Running setup.py develop for local-package
       Successfully installed gunicorn local-package
-----> Running post-compile hook
Running entrypoint for the local package: Traceback (most recent call last):
  File "/app/.heroku/python/bin/local_package", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3262, in <module>
    def _initialize_master_working_set():
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3245, in _call_aside
    f(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3274, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/app/.heroku/python/lib/python3.10/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pep562' distribution was not found and is required by local-package

Upgrading setuptools to 49.0.0 or newer resolved the issue - I believe this is the change that fixed it:
pypa/setuptools@a877dab

I'll prioritise finding a shorter term fix for #1006 (compared to the ideal fix which is further out), so we can finally update setuptools.

@sileht In the meantime, could you try adding setuptools>=49.0.0 to the top of your requirements.txt to see if that work around the issue?

[sileht]

I setup a tiny project that reproduces the issue, too, but it looks like you just beat me 😛

https://github.com/sileht/heroku-py3.10-bug

Anyways, upgrading setuptools, workaround the issue.

[edmorley]

Thank you! I'll look into this more tomorrow :-)