Support poetry, flit, pipenv, or ...?

[stevepiercy]

This is a placeholder for supporting whatever becomes the successor to pip and managing virtual environments in Pyramid, including poetry, flit, and pipenv. Please add to it.

Questions

1. How and where to install SUCCESSOR?
2. Where does installation of SUCCESSOR fit into our current installation documentation?
3. Are there any features of SUCCESSOR that we would use beside installation of packages and distribution of an app?
4. What would happen to the environment variable $VENV in most of our commands?
5. How do we deal with features in setup.py only, specifically entry points of console_scripts and paster.app_factory?
6. What will we do with the scaffolds and pcreate?
7. Should we remove instructions for how to make a distributable Pyramid application, and replace it with references to suggested tools?
8. Assuming a rewrite of commands for SUCCESSOR, do we retain or remove Windows commands for everything except installation and setup of an environment, with a disclaimer that it is up to the user to know how to translate Unix commands to their platform of choice? Refs: Tutorial instructions for windows may not work correctly in Powershell #3260 (comment)

Things to update

1. Documentation wherever pip is used.
2. Pyramid's packaging related files, including setup.py.
3. Pyramid cookiecutters.

References

1. Poetry: https://github.com/sdispater/poetry
2. Summary article about pipenv: https://realpython.com/pipenv-guide/

[mmerickel]

The main issue I have seen in my experience using pipenv is that it doesn't mix very cleanly with a setup.py. It's possible but you run into questions about where to define the dependencies. I have one project where the Pipfile is literally just a shim pointing to the setup.py where all the deps were declared. This works pretty well actually but it does feel odd.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6.5"

[packages]
myapp = {path = ".", editable = true}

[dev-packages]
myapp = {path = ".", extras = ["testing"], editable = true}

The workflow here is to basically edit setup.py and then run pipenv lock which updates Pipfile.lock and you never really change the Pipfile.

The main reasons why we use setup.py in our projects is for console_scripts and paster.app_factory entry points. If a) python had a better way to define entry points, or b) if we were less dependent on them, then we could probably embrace pipenv a little bit more cleanly.

[stevepiercy]

What would be a better way to define entry points? Is the PyPA working on this?

This does not bode well:
https://docs.pipenv.org/search/?q=entry+point

[mmerickel]

Entry points is an issue that pypa is starting to talk about on distutils-sig as the next likely thing to get split out from setuptools... but very slowly. The more practical option I think will be to rip setup.py out of the guide entirely and just show people how to make pyramid apps without needing to make a distributable python package. Lots of projects seem to like that approach - see warehouse for an example.

[stevepiercy]

I've updated and added to the questions above.

Regarding making a distributable Python package, do you mean you want to remove the these two pages?

• https://docs.pylonsproject.org/projects/pyramid/en/latest/tutorials/wiki2/distributing.html
• https://docs.pylonsproject.org/projects/pyramid/en/latest/tutorials/wiki/distributing.html

If so, I'd like to at least mention why someone would want to make a dist and provide references to resources for how to do so, instead of simply deleting those pages from the two wiki tutorials.

Or do you mean something else?

[piotr-dobrogost]

Maybe https://github.com/takluyver/entrypoints could be used instead of setuptools/pkg_resources for entry points?

[mmerickel]

@piotr-dobrogost Unfortunately the issue here is how to declare entry points, not how to consume them. That project only looks to consume existing *.egg-info/entry_points.txt files in sys.path.

[piotr-dobrogost]

Ok, how about switching from setuptools to flit which does have support for entry points (https://flit.readthedocs.io/en/latest/pyproject_toml.html#entry-points-sections)?

[mmerickel]

I like flit and would be open to promoting it more. Unfortunately for the purposes of integrating with pipenv it's basically the same as setup.py - it's making your project a redistributable package and thus provides a place to define dependencies which conflicts with the Pipfile. We would need to make a decision where the dependencies are defined. For example I think you'd end up with a Pipfile like I pasted above.

The alternative that integrates with pipenv more cleanly is the flask approach of punting on this and not advocating patterns that use setup.py. You're basically responsible yourself for putting your project on the PYTHONPATH however you want - usually just running it out of the CWD. The real benefits to having an installable package are:

1. Your project, even in editable mode, can be run from anywhere. You can import it and thanks to pth files it's always on the path.
2. Your project has a way to describe dependencies.
3. Your project has a way to provide implementations for entry points.

pipenv's approach (in my experience using it) has been to directly compete with point 2 and doesn't provide a solution for points 1 and 3. It seems mostly useful for pulling together third party projects into a lockfile and less about managing the link between the project-under-development and the virtualenv. Please correct me if anyone disagrees, but this is what I've seen, and yes I have used it for a real project in this way.

[mmerickel]

https://github.com/sdispater/poetry is by far the best solution I have seen to managing a project yet. I think it blows pipenv's approach out of the water from what I have seen so far and solves all the issues I've mentioned above.

[merwok]

Compared to flit for example, it seems like poetry is made by one person who does not follow Python packaging discussions, so I’m not sure about the degree of integration (i.e. implementation of standards) that this tool provides. For example, the author doesn’t see the point of develop installs, and doesn’t accept the uses cases brought up in the ticket.

[digitalresistor]

@merwok there are a variety of projects that don't use develop installs at all, for example warehouse doesn't instead it expects warehouse to be in it's PYTHONPATH automatically. Others like Flask and I think Django too, punt on having installable packages, unless you manually do a bunch of work (its not documented, and not the default recommendation) and thus they don't have a need for editable installs.

While we have promoted setup.py and having installable packages for Pyramid applications it is not a requirement. The author also agrees that there are some use cases for editable installs: python-poetry/poetry#34 (comment) and seems amenable to adding them for dependencies.

I personally really like the idea of a single file that contains package pins/Information rather than a "Pipfile" vs "setup.py" (or one of it's replacements) where the data is duplicated, or you end up having on rely on the other.

Using flit doesn't do a bunch of the project management things you might have come to expect from tools like npm/yarn.

[mmerickel]

Right... poetry solves the editable install issue by basically requiring you to use poetry run or poetry scripts for your commands during development... which, coming from nodejs with yarn run and npm run is actually a really nice thing in my opinion.

Finally, the sdist/wheel it outputs if you do choose to poetry build does contain the appropriate metadata for pip to install it correctly in anyone else's environment. With this in mind I've re-thought the benefits of an editable install entirely. For example, it's a pretty common problem for people to forget to do a pip install -e . after changing entry points. This doesn't suffer from that issue because it makes sure they're correct each time you run your commands in development mode.

On top of that, I guess that as @bertjwregeer said, poetry is adding some editable install support in a future version anyway.