Building c-extension wheels for multiple python versions

[gaborbernat]

Assume you have a c-extension project (something with cython), that builds a wheel only for the current python major/minor python version. Currently building wheels for all supported python versions (e.g. CPython 3.6, 3.7, 3.8, 3.9 is very tedious with the build project). You have to create four virtual environments, install the build four times and invoke it four times (make sure your build executable names pick up the right host python version). I propose we allow a mode to build these set of wheels in a single command. Something like:

python -m build --wheel py36,py37,py38,py39 --sdist .

The good news is implementing this is simple with what we have, if we make this use case require the virtualenv extra. All we have to do is to invoke the build in a loop and pass on these python specification strings to virtualenv here. I think it's alright to support non-host python targets with the virtualenv extra only (essentially making the default --wheel py).

@FFY00 @layday @henryiii thoughts? If we agree I can do the implementation.

[henryiii]

Building proper wheels is tricky; I'm afraid this would get misused. If someone wants multiple wheels they likely are going to try to distribute them, and then they should probably use cibuildwheel, which handles building in a manylinux container (on linux), using a macOS 10.9 compatible Python (macOS), dealocating / audit wheel, universal wheels, testing, etc. pypa/build is currently being integrated into cibuildwheel as an option; it was recently added to manylinux for that purpose.

[gaborbernat]

Building proper wheels is tricky; I'm afraid this would get misused.

Can you please list proper examples of such worries? I don't want to talk about hypothetical.

should probably use cibuildwheel

In lots of places (e.g. behind enterprise firewalls), this is just not possible. Let's not consider the cibuildwheel the solution, is a solution for some but I don't think generalizes correctly. If you go by that logic we should remove the --wheel flag because building wheels is tricky to get right. My use case is something someone can do today, I just was hoping we can make it less painful.

[henryiii]

Can you please list proper examples of such worries?

I thought I did? Manylinux needs to be built inside the manylinux container, macOS needs the official Python downloads to support 10.9+ (or generally anything older than the current macOS version, which is what most other Pythons are built for), cross compiling for Apple Silicon/Universal2 requires various workarounds, delocating/auditing for distribution, etc.

If we remove the --wheel flag, then what about Pure Python wheels, which is currently probably the main use of pypa/build? What about using this in a well prepared environment (such as cibuildwheel or multibuild, or your company environment, or your current environment for your own use, or your package manager environment)? All of these are perfectly valid uses of --wheel.

If we support python -m build --wheel py36,py37,py38,py39, I can easily see users expecting this to produce binary wheels ready to be uploaded to PyPI. This will not be the case on macOS or Linux (though on Windows it might work pretty well). It's not build's job to handle all the trickery involved in building wheels to be distributed.

This also bothers me because it detaches the Python version from the version of Python running build, relying on a search procedure (from virtualenv, I assume). When building a binary wheel, which Python can be important; "any" copy of Python x.y might not do (esp on macOS).

Just to be clear, I'm not strongly opposed, I'm just voicing my concerns that this feature might cause more harm than good. "just passing through" these to virtualenv seems pretty simple. It is a little irritating that this would only work if you activate the virtualenv extra; this extra is also not available in manylinux (except for manylinux1 Python 2.7), which is arguably one of the most important places to have this feature. Also nothing is installed into the default Python in manylinux, making this a bit weird to use.

[henryiii]

If implemented, I think I would rather something explicit and without a loop, like --virtualenv=py36 or something like that; putting the loop inside build is not really necessary, I think, and might help with my concern of this being too easy to use for the wrong purpose. Also is more obvious it requires virtualenv, and more explicit.

[gaborbernat]

I thought I did? Manylinux needs to be built inside the manylinux container, macOS needs the official Python downloads to support 10.9+ (or generally anything older than the current macOS version, which is what most other Pythons are built for), cross compiling for Apple Silicon/Universal2 requires various workarounds, delocating/auditing for distribution, etc.

You're assuming here that people want to build manylinux wheels. Often you don't need them or are straight up a bad idea (when you can ensure that your target deploy environment will contain the dynamically linked libraries it's actually preferable to not bundle them). Often you build wheels to target your current OS and that's it. I don't view build project as the project to use to build wheels to upload to PyPi. build is a project to build sdist/wheel, a frontend to PEP-517. If you want manylinux/deallocated wheels then you probably still need build to build the base wheel and then customize it. But again, build is not a PyPi wheel builder, it's a generic PEP-517 frontend, and should not worry about PyPi.

If we support python -m build --wheel py36,py37,py38,py39, I can easily see users expecting this to produce binary wheels ready to be uploaded to PyPI.

I don't think this is related to this topic. Users can already assume today that --wheel will build a PyPi ready to upload wheel even if it's a c-extension wheel, which clearly does not. At no point, we check if the wheel is not a pure python wheel and raise otherwise to shield users from hurting themselves. So while I understand your worries, those worries are a thing today too, and this feature would not make that any better or worse.

This also bothers me because it detaches the Python version from the version of Python running build, relying on a search procedure (from virtualenv, I assume). When building a binary wheel, which Python can be important; "any" copy of Python x.y might not do (esp on macOS).

If you bother to read the documentation I've linked you'll see you have very strict control what python you're building your wheel (from specifying just major python version all the way down to passing the explicit path),

[henryiii]

Often you build wheels to target your current OS and that's it

Yes, and this exactly is what you get today with --wheel. But you are trying to add a way to target a bunch of Python versions easily, which makes it look like it builds general wheels and not "current environment" wheels.

If you bother to read the documentation I've linked

I did, though I didn't see the explicit path part.

What about my suggestion that we not integrate the loop, but have an explicit --virtualenv=<spec> flag?

Again, I'm not totally against this (I'm a fan of a similar, but non-looped version of this in pipx), but I have concerns I'm voicing.

[gaborbernat]

What about my suggestion that we not integrate the loop, but have an explicit --virtualenv=<spec> flag?

I'm happy with that part.

Actually, I realized one can already do this today via environment variables, but that doesn't translate that nicely to usage within tox:

env VIRTUALENV_PYTHON=py38 python -m build .
env VIRTUALENV_PYTHON=py37 python -m build .

Yes, and this exactly is what you get today with --wheel. But you are trying to add a way to target a bunch of Python versions easily, which makes it look like it builds general wheels and not "current environment" wheels.

Ignore cibuildwheel, which is unusable in some environments that can't use those CI environments it supports. The first step to generate auditwheels is to build a wheel, and I think build should be used for that.

[FFY00]

Sorry for the short reply, it is late and I am on my phone. I am remodeling at home, so my setup will be dismantled until Wednesday, I should still be able to jump on my laptop if needed.

I do not agree with this interface, I think #104 is the correct way forward. We can make it so that the interpreter argument can receive multiple values. The only practical difference from what you propose is that the wheels and sdist will have to be built in different commands, which I think it's fine.

python -m build --sdist
python -m build --wheel --python 3.6,3.7,3.8,3.8

[gaborbernat]

I see no actual proposal in #104, can you clarify what you meant?

[henryiii]

Ignore cibuildwheel, which is unusable in some environments that can't use those CI environments it supports. The first step to generate auditwheels is to build a wheel, and I think build should be used for that

This was not related to cibuildwheel. My point is making a list of Python versions looks like you are building for general Python versions, which you are not with build; build just builds a Python wheel with an existing environment, which I think is much more clear when you list it explicitly and run build once per wheel. Also, which environment does the sdist build in? It shouldn't matter (though I can customize my sdist command to store information in the SDist about what Python it came from ;) ), but this logically either has to do something with the SDist; if you don't make it a loop, then it's explicit and up to the programmer.

I really like the environment variable method, actually. It seems odd to add a command line argument that only works when virtualenv is installed; I'm guessing most of the time it is not installed, since it is not needed on Python 3. You can control other aspects of the build via PIP_ environment variables, too, so it's consistent with what we currently have, say for PIP_ONLY_BINARY. An environment variable also works in the current version of build. :)

doesn't translate that nicely to usage within tox

Isn't this tox's fault then, not build? ;)

I do not agree with this interface, I think #104 is the correct way forward.

I'm also not sure what's different in that vs. the proposal here? I think @gaborbernat's suggestion of using the virtualenv python specification strings is necessary; you have to be able to select 32 bit vs. 64 bit Python, PyPy, sometimes an exact path, etc. A simple number is not enough for all cases (though it reduces to exactly the same number in the simplest case, so I don't think this is actually any different that the proposal above). (This is also part of why I don't like looping - you are not building a "3.8" wheel, you are building a wheel with a specific Python, which is 32 or 64 bit, which has some minimum version of macOS support, etc. One at a time specifications make this clearer, IMO.)

If it's the name, --virtualenv vs. --python, that would be open for discussion; --virtualenv helps highlight it's only available if you have the virtualenv extra installed, and that the specification comes from virtualenv. --python clearly highlights what you are doing, selecting the base Python interpreter. However, I currently like the environment variable; it works today, it's clearly virtualenv-only, and it's a standard name from virtualenv itself, just like the pip settings.

[henryiii]

Keep in mind, manylinux will not support this, as it does not have virtualenv installed, and it does not touch the "base" python, so you need to be using one of the specific Python interpreters anyway. And this will not work correctly for redistributable wheels on macOS unless someone manually installs all the official CPython downloads. So it's usefulness is mostly limited to specific environments and maybe Windows. And if someone is targeting a specific environment, they will very likely be in said environment, not building for lots of Pythons at once.

It's really easy to loop on the command line, or manually list N lines for N interpreters (N is pretty small).

python -m build --sdist .
env VIRTUALENV_PYTHON=py3.8-32 python -m build --wheel .
env VIRTUALENV_PYTHON=py3.8-64 python -m build --wheel .
env VIRTUALENV_PYTHON=py3.7-32 python -m build --wheel .
env VIRTUALENV_PYTHON=py3.7-64 python -m build --wheel .
env VIRTUALENV_PYTHON=pypy3.7 python -m build --wheel .

[henryiii]

Ignore cibuildwheel, which is unusable in some environments that can't use those CI environments it supports

Final quick comment, not related to the discussion here but addressing the comment above since I'm involved in cibuildwheel and can't let it slide by: cibuildwheel does not "support specific CI environments", it has a list of "tested" ci environments. You can (and it is) run on custom company machines, etc. You can even run it locally, though if you target macOS or Windows, you have to be okay with it installing official, global copies of Python on your system, making it better for "ci-like" deployments. It's perfectly okay to target linux on any system due to the use of docker, though. :) You should think of cibuildwheel more like a special tox or nox that handles Python-for-wheel installing and is customized for building. It is not a collection of random CI scripts that just works on CI. It's a regular Python package.

It was just an example above, though. The same holds true for multibuild, the azure template based solutions, or any other system. Build should be seen as a piece of a pipeline (building wheels), and not try to take on replacing these tools, because there are a lot of components to getting it right, building for multiple versions of Python in a loop is just one of them. To users, often this is seen as the most important one, but there's really a lot more to it.

[gaborbernat]

Isn't this tox's fault then, not build? ;)

It's highly unusual for tools to provide env-var only interface.

cibuildwheel does not "support specific CI environments", it has a list of "tested" ci environments. You can (and it is) run on custom company machines, etc

As said above cibuildwheel is an orthogonal project to build --wheel command (they have different use cases and scenarios when you want to use them), so I consider the entire thread on this off-topic. Let's not dwell any further on it. As said there's plenty of use cases when you absolutely don't want auditwheel/dellocate/etc.

[layday]

This seems like a minor gain in circumstances where "cross-compilation" is feasible. For manylinux, it offers no actual benefit - the build dependencies are installed under each Python version available.

We cannot reuse --wheel for this because it's a flag.