Initial changes exploring python3.14 support

[ablakey]

Problem 1

The first problem I ran into when trying to run rospy with Python 3.14 is that it wouldn't build. I chased down errors and found that it was trying to utilize a different version of python to do the building than the version we wanted to actually run with. I think this just generally creates a lot of potential for compatibility issues.

In fact, my version of Ubuntu 22.04 doesn't even have python available. I dunno if that was a bug or is part of Python2 being discontinued, but it caused the changed code, when reached, to error. When I manually set it to python3.10 I got new kinds of errors related to pip and all that. I think it generally is a bad idea to use Python 3.10 to construct a virtualenv and gather all the dependencies for a Python3.14 project.

Instead of calling python by string name, we use the target version of Python, which is passed in from CMakeLists.txt via argparse. To do this in a number of functions, we move python to the constructor args and make it an instance member.

Problem 2

Once I started using this further, I found that there is a further problem with setuptools missing. This came when trying to depend on locus_py. I found that setuptools is no-longer pre-installed in venvs (Changed in version 3.12: setuptools is no longer a core venv dependency.)

I hacked away at venv.py to install setuptools as part of preinstall. Also went to a newer version of pip and pip-tools. Pretty sure this breaks Python2 support.

Problem 3

When depending on tycho_py I ran into install failure issues with numpy. Meson complains: ../meson.build:41:12: ERROR: Python dependency not found

I tried updating numpy in tycho_py requirements.txt to the latest version: 2.3.4 and get a version resolution error:

ERROR: Ignored the following versions that require a different python version: 2.3.0 Requires-Python >=3.11; 2.3.1 Requires-Python >=3.11; 2.3.2 Requires-Python >=3.11; 2.3.3 Requires-Python >=3.11; 2.3.4 Requires-Python >=3.11
ERROR: Could not find a version that satisfies the requirement numpy==2.3.4 (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5, 1.20.0, 1.20.1, 1.20.2, 1.20.3, 1.21.0, 1.21.1, 1.21.2, 1.21.3, 1.21.4, 1.21.5, 1.21.6, 1.22.0, 1.22.1, 1.22.2, 1.22.3, 1.22.4, 1.23.0, 1.23.1, 1.23.2, 1.23.3, 1.23.4, 1.23.5, 1.24.0, 1.24.1, 1.24.2, 1.24.3, 1.24.4, 1.25.0, 1.25.1, 1.25.2, 1.26.0, 1.26.1, 1.26.2, 1.26.3, 1.26.4, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.2.5, 2.2.6)
ERROR: No matching distribution found for numpy==2.3.4

It looks like the issue here is due to the design of catkin_virtualenv , which uses requirements.txt found in every dependency that also uses catkin_virtualenv. Consider if a maintainer has pinned requirements.in -> requirements.txt because when that package's scripts are run, it wants to use python3.10, so numpy was pinned to 2.2.6. Great! But if another package wants to use that package's python modules with python3.14, trying to use numpy=2.2.6 breaks because that's not valid.

This is why lockfiles (eg. npm's package-lock.json) are generated for the target package, containing the entire tree of dependencies, and do this from the non-locked package.json. If a dependency maintainer has opinions on dependency versions, they define that in package.json with, ideally, the widest ranges possible. I think this should be in requirements.in.

I'm wondering ifcatkin_virtualenv should:

• ignore requirements.txt in favour of requirements.in for all dependencies
• generate requirements.txt to include the entire tree of dependencies
• use requirements.txt if it exists for all dependencies, not even looking at dependency requirements files.
• detect when requirements.txt no-longer satifies the requirements.in tree

I'm also wondering how much wheel we're re-inventing when it comes to resolving dependency trees and lockfiles. As long as we're carefully respecting both the semantics of requirements.in and requirements.txt as well as the core intent, this still makes sense?

Drafted Changes

Python Interpreter Use

All scripts now use the PYTHON_INTERPRETER if defined. Otherwise python3. Some scripts don't actually care about Python version, but having to reason about when Virtualenv does and does not care is not a great design.

setuptools

Changed some of the pip preinstall dependency versions and added setuptools to be installed.

Handling the lockfile issues

I'm trying to add an EXPERIMENTAL_GENERATE_FULL_LOCKFILE flag to play with the idea of fundamentally changing how catkin_virtualenv generates and uses lockfiles. The idea is that the package being built is responsible for defining the whole lockfile for the entire dependency tree used for populating the venv. It stops reading requirements.txt and starts reading requirements.in.

This will require making requirements.in available in dependency projects. I'm not sure how we'd want to approach that. For now I'm hacking at this to prove the concept.

However, it may also be as simple as changing the semantics of pip_requirements, expecting it to point to requirements.in and not the lockfile requirements.txt.

Testing

You can test this with this repo: https://github.com/ablakey/rospypypi/tree/main Though you may have to add additionally complex dependencies to figure out what else is breaking where.

Other Thoughts

How was catkin_virtualenv supposed to work if it required every dependency to require the exact same version of a dependency from the lockfile, rather than finding a version that satisfies all of the requirements.in files?

[paulbovbel]

I'm trying to add an EXPERIMENTAL_GENERATE_FULL_LOCKFILE flag to play with the idea of fundamentally changing how catkin_virtualenv generates and uses lockfiles. The idea is that the package being built is responsible for defining the whole lockfile for the entire dependency tree used for populating the venv. It stops reading requirements.txt and starts reading requirements.in.

makes sense

[paulbovbel]

this is weird, because AFAIK, whatever python interpreter created the python, will be symlinked into venv/bin/python, and so used by self._venv_bin("python")

# python3.14 -m venv venv
# ll venv/bin/python
lrwxrwxrwx 1 pbovbel pbovbel 10 Nov 12 11:57 venv/bin/python -> python3.14*

[ablakey]

There's a number of cases that might have different system expressions, including the fresh build, re-build, builds fail, or calling install, lock, etc. independently.

I might have been doing something wrong, but I was running into cases that had pythonreferring to 3.10 and not the version we're trying to target with PYTHON_INTERPRETER.

Being explicit is probably good here anyways, given just how complex this all gets and how brittle Python versioning management feels. One less thing to reason about when stuff breaks.