Production dependencies not getting updated by Dependabot

[StevenMaude]

Relates to the past issue #579.

We currently have quite outdated dependencies.

Dependabot is currently failing on pyproject.toml:

updater | ERROR <job_585486053> Error processing pydocstyle[toml] (Dependabot::SharedHelpers::HelperSubprocessFailed)
updater | ERROR <job_585486053> Using indexes:
updater | <job_585486053>   https://pypi.org/simple
updater | <job_585486053> 
updater | <job_585486053>                           ROUND 1                           
updater | <job_585486053> /usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/req/req_install.py:866: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
updater | <job_585486053>   deprecated(
updater | <job_585486053> Traceback (most recent call last):
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/bin/pip-compile", line 8, in <module>
updater | <job_585486053>     sys.exit(cli())
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
updater | <job_585486053>     return self.main(*args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1055, in main
updater | <job_585486053>     rv = self.invoke(ctx)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
updater | <job_585486053>     return ctx.invoke(self.callback, **ctx.params)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/core.py", line 760, in invoke
updater | <job_585486053>     return __callback(*args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
updater | <job_585486053>     return f(get_current_context(), *args, **kwargs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/scripts/compile.py", line 580, in cli
updater | <job_585486053>     results = resolver.resolve(max_rounds=max_rounds)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/resolver.py", line 590, in resolve
updater | <job_585486053>     is_resolved = self._do_resolve(
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/piptools/resolver.py", line 622, in _do_resolve
updater | <job_585486053>     resolver.resolve(
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
updater | <job_585486053>     collected = self.factory.collect_root_requirements(root_reqs)
updater | <job_585486053>   File "/usr/local/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 481, in collect_root_requirements
updater | <job_585486053>     raise InstallationError(problem)
updater | <job_585486053> pip._internal.exceptions.InstallationError: Constraints cannot have extras
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:224:in `run_command'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:282:in `run_pip_compile_command'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:82:in `block (3 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `each'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:76:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:168:in `with_git_configured'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:72:in `block in fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:71:in `fetch_latest_resolvable_version_string'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb:48:in `latest_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker.rb:47:in `latest_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_585486053> /home/dependabot/python/lib/dependabot/python/update_checker.rb:126:in `preferred_version_resolvable_with_unlock?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | ERROR <job_585486053> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:486:in `requirements_to_unlock'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:259:in `check_and_create_pull_request'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:109:in `check_and_create_pr_with_error_handling'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `block in run'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `each'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:82:in `run'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/update_files_job.rb:17:in `perform_job'
updater | ERROR <job_585486053> /home/dependabot/dependabot-updater/lib/dependabot/base_job.rb:50:in `run'
updater | ERROR <job_585486053> bin/update_files.rb:23:in `<main>'

We have this dependency because we have a pydocstyle configuration in pyproject.toml.

Running pip-compile locally does work though, but maybe there's some difference in my local setup compared with Dependabot 🤷‍♂️

[StevenMaude]

Simple workarounds:

• Just use a .pydocstyle file instead, live with the additional configuration file and remove the extra specified dependency as it's no longer needed.
• Add tomli as a separate optional dependency and remove the extra specified dependency — there might be issues if the required version by pydocstyle diverges in future from that which we pin.
• Just remove toml as an optional dependency because tomli is added by other dependencies anyway, and the version in pydocstyle is currently unpinned.

Also worth noting that pydocstyle will use the standard library tomllib in Python 3.11.

More complicated workaround:

• Try and workaround the pip issues: --require-hashes does not correctly handle pinned package with extras pypa/pip#9644

[StevenMaude]

Other weird features of this issue:

• It seems to prevent the requirements.prod.txt from being automatically updated. But the actual dependency causing an error is a dev requirement.
• coverage[toml] doesn't seem to cause an error, but that's maybe because it's a transitive dependency.

[StevenMaude]

I think this is a relevant issue: dependabot/dependabot-core#6550

And I think it also affects a few other of our repositories.

[StevenMaude]

The linked Dependabot issue is fixed, so we don't get the:

pip._internal.exceptions.InstallationError: Constraints cannot have extras

any longer. Nor do we don't get associated errors in the panel that shows "last checked" for pyproject.toml.

But I'm still not convinced think it is entirely working, because we still don't get updates for the out-of-date dependencies.

[StevenMaude]

In fact, if you look at Dependabot's version update log, it doesn't even check the production requirements at all. There's no "No update needed" for any of the production requirements.

I think this Dependabot issue might be related, or it's something similar (Dependabot just picks one update ecosystem and use that; so it updates the requirements based on the requirements.dev.in file, and not pyproject.toml.)

It might also be worth trying to ditch the .in file and put the dev requirements into pyproject.toml. That's also more consistent. Here's a blog post with an example.

[evansd]

It might also be worth trying to ditch the .in file and put the dev requirements into pyproject.toml. That's also more consistent. Here's a blog post with an example.

Ooh, this looks nice.

[StevenMaude]

Moving everything into pyproject.toml works.

[StevenMaude]

Moving everything into pyproject.toml works.

That is, it works to get Dependabot back up and checking dependencies.

There's then an issue with transitive dependencies seemingly not being updated. So, the update of the virtualenv package fails because distlib (introduced via virtualenv) doesn't get bumped.

[StevenMaude]

Other related Dependabot issues on not updating Python dependencies from different sources:

• requirements.txt is ignored when Pipfile exists dependabot/dependabot-core#1946
• Dependabot fails to update same dependency in same pull request dependabot/dependabot-core#6200