Performance Tracking: `generate-lockfiles`

[sureshjoshi]

There is a common mention/complaint that calling pants generate-lockfiles :: is slow. Creating this ticket to capture associated discussions, tickets, and even better, sample repos.

• pants generate-lockfiles to use uv for dependency resolution #20679
• New Python Backend wishlist #20897 (comment)
• New Python Backend wishlist #20897 (comment)
• New Python Backend wishlist #20897 (comment)

From my brief experiments, using pip 24.1 improves cold lockfile generation by about 50% on a small set of requirements, but does not affect incremental checking.

Some ideas that have been floated around:

• Use pex's incremental lockfile updates (available, but not wired into Pants)
• Use uv to some degree
• Use fast-deps if we're not currently doing so (though, whether that will continue to be a standalone option: perform 1-3 HTTP requests for each wheel using fast-deps pypa/pip#12208)

[jsirois]

Use pex's incremental lockfile updates (available, but not wired into Pants)

This is widely misunderstood. Pex does incremental updates of lock files, but it does this using a full Pip resolve which is exactly as fast as a normal lock create. This bug tracks doing an incremental resolve: pex-tool/pex#2044 These are two orthogonal things.

Use fast-deps if we're not currently doing so

See here, but I tried this and found its still slow: pex-tool/pex#2375 (comment)

[sureshjoshi]

This, bug tracks doing an incremental resolve: pex-tool/pex#2044 These are two orthogonal things.

I was just writing a comment on that ticket asking whether what we have today vs what the ticket refers to were comparable. So "incremental update" vs "incremental resolve" would be a better distinction on the pex side.

See here, but I tried this and found its still slow: pex-tool/pex#2375 (comment)

Ah, that sucks, but yeah - that was just an example I grabbed from one of the many slack/discussions/issues about this.

[cburroughs]

Tracker of several pending pip performance improvements: pypa/pip#12184

[cburroughs]

Sorry, the correct tracker is now: pypa/pip#12921

[cburroughs]

Crosslinking some other pip items of possible relevance:

• Prefer upper bounds when resolving/backtracking pypa/pip#13017
• pip prefers old sdists that "obviously" can't work over recent wheels pypa/pip#13037

[notatallshaw]

It would be very helpful if you can report resolutions that are slow or problamatic on either https://github.com/notatallshaw/Pip-Resolution-Scenarios-and-Benchmarks or https://github.com/pypa/pip.

Probably you would be best switching to uv (which I do also report lots of resolution issues to), but I do aim to keep improving pip's resolver.

[jsirois]

Pex now supports creating PEXes from a --venv-repository: https://github.com/pex-tool/pex/releases/tag/v2.59.0. If you really want fast locks, you should use uv and just build any needed PEXes from venvs you get uv to build via uv venv & uv sync, etc.