Skip to content

Update pip compatibility guide to note transitive URL dependency support #8081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
charliermarsh merged 1 commit into from
Oct 10, 2024
Merged

Update pip compatibility guide to note transitive URL dependency support #8081

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions docs/pip/compatibility.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,22 +174,25 @@ uv pip install wheel && uv pip install --no-build-isolation biopython==1.77
For a list of packages that are known to fail under PEP 517 build isolation, see
[#2252](https://github.com/astral-sh/uv/issues/2252).

## Transitive direct URL dependencies for constraints and overrides
## Transitive URL dependencies

While uv does support URL dependencies (e.g., `black @ https://...`), it does not support
_transitive_ (i.e., "nested") direct URL dependencies for constraints and overrides.
While uv includes first-class support for URL dependencies (e.g., `ruff @ https://...`), it differs
from pip in its handling of _transitive_ URL dependencies in two ways.

Specifically, if a constraint or override is defined using a direct URL dependency, and the
constrained package has a direct URL dependency of its own, uv _may_ reject that transitive direct
URL dependency during resolution.
First, uv makes the assumption that non-URL dependencies do not introduce URL dependencies into the
resolution. In other words, it assumes that dependencies fetched from a registry do not themselves
depend on URLs. If a non-URL dependency _does_ introduce a URL dependency, uv will reject the URL
dependency during resolution. (Note that PyPI does not allow published packages to depend on URL
dependencies; other registries may be more permissive.)

uv also makes the assumption that non-URL dependencies won't introduce URL dependencies (i.e., that
dependencies fetched from a registry will not themselves have direct URL dependencies). If a non-URL
dependency _does_ introduce a URL dependency, uv will reject the URL dependency during resolution.
Second, if a constraint (`--constraint`) or override (`--override`) is defined using a direct URL
dependency, and the constrained package has a direct URL dependency of its own, uv _may_ reject that
transitive direct URL dependency during resolution, if the URL isn't referenced elsewhere in the set
of input requirements.

If uv rejects a transitive URL dependency in either case, the best course of action is to provide
the URL dependency as a direct dependency in the `requirements.in` file, rather than as a
constraint, override, or transitive dependency.
If uv rejects a transitive URL dependency, the best course of action is to provide the URL
dependency as a direct dependency in the relevant `pyproject.toml` or `requirement.in` file, as the
above constraints do not apply to direct dependencies.
Copy link
Copy Markdown

@slochower slochower Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When reading the original text and this version, it sounds (to me) like you are suggesting a workaround of adding the URL dependency to your pyproject.toml or requirements.in (in the case that you depend on registry package A which depends on package B with a URL dependency). Rather, I think the advice is to add the URL dependency directly to package A in that scenario. Would phrasing like this be more clear?

If uv rejects a transitive URL dependency in a package from a registry, the best course of action is to provide the URL dependency as a direct dependency in the pyproject.toml or requirement.in file of the registry package, as the above constraints do not apply to direct dependencies.

Maybe it is a little clunky, but I definitely spent a few hours thinking I could just add the transitive URL dependency as a direct dependency in my package and that would satisfy uv.

Copy link
Copy Markdown
Member Author

@charliermarsh charliermarsh Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you depend on a package A from the registry, which lists package B as a direct URL dependency (e.g., B @ https://...), adding package B to your pyproject.toml will work. That's the recommended solution. Something like:

[project]
dependencies = ["A", "B @ https://..."]

Copy link
Copy Markdown

@slochower slochower Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry -- I'm talking about one more layer of redirection. Say I'm writing a script with inline metadata. The script depends on registry package A. Registry package A depends on registry package B. In package B's pyproject.toml is a URL dependency. Adding that URL dependency directly to the inline metadata of a script will not satisfy uv run. I'm not sure there is a solution. Is there?

Copy link
Copy Markdown
Member Author

@charliermarsh charliermarsh Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would actually still expect that to work. Are you running into a failure there? The way to think about it is: it's actually fine if registry packages depend on URL dependencies, but uv has to know about all possible URL dependencies upfront.

Copy link
Copy Markdown

@slochower slochower Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I run without adding the URL dependency, I get:

error: Package `openeye-toolkits-python3-osx-universal` attempted to resolve via URL: https://pypi.anaconda.org/openeye/simple/openeye-toolkits-python3-osx-universal/2023.1.1/OpenEye-toolkits-python3-osx-universal-2023.1.1.tar.gz. URL dependencies must be expressed as direct requirements or constraints....

If I add openeye-toolkits-python3-osx-universal @ https://... to my script metadata, I then see:

error: No `project` table found in: `/Volumes/xxx/pyproject.toml

If I use uv run --no-project I still see the same thing, confusingly.

❯ uv run tmp.py --no-project

error: No `project` table found in: `/Volumes/xxx/pyproject.toml`
(.venv)

The script metadata looks like this:

# /// script
# requires-python = ">=3.10"
# dependencies = [
#   "package-hosted-on-internal-registry==3.3.0",
#   "openeye-toolkits-python3-osx-universal @ https://pypi.anaconda.org/openeye/simple/openeye-toolkits-python3-osx-universal/2023.1.1/OpenEye-toolkits-python3-osx-universal-2023.1.1.tar.gz"
# ]
# ///

Copy link
Copy Markdown
Member Author

@charliermarsh charliermarsh Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to put the no-project flag before the filename, not after. Otherwise, it’s treated as an argument to the Python call.

Copy link
Copy Markdown

@slochower slochower Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Great, thanks.

This comment of yours really helped:

The way to think about it is: it's actually fine if registry packages depend on URL dependencies, but uv has to know about all possible URL dependencies upfront.

And maybe some form could be put into the documentation.


## Virtual environments by default

Expand Down