Allow external package repositories to be configured as non-legacy?

[chriskuehl]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

My understanding is that currently Poetry supports two different types of package repositories:

• Legacy repositories using the traditional PEP503 repository layout, which involves parsing package metadata from HTML files like /simple/mypackage.
• "PyPi repositories" which use the pypi.org JSON api for fetching metadata.

The benefit of the JSON API is that it can include the requires_dist key to indicate the requirements of a package. This improves performance during version resolution because Poetry does not need to download each package from PyPI in order to check its dependencies.

From reading the code, it doesn't appear possible to tell Poetry to use the JSON API for private registries; instead, it always uses the legacy registry:

poetry/src/poetry/factory.py

Lines 123 to 128 in 15f6ec1

	@classmethod
	def configure_sources(
	cls, poetry: Poetry, sources: list[dict[str, str]], config: Config, io: IO
	) -> None:
	for source in sources:
	repository = cls.create_legacy_repository(source, config)

If Poetry supported an option to configure an internal registry as pypi.org-compatible, we could speed up certain version resolutions significantly. To give an example, on one internal project my company is testing out, this saves ~2.5 minutes and ~10 GB of downloaded files when resolving a particularly tricky dependency situation.

Would Poetry be willing to entertain this change? We would be happy to submit pull requests to implement this change if so.

Simple proof-of-concept patch

For reference, here is the patch I'm using to test this change currently:

diff --git a/src/poetry/factory.py b/src/poetry/factory.py
index 2a0b0b69..27895442 100644
--- a/src/poetry/factory.py
+++ b/src/poetry/factory.py
@@ -125,7 +125,12 @@ class Factory(BaseFactory):
         cls, poetry: Poetry, sources: list[dict[str, str]], config: Config, io: IO
     ) -> None:
         for source in sources:
-            repository = cls.create_legacy_repository(source, config)
+            if not bool(source.get("use_json_api", False)):
+                repository = cls.create_legacy_repository(source, config)
+            else:
+                from poetry.repositories.pypi_repository import PyPiRepository
+                repository = PyPiRepository(source["url"])
+
             is_default = bool(source.get("default", False))
             is_secondary = bool(source.get("secondary", False))
             if io.is_debug():
diff --git a/src/poetry/json/schemas/poetry-schema.json b/src/poetry/json/schemas/poetry-schema.json
index fe916f1e..3d6281e0 100644
--- a/src/poetry/json/schemas/poetry-schema.json
+++ b/src/poetry/json/schemas/poetry-schema.json
@@ -527,6 +527,10 @@
           "type": "boolean",
           "description": "Make this repository the default (disable PyPI)"
         },
+        "use_json_api": {
+          "type": "boolean",
+          "description": "Use the PyPI JSON API for fetching package metadata (faster if supported by your registry)."
+        },
         "secondary": {
           "type": "boolean",
           "description": "Declare this repository as secondary, i.e. it will only be looked up last for packages."

The actual solution would probably need to be a little bit more involved in order to support authentication (like is supported for legacy registries currently).

[chriskuehl]

Seems like https://peps.python.org/pep-0658/ is probably a better way to achieve this.

I see that the Poetry docs say that it is supported already but I think it may not have been fully merged? #5509 is still open and while I see that the poetry-core change is merged, it seems like Poetry wasn't updated to take advantage of the tag yet? (Haven't tested since I don't have a registry handy with PEP658 support, but can't find any relevant code that would handle it.)

[Secrus]

@chriskuehl I took over #5509 and am working on it. It has proven to be a bit more complex and time-consuming than I expected. It's coming, but most likely not in the next release.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.