Global configuration of custom sources

[andreas-vester]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

I am using private repositories to download dependencies as I am working behind a corporate firewall. According to the docs (https://python-poetry.org/docs/repositories/#install-dependencies-from-a-private-repository), I am supposed to put a private repo into pyproject.toml.

# pyproject.toml
[[tool.poetry.source]]
name = "foo"
url = "https://foo.bar/simple/"
default = true

If I do that, I am basically not able to commit pyproject.toml to GitHub as I don't want to reveal internal server names. Committing these information would be useless for others anyway, as they are not able to access our internal repos.

I am wondering if it would make sense to define private repos in poetry's own configuration on my local machine? Or maybe I can set some environment variables?

[andreas-vester]

I am also wondering why we need to put the private repo into pyproject.toml at all?

We have to configure it in the actual poetry configuration, too (https://python-poetry.org/docs/repositories/#adding-a-repository)? Why can't we use this information?

Can't we just have a configuration for a repo to publish to and for repo(s) to download packages from without exposing any information in pyproject.toml?

[andreas-vester]

Any kind of response would be appreciated.

[andreas-vester]

If I can't get rid of the [[tool.poetry.source]] section in pyproject.toml, I am not able to commit my code to GitHub at all for the simple reason that corporate rules rightly forbid to publish internal server names on GitHub.

As a result, I can't use poetry.

I think this is an important issue/feature request as other developers working in corporations will probably face comparable restrictions.

[finswimmer]

Hey,

to be honest I haven't used privat repos until now, so please forgive me if my question is might be naiv 😃

If the source of the package is not defined in the pyproject.toml, how do you avoid a supply chain attack? I mean, you have to be sure, that the source is configured everywhere, where one tries to install the package, otherwise the deps are received from PyPi and there can be a package with the same name, not under you control. Sounds like a good way to shoot yourself into the foot 🤔

fin swimmer

[PidgeyBE]

I'm facing the same issues...
Would be great to have something similar as pip's --index-url or --extra-index-url ...

In this way @andreas-vester could install locally via poetry install --index-url https://company-pypi-mirror.

[PidgeyBE]

@andreas-vester
I see poetry==1.2 passes env vars to pip: https://github.com/python-poetry/poetry/blob/1.2/src/poetry/utils/env.py#L1443
So probably you can set the env var PIP_INDEX_URL to guide pip/poetry to the right index?

[neersighted]

It is possible we could allow configuring the implicit PyPI URL, but authentication would get messy. I'm still not 100% sure on if this is a useful/desirable feature as it decreases the repeatability of Poetry installations and may have security/correctness implications.

As far as configuring primary (--extra-index-url) sources through the environment, that seems a bit more fraught from a design standpoint, but also would require much less refactoring and is easier to expand existing test coverage to cover.

In short, what is desired here is something we could work towards, but I think we would want to think long and hard about Poetry fetching different code from different sources due to environmental factors (up to this point, everything Poetry fetches and installs is strictly defined in the lock file and pyproject.toml).

[andreas-vester]

@finswimmer To be honest, I haven't thought about a supply chain attack because the packages on our corporate private repo are exactly the same as on pypi. In fact, we are maintaining a pypi mirror for the sole reason that it is forbidden to download software from the internet.

In this case, if I install, let's say, the pandas library from our private repo, it's the same package that you would install from pypi.

You're right in demanding that the dependencies must be reachable for everybody. That's probably the task of the package maintainer.

In my particular case, all the packages are freely available from pypi for those outside our corporate network. If this wouldn't be the case, it probably wouldn't make sense to take the project public in the first place. For example, you wouldn't publish a project that has dependencies from, let's say, a private GitHub repo, wouldn't you?

[andreas-vester]

@neersighted I am not saying that there is an easy solution to this problem. All I can say is that putting a private repo into pyproject.toml doesn't make any sense as soon as you want to publish your project.

Let's assume I were allowed to place our private repo/server names in pyproject.toml and set it as the default source. As soon as you fork the repo from GitHub, you get a configuration that is not working on your end because you are technically not able to reach our internal server. You need pypi.

At this point in this discussion it is firstly important to me that we are on the same page with respect to acknowledging the fact that some corporate circumstances are restricting the use of poetry when it comes to public open source projects.

The next step would then be to find a suitable solution. At that point I can only make some suggestions and the poetry maintainers have to make a decision.

Some suggestions that come to my mind could involve the following:

• Manage the sources for up- and downloading dependencies equally using poetry config.
• Using env vars such as pip does (extra-index etc.)
• Come up with a workaround, something like a Git hook, that removes the private source from the pyproject.toml before every commit and re-inserts it afterwards.
  -...

[andreas-vester]

@PidgeyBE So probably you can set the env var PIP_INDEX_URL to guide pip/poetry to the right index?

I did try the following without success:

set PIP_INDEX_URL=https://<my_proxy_username>:<my_proxy_password>@foo.bar/simple/

When trying to add a package, poetry tries to reach pypi.

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

We have to configure it in the actual poetry configuration, too

This isn't technically accurate. poetry source add ... just adds it to pyproject.toml, I believe. It only needs to be added to global configuration when publishing: https://python-poetry.org/docs/repositories/#publishable-repositories

If the source of the package is not defined in the pyproject.toml, how do you avoid a supply chain attack?

I'm not sure this needs to be handled by Poetry, though. All Poetry should care about is "Here are dependency requirements, here are the locked versions of all transitive dependencies, install those". If the user cares about supply chain attack, they should set the index url to a trusted url, but the point is that that trusted url should be swappable. Does it matter if user A's trusted url is different from user B's trusted url?

To me, it seems like the issue with not hardcoding a source comes down to two scenarios:

1. If user A's index URL has a patched version of package A v1.0 and user B's index URL has the unmodified version of package A v1.0.
   • IMO, users should not do this, they should instead serve the patched v1.0 version as a separate v1.0-patch version, if their package does, indeed require the patch
2. If package A is an internal package only served in user A's index URL, and not user B's index URL.
   • IMO, it would be a perfectly reasonable user experience for Poetry to say "cannot find package at " and for the user to realize "oh the package is internal and i need to use another index url". User A in this case should put in the README "this package depends on an internal package, so use ".

While "repeatability of Poetry installations" is definitely a good thing, IMO that should only be vis a vis locking transitive deps. It shouldn't matter if package A v1.1 came from PyPI or a company mirror; if package A v1.1 is the same in both PyPI or the mirror, it's just as repeatable if you change which source you install from.

---

My company's usecase is we have an Artifactory mirror that's externally facing, and an Artifactory mirror that is only exposed via our company VPN. The VPN mirror is useful for running installs within the AWS VPC (so requests can stay within the AWS region), but the external mirror is useful for devs, so we don't force devs to be on the VPN to install things.

We also have a lot of Python projects in our monorepo, and it's a bit annoying to have to copy-paste the same tool.poetry.source block in each of them. Ideally, we'd just set an env var or something that our devs can just set once and have it be used in any of the projects in the monorepo.

IMO the ideal workflow here would be:

1. No tool.poetry.source in pyproject.toml or poetry.lock
2. Look for sources in the following locations, in order:
   • --index-url flag, that can be given to any poetry command (repeatable)
   • POETRY_INDEX_URL env var (semicolon separated?)
   • $project/poetry.toml (i.e. poetry config --local): TOML list
   • $config_home/config.toml (i.e. poetry config): TOML list
3. If no sources specified, use pypi.org

Authentication can be put directly into the url (e.g. https://myuser:password@pypi.mycompany.com) or in $config_home/config.toml mapping domain to credentials

It would be great if we could also set up this private repository for all users, rather than having to do it per-user. pip supports this with the index-url and cert fields in %PROGRAMDATA%\pip\pip.ini (on Windows).

[neersighted]

I think there is certainly room for flexibility here, but any design is going to have to come after we first make the existing functionality more consistent/robust (e.g. as described in #5984 (comment), or an alternative).

To add new sources of config here or new behaviors before first making the existing functionality better would be an own goal. It will be much easier to reason about a safe/consistent design after refactoring of what we already have.