GUIDE: How to install PyTorch via Pipenv (and how to add other 3rd Party Repositories).

[Arcitec]

Issue description

Try this Pipfile on 64-bit Linux. It fails to find the specified version:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/torch_stable.html"
verify_ssl = true
name = "pytorch-cu113"

[packages]
torch = {version = "==1.10.2+cu113", index = "pytorch-cu113"}

[dev-packages]

[requires]
python_version = "3.9"

Edit: The problem has been found. Torch's repo (the URL I used) is a non-standardized repo that only pip understands. Pipenv only reads the PEP 503-standardized repos. See solution in my followup comments below.

[Arcitec]

edit: Solved, see below. It's Torch's fault, not Pipenv's.

[Arcitec]

Alright I've figured it out. Pipenv only supports standardized PEP 503 repositories. Torch, by default, gives us the URL for a non-standard "pip" bastard repo. I will post a followup comment with the full answer/solution for people.

[Arcitec]

Solved: This is how to install Torch CORRECTLY in Pipenv. Most people do this incorrectly even in various "solutions" I've seen from other people... I've researched to the bottom of this problem and found out the real, intended solution that everyone should use.

This technique is the correct one and it's documented here:

https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

Basically, you add a new "source repository" to your Pipfile. And you should always use verify_ssl = true for extra security when possible.

Anyway, there's a shortcut to adding a custom repository without editing the pipfile. The --extra-index-url shortcut says "Add this PEP 503-standard repo URL":

• Version that uses CUDA Toolkit 11.3, with GPU acceleration (this toolkit is required by NVIDIA Ampere (RTX 30x0) GPUs and newer):

pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"

• Version that uses CUDA Toolkit 10.2, with GPU acceleration:

pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cu102"

• Version that uses CPU instead:

pipenv install --extra-index-url https://download.pytorch.org/whl/ "torch==1.10.1+cpu"

Now you may notice something: There's no ".html" in my repo filenames.

That's because the repos with a HTML filename (such as https://download.pytorch.org/whl/torch_stable.html) are NON-STANDARD repos that ONLY pip understands.

You CANNOT use those URLs in Pipenv.

Pipenv uses PEP 503 standardized repos. Torch publishes those at the URLs WITHOUT ".html" filenames. So remove the filename to get the proper repo URLs for Pipenv usage.

This fact is documented here, when Torch's team decided to finally publish PEP 503 repos: pytorch/pytorch#25639 (comment)

You may notice something else: I always specify a version with +cpu or +cu113 or whatever.

This is NECESSARY, because otherwise Pipenv doesn't know which version to install and won't get GPU acceleration. The "architecture" (GPU vs CPU vs different CUDA toolkits) is baked into the version number. It's literally what Torch decided to do, and the ONLY correct way to download them is to specify the exact version and architecture (such as +cu113) you want.

If you want to figure out the URL to the correct repos/versions in the future, do as follows:

Go to pytorch.org and select "Stable, Pip, Python, CUDA 11.3" (or whatever is the latest CUDA you may be using), and then the command textbox will reveal the repo URL for the latest CUDA toolkit. You MUST then remove the HTML filename because that's the pip-specific repo.

Visit the resulting URL, such as https://download.pytorch.org/whl/cu113/, and navigate into the desired library folder. Then search for the version you wanted, and be sure to specify its FULL version specifier, such as 1.10.1+cu113.

Oh and sometimes, the Torch team forgets to upload the latest version to the PEP 503 repo, such as right now, where 1.10.2 is available but only in the non-standard pip-style repo. That's why you should always visit the PEP 503 repo to see the latest version they've uploaded into that repo. (I would love if they completely stop providing the old-school pip-repo at all, since pip supports PEP 503 too, so they should focus on the standards-compliant repo...)

See my examples above for perfect installation instructions for the current versions as of this writing.

One final note: ALWAYS RESPECT THE TORCH PYTHON VERSIONS.

The packages will have names such as "cp37", "cp38", "cp39". As of this writing, the highest version they have created is for Python 3.9. There's no Python 3.10 version of Torch.

Therefore, it helps to install Pyenv and specify an exact Python version in your Pipenv via pipenv install --python=3.9 to ensure that you have the latest version that Torch supports and not anything "too new/unsupported". :)

Good luck.

[Arcitec]

I've gone on a posting spree in various incorrect topics (pretty much all of them were wrong), and linking people here so that random interwebs cybersurfers can find the correct way to install PyTorch and other PEP 503 repos in Pipenv!

Hopefully this will stop the flow of misinformation and "wrong solutions" that's currently out there! 😄

Welcome, everyone!

[DanielPerezJensen]

Thanks for this guide, but when I try to do: pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.10.1+cu113"

I get an AttributeError: 'list' object has no attribute 'startswith'.

The full error is:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pipenv/cli/command.py", line 194, in install
    do_install(
  File "/usr/local/lib/python3.8/dist-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'

[Arcitec]

Edit for anyone reading this thread in the future: You can skip the back and forth testing between me and Daniel. There is/was a bug in Pipenv that prevents installation of PEP 503 packages. There are multiple bug report tickets about it. The last known working version of Pipenv was 2022.1.8. The March 2022 versions are broken. I expect that Pipenv versions made after March 2022 will be fixed soon.

---

@DanielPerezJensen Thanks for posting the full traceback. It shows that your error is just a bug inside pipenv. Try updating pipenv. If that fails, try updating Python itself (you can go up to Python 3.10 now if you want to use the latest Torch version).

The bug is on this line of code in pipenv, where their code tries to do a string.startswith() on list object:

  File "/usr/local/lib/python3.8/dist-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'

PS: There are newer versions of Torch out now. Check my guide steps above for how you can update the commands to the latest version. You will need to follow my guide steps to set the correct, new version number in the command.