Cannot install pip packages

[Tonkonozhenko]

Hi

I get next error when trying to install package:

(.venv) ➜  ml pip install scipy 
Looking in indexes: https://pypi.org/simple, https://xxxxxxx/pypi/packages
Collecting scipy
  Downloading https://files.pythonhosted.org/packages/9a/3f/fb702aa44220014ee8803a5a0bb597e133a512d75d21463d1a2267c059df/scipy-1.0.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (16.4MB)
    4% |█▍                              | 727kB 121kB/s eta 0:02:10Exception:
Traceback (most recent call last):
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 60, in read
    data = self.__fp.read(amt)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 449, in read
    n = self.readinto(b)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 493, in readinto
    n = self.fp.readinto(b)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 291, in run
    resolver.resolve(requirement_set)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 103, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 257, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/resolve.py", line 210, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/operations/prepare.py", line 310, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 837, in unpack_url
    progress_bar=progress_bar
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 674, in unpack_http_url
    progress_bar)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 898, in _download_http_url
    _download_url(resp, link, content_file, hashes, progress_bar)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 618, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/utils/hashes.py", line 48, in check_against_chunks
    for chunk in chunks:
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 586, in written_chunks
    for chunk in chunks:
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/utils/ui.py", line 159, in iter
    for x in it:
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_internal/download.py", line 575, in resp_read
    decode_content=False):
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 401, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/alex/projects/stuff-meta/ml/.venv/lib/python3.6/site-packages/pip/_vendor/urllib3/response.py", line 307, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

[Tonkonozhenko]

Now that command worked, but I hope stacktrace will be useful for you.

[ewdurbin]

Hello @Tonkonozhenko! Was this error reproducible more than once? Occasional timeouts will happen.

If this was reproducible for any duration of time we may need to do some tuning of our CDN configuration.

[Tonkonozhenko]

Hi @ewdurbin
I tried it at 9:55 UTC after my colleague told me that he has the error (9:07 UTC).

[Tonkonozhenko]

@ewdurbin
We are in different locations - Kiev and Odessa, both in Ukraine

[di]

@Tonkonozhenko Are you or your colleague still experiencing this?

[Tonkonozhenko]

Hi @di
Yes, and I experience same issues from my Jenkins (AWS eu-west-1 - Ireland):

Collecting retrying==1.3.3 (from -r requirements.txt (line 47))
  Could not find a version that satisfies the requirement retrying==1.3.3 (from -r requirements.txt (line 47)) (from versions: )
No matching distribution found for retrying==1.3.3 (from -r requirements.txt (line 47))

[di]

@Tonkonozhenko This seems a bit different from your original error. What's the output of pip install retrying==1.3.3 -vvv?

[Tonkonozhenko]

@di now it works. On jenkins I get messages about different packages each time. Today I got

Collecting pycparser==2.18 (from -r requirements.txt (line 37))
  Could not find a version that satisfies the requirement pycparser==2.18 (from -r requirements.txt (line 37)) (from versions: )
No matching distribution found for pycparser==2.18 (from -r requirements.txt (line 37))

Most often I get messages about futures==3.2.0.

[ssbarnea]

Since mid-April migration to the CDN, we see these errors quite often, probably in 1-3% of builds (which is a huge number). There is nothing special about the package name, the seem to be random and apparently pip does not attempt to retry, it just fail with the conclusion that there are no packages at all. This failed even with not pinned or open ranges, looks like an index requires returns nothing or an error is treated as a 404.

Since two hours ago I eneabled PIP_VERBOSE=1 and I hope that soon I will have more detailed output.

[ssbarnea]

I already started to get failure results, see https://gist.github.com/ssbarnea/78df2a80137cca04ca72872b1d14498f as the last failure with extra logging.

I counted at least 104 build failures caused by this since the CDN migration. Read value could be much more as that query reported only failures matching No matching distribution found for text.

[di]

@ssbarnea Can you provide the command which produced that output? I would expect there to be more output from pip about trying to install the deprecation package.

[ssbarnea]

@di I updated the gist with a more detailed one. What was the original install command is irrelevant as the is no way to reproduce the problem in a consistent way. Talke at look at https://gist.github.com/ssbarnea/78df2a80137cca04ca72872b1d14498f#file-pypi-pip-failure-log-L34

My impression is that this explains what happens:

  https://pypi.org:443 "GET /simple/deprecation/ HTTP/1.1" 304 0
  Skipping page https://pypi.org/simple/deprecation/ because of Content-Type: unknown

Mainly pip decides that there is no package when it does receive a 304 response.

[njsmith]

See also: pypa/pip#5345