Missing pytz dependency

[tomwphillips]

Environment details

• OS type and version: macOS 11.4
• Python version: 3.8.2
• pip version: 21.1.3
• google-cloud-pubsub version: 2.7.0

Steps to reproduce

1. Install google-cloud-pubsub, e.g. pipenv install --python 3.8 google-cloud-pubsub.
2. Run code below.

Code example

from concurrent.futures import TimeoutError
from typing import Callable

from google.cloud.pubsub_v1 import SubscriberClient
from google.cloud.pubsub_v1.types import PubsubMessage


def process_messages(
    project_id: str,
    subscription_id: str,
    callback: Callable[[PubsubMessage], None],
    timeout: int = 5,
) -> None:
    subscriber = SubscriberClient()
    subscription_path = subscriber.subscription_path(project_id, subscription_id)
    streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
    with subscriber:
        try:
            streaming_pull_future.result(timeout=timeout)
        except TimeoutError:
            streaming_pull_future.cancel()  # Trigger the shutdown.
            streaming_pull_future.result()  # Block until the shutdown is complete.


if __name__ == "__main__":
    process_messages("my-project", "my-sub")

Stack trace

Traceback (most recent call last):
  File "loaders.py", line 4, in <module>
    from google.cloud.pubsub_v1 import SubscriberClient
  File "/Users/tomphillips/.local/share/virtualenvs/loaders-emkxO8nx/lib/python3.8/site-packages/google/cloud/pubsub_v1/__init__.py", line 19, in <module>
    from google.cloud.pubsub_v1 import subscriber
  File "/Users/tomphillips/.local/share/virtualenvs/loaders-emkxO8nx/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1.subscriber.client import Client
  File "/Users/tomphillips/.local/share/virtualenvs/loaders-emkxO8nx/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/client.py", line 26, in <module>
    from google.cloud.pubsub_v1.subscriber._protocol import streaming_pull_manager
  File "/Users/tomphillips/.local/share/virtualenvs/loaders-emkxO8nx/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 35, in <module>
    import google.cloud.pubsub_v1.subscriber.message
  File "/Users/tomphillips/.local/share/virtualenvs/loaders-emkxO8nx/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/message.py", line 20, in <module>
    import pytz
ModuleNotFoundError: No module named 'pytz'

[plamut]

@tomwphillips Interesting, thanks for reporting this.

The workaround should of course be to install pytz manually, but it would still be good to get to the bottom of this.

I tried reproducing this in a slightly different way:

$ python3.8 -m venv env/new-3.8
$ . env/new-3.8/bin/activate
$ pip install google-cloud-pubsub
$ pip show pytz
Name: pytz
Version: 2021.1
Summary: World timezone definitions, modern and historical
Home-page: http://pythonhosted.org/pytz
Author: Stuart Bishop
Author-email: stuart@stuartbishop.net
License: MIT
Location: /home/peter/workspace/python-pubsub/env/new-3.8/lib/python3.8/site-packages
Requires: 
Required-by: google-api-core

Using the venv module to create a fresh virtual environment worked fine, the library was installed with all its dependencies, including pytz. Could this be a bug in pipenv? Which pipenv version did you use?

FWIW, I tried this on Ubuntu Linux, but perhaps something is different on macOS. Could you perhaps try reproducing the issue with python3.8 -m venv as well?

Thanks!

[tomwphillips]

@plamut I think you might be right that this is a pipenv bug. I'm using version 2021.5.29.

Following your example, I installed google-cloud-pubsub using pip and pytz is installed.

tomphillips@OCTO-MAC-C02FJ1RRMD6M pubsub-issue % pipenv --python 3.8
Creating a virtualenv for this project...
Pipfile: /Users/tomphillips/pubsub-issue/Pipfile
Using /usr/bin/python3 (3.8.2) to create virtualenv...
⠏ Creating virtual environment...created virtual environment CPython3.8.2.final.0-64 in 577ms
  creator CPython3macOsFramework(dest=/Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/tomphillips/Library/Application Support/virtualenv)
    added seed packages: pip==21.1.3, setuptools==57.1.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY
Creating a Pipfile for this project...
tomphillips@OCTO-MAC-C02FJ1RRMD6M pubsub-issue % pipenv shell
Launching subshell in virtual environment...
 . /Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY/bin/activate
tomphillips@OCTO-MAC-C02FJ1RRMD6M pubsub-issue %  . /Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY/bin/activate
(pubsub-issue) tomphillips@OCTO-MAC-C02FJ1RRMD6M pubsub-issue % pip install google-cloud-pubsub
Collecting google-cloud-pubsub
  Downloading google_cloud_pubsub-2.7.0-py2.py3-none-any.whl (217 kB)
     |████████████████████████████████| 217 kB 6.3 MB/s 
Collecting packaging>=14.3
  Downloading packaging-21.0-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 5.8 MB/s 
Collecting grpcio<2.0dev,>=1.38.1
  Downloading grpcio-1.39.0-cp38-cp38-macosx_10_10_x86_64.whl (3.9 MB)
     |████████████████████████████████| 3.9 MB 8.5 MB/s 
Collecting libcst>=0.3.10
  Downloading libcst-0.3.20-py3-none-any.whl (514 kB)
     |████████████████████████████████| 514 kB 8.6 MB/s 
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
Collecting google-api-core[grpc]<3.0.0dev,>=1.26.0
  Downloading google_api_core-1.31.1-py2.py3-none-any.whl (93 kB)
     |████████████████████████████████| 93 kB 3.1 MB/s 
Collecting proto-plus>=1.7.1
  Using cached proto_plus-1.19.0-py3-none-any.whl (42 kB)
Collecting pytz
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Requirement already satisfied: setuptools>=40.3.0 in /Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY/lib/python3.8/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-pubsub) (57.1.0)
Collecting six>=1.13.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting protobuf>=3.12.0
  Using cached protobuf-3.17.3-cp38-cp38-macosx_10_9_x86_64.whl (1.0 MB)
Collecting google-auth<2.0dev,>=1.25.0
  Downloading google_auth-1.34.0-py2.py3-none-any.whl (152 kB)
     |████████████████████████████████| 152 kB 8.7 MB/s 
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting typing-extensions>=3.7.4.2
  Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting typing-inspect>=0.4.0
  Downloading typing_inspect-0.7.1-py3-none-any.whl (8.4 kB)
Collecting pyyaml>=5.2
  Using cached PyYAML-5.4.1-cp38-cp38-macosx_10_9_x86_64.whl (253 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting charset-normalizer~=2.0.0
  Downloading charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting mypy-extensions>=0.3.0
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Building wheels for collected packages: grpc-google-iam-v1
  Building wheel for grpc-google-iam-v1 (setup.py) ... done
  Created wheel for grpc-google-iam-v1: filename=grpc_google_iam_v1-0.12.3-py3-none-any.whl size=18514 sha256=9931cf8c6139648a04596429d70b1f559538aebbbcfabd264ce8a359bdf065b3
  Stored in directory: /Users/tomphillips/Library/Caches/pip/wheels/8f/b9/13/fce3d62261f63c01b28281fe6a9d704a7af65d96ff2c88552e
Successfully built grpc-google-iam-v1
Installing collected packages: six, pyasn1, urllib3, rsa, pyparsing, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, typing-extensions, requests, pytz, packaging, mypy-extensions, grpcio, googleapis-common-protos, google-auth, typing-inspect, pyyaml, google-api-core, proto-plus, libcst, grpc-google-iam-v1, google-cloud-pubsub
Successfully installed cachetools-4.2.2 certifi-2021.5.30 charset-normalizer-2.0.4 google-api-core-1.31.1 google-auth-1.34.0 google-cloud-pubsub-2.7.0 googleapis-common-protos-1.53.0 grpc-google-iam-v1-0.12.3 grpcio-1.39.0 idna-3.2 libcst-0.3.20 mypy-extensions-0.4.3 packaging-21.0 proto-plus-1.19.0 protobuf-3.17.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-2.4.7 pytz-2021.1 pyyaml-5.4.1 requests-2.26.0 rsa-4.7.2 six-1.16.0 typing-extensions-3.10.0.0 typing-inspect-0.7.1 urllib3-1.26.6
(pubsub-issue) tomphillips@OCTO-MAC-C02FJ1RRMD6M pubsub-issue % pip show pytz
Name: pytz
Version: 2021.1
Summary: World timezone definitions, modern and historical
Home-page: http://pythonhosted.org/pytz
Author: Stuart Bishop
Author-email: stuart@stuartbishop.net
License: MIT
Location: /Users/tomphillips/.local/share/virtualenvs/pubsub-issue-CgdtFZxY/lib/python3.8/site-packages
Requires: 
Required-by: google-api-core

[plamut]

@tomwphillips Thanks for confirming that venv works for you, too, it indeed seems likely that it's something in pipenv. Have you perhaps reported the issue there yet?

Also, would using venv in your process represent a feasible workaround?

[tomwphillips]

I'll report it to pipenv and specify pytz as a dependency in my Pipfile as a workaround. Thanks, @plamut.

[plamut]

@tomwphillips FYI, we got the same issue report elsewhere, and it seems that it's a bug in pip dependency resolver.

We will remove the transitive dependency pytz (through google-api-core) in #472, the change should be visible in the next release.

[plamut]

@tomwphillips Just mentioning if interested - it turned out this was caused because the library implicitly depended on pytz, which was pulled in by the google-api-core dependency. When the latter dropped pytz, it broke the Pub/Sub client and a few other client libraries as well.

This has now been fixed in the v2.7.1 release. The Python Pub/Sub client does not use pytz anymore, thus any workarounds related to that can now be removed as well.