Pipenv searches virtual environment's python binary in wrong path

[elegos]

Hello,

I'm having trouble using Pipenv lately. In particular, after I updated my OS (Fedora 36), pipenv during the install command tries to search the python binary inside {local_virtualenv}/local/bin/python, while the symlink to python is in {local_virtualenv}/bin/python. I tried to remove and reinstall the virtual environment (pipenv --rm and pipenv install), but the problem persists.

Application versions

➜  etl git:(develop) pip --version
pip 22.1.1 from /home/gfurlan/.local/lib/python3.10/site-packages/pip (python 3.10)
➜  etl git:(develop) pipenv --version
pipenv, version 2022.5.2
➜  etl git:(develop) python --version
Python 3.10.4
➜  etl git:(develop) which python
/usr/bin/python

Pipenv install

➜  etl git:(develop) pipenv install -d
Creating a virtualenv for this project...
Pipfile: /home/gfurlan/src/project/glue/etl/Pipfile
Using /usr/bin/python3.7m (3.7.13) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.7.13.final.0-64 in 89ms
  creator CPython3Posix(dest=/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/gfurlan/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Traceback (most recent call last):
  File "/home/gfurlan/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/cli/options.py", line 56, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/cli/command.py", line 222, in install
    do_install(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 1965, in do_install
    ensure_project(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 553, in ensure_project
    ensure_virtualenv(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 486, in ensure_virtualenv
    do_create_virtualenv(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/core.py", line 1013, in do_create_virtualenv
    project._environment = Environment(
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/environment.py", line 70, in __init__
    self._base_paths = self.get_paths()
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/environment.py", line 394, in get_paths
    c = subprocess_run(command)
  File "/home/gfurlan/.local/lib/python3.10/site-packages/pipenv/utils/processes.py", line 75, in subprocess_run
    return subprocess.run(
  File "/usr/lib64/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib64/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/local/bin/python'

pipenv shell and echo $PATH:

➜  etl git:(develop) pipenv shell
Launching subshell in virtual environment...
 . /home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin/activate
➜  etl git:(develop)  . /home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin/activate
(etl) ➜  etl git:(develop) echo $PATH
/home/gfurlan/.local/share/virtualenvs/etl-XCf49Mm9/bin:/home/gfurlan/.local/bin:/home/gfurlan/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

As you can see the PATH variable is correctly set, though pipenv tries to use the local prefix for calling python for some reason.

May you please help me sorting this out?

Thanks

EDIT: I think I got the point where the issue starts, that is in environment.py, in function script_basedir which gets the paths from sysconfig.get_paths, setting base and platbase variables to the virtual environment prefix. That function returns for the scripts key /my/custom/venv/path/local/bin.

Now the question is: if it uses sysconfig.get_paths to find python, shouldn't it set PATH and the symlink to python accordingly, during the initial setup?

EDIT2:
reading sysconfig.py from the OS installation, the following is written:

# backup the original posix_prefix as rpm_prefix
# RPM packages use it and we need to be able to read it even when changed
_INSTALL_SCHEMES['rpm_prefix'] = _INSTALL_SCHEMES['posix_prefix']
# Virtualenv >= 20.10.0 favors the "venv" scheme over the defaults when creating virtual environments.
# See: https://github.com/pypa/virtualenv/commit/8da79db86d8a5c74d03667a40e64ff832076445e
# See: https://bugs.python.org/issue45413
# "venv" should be the same as the unpatched posix_prefix for us,
# so new virtual environments aren't created with paths like venv/local/bin/python.
_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_prefix']

Thus in pipenv's environment.py file, script_basedir function, if I overwrite the install_scheme as follows:

install_scheme = "nt" if (os.name == "nt") else "venv" # "venv" instad of "posix_prefix"

the path is now correct again.

The problem seems to be fixed now. I wonder if this is the default Python behavior, of if it's some kind of Fedora 36 customization?

EDIT3:
and I'm still unsure whether the EDIT2 sysconfig.py code is a custom Fedora add-in, but I found this issue on the CPython github: python/cpython#89576

[undefx]

same here. many thanks @elegos for your detailed workaround. it works for me also.

system info:

$ cat /etc/redhat-release 
Fedora release 36 (Thirty Six)

$ pip --version
pip 21.3.1 from /usr/lib/python3.10/site-packages/pip (python 3.10)

$ pipenv --version
pipenv, version 2022.4.8

$ python --version
Python 3.10.4

$ which python
/usr/bin/python

sample of original failure:

$ pipenv --rm
Removing virtualenv (/home/username/.local/share/virtualenvs/sample-_RaNdOm)...

$ pipenv install
Creating a virtualenv for this project...
Pipfile: /home/username/path/Pipfile
Using /usr/bin/python (3.10.4) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.10.4.final.0-64 in 194ms
  creator CPython3Posix(dest=/home/username/.local/share/virtualenvs/sample-__RaNdOm, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/username/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/home/username/.local/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)

[... snip ...]

FileNotFoundError: [Errno 2] No such file or directory: '/home/username/.local/share/virtualenvs/sample-_RaNdOm/local/bin/python'

distilled @elegos workaround:

# find pipenv install location
$ python3 -c 'import os; import pipenv as p; print(os.path.split(p.__file__)[0])'
/home/username/.local/lib/python3.10/site-packages/pipenv

# locate file `environment.py` under path above
$ find /home/username/.local/lib/python3.10/site-packages/pipenv | grep -v vendor | grep environment.py
/home/username/.local/lib/python3.10/site-packages/pipenv/environment.py

# replace 'posix_prefix' with 'venv' in file above
$ sed -i 's/posix_prefix"/venv"  # was posix_prefix, not venv/g' /home/username/.local/lib/python3.10/site-packages/pipenv/environment.py

# verify that it works
$ pipenv check
Creating a virtualenv for this project...
Pipfile: /home/username/path/Pipfile
Using /usr/bin/python (3.10.4) to create virtualenv...
⠴ Creating virtual environment...created virtual environment CPython3.10.4.final.0-64 in 307ms
  creator CPython3Posix(dest=/home/username/.local/share/virtualenvs/sample-_RaNdOm, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/username/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/username/.local/share/virtualenvs/sample-_RaNdOm
Creating a Pipfile for this project...
Checking PEP 508 requirements...
Passed!
Checking installed package safety...
All good!

---

edit: formatting

Thank you for the workaround @elegos - This resolved the same issue on my system, which occurred after recently applying Fedora updates.

$ cat /etc/fedora-release 
Fedora release 36 (Thirty Six)

$ pip --version
pip 22.1 from /home/gareth/.local/lib/python3.10/site-packages/pip (python 3.10)

$ pipenv --version
pipenv, version 2022.5.2

$python --version
Python 3.10.4

$ which python
/usr/bin/python

[RonaldTheodoro]

Thanks @elegos the workaround also worked for me, this problem started after I update to fedora 36.

$ cat /etc/fedora-release 
Fedora release 36 (Thirty Six)
$ pip --version
pip 22.1 from /home/ronaldtheodoro/.local/lib/python3.10/site-packages/pip (python 3.10)
$ pipenv --version
pipenv, version 2022.5.2
$ python --version
Python 3.10.4
$ which python
/usr/bin/python

[matteius]

@RonaldTheodoro @garethheath @undefx and @elegos -- Do you think this PR would fix the issue and eliminate the need for the workaround? #5096

[elegos]

Hello @matteius yes, it should exactly do the trick we need :)

[torsava]

Hi @elegos , I've already deployed the fix for this (#5096) in Fedora rawhide and we're testing it there. The RPM might be installable on F36 too if you'd like to help us test it: https://koji.fedoraproject.org/koji/buildinfo?buildID=1965245

[mattiasb]

Hi @elegos , I've already deployed the fix for this (#5096) in Fedora rawhide and we're testing it there. The RPM might be installable on F36 too if you'd like to help us test it: https://koji.fedoraproject.org/koji/buildinfo?buildID=1965245

Do you expect this to land in F36 later on? (It's one of the issues holding me back from upgrading currently. :)).

[jguiditta]

Hi @elegos , I've already deployed the fix for this (#5096) in Fedora rawhide and we're testing it there. The RPM might be installable on F36 too if you'd like to help us test it: https://koji.fedoraproject.org/koji/buildinfo?buildID=1965245

Do you expect this to land in F36 later on? (It's one of the issues holding me back from upgrading currently. :)).

+1 We use this in CI, so if this is not going into an F36 rpm update (or at least a pypi update) soon, we will have to implement a workaround in CI, which I would prefer not to do if we can help it.

[matteius]

FWIW it has been merged into master branch of pipenv and I plan to do a pipenv release at some point soon, like say in the first week of June, probably. Trying to see what else we can get into that release that would be fixing an issue as well.

[torsava]

Hi @elegos , I've already deployed the fix for this (#5096) in Fedora rawhide and we're testing it there. The RPM might be installable on F36 too if you'd like to help us test it: https://koji.fedoraproject.org/koji/buildinfo?buildID=1965245

Do you expect this to land in F36 later on? (It's one of the issues holding me back from upgrading currently. :)).

We have an older version of pipenv packaged in F36 which according to my testing does not suffer from this issue. I suspect you're using a newer version of pipenv installed from PyPI and thus getting the error, am I correct? If so, a new release of pipenv will fix that.