Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip.

[Gobz]

[guillaumekln]

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:

pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

[hoonlight]

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:

pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

Is it possible to get it to work with VENV on Windows by any chance? I have cuda 11.8, cudnn 11.x all installed properly, but it's not working.

Could not load library cudnn_cnn_infer64_8.dll. Error code 126
Please make sure cudnn_cnn_infer64_8.dll is in your library path!

would installing pytorch help?

What's really weird is that a few days ago, it worked fine without any CUDA, CUDNN, zlib installation. After a clean install of windows, it doesn't work.

edited: The same error occurs even in a non-virtual environment. I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

[guillaumekln]

This solution does not work on Windows because NVIDIA only provides Linux binaries for the cuDNN package:

https://pypi.org/project/nvidia-cudnn-cu11/#files

Installing PyTorch will not help in this case.

I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

Maybe you should double-check the PATH setting. I know it can be tricky to get it right.

You could also look at @Purfview's standalone executable: https://github.com/Purfview/whisper-standalone-win. There you can download the NVIDIA libraries and simply put them in the same directory as the executable.

[hoonlight]

This solution does not work on Windows because NVIDIA only provides Linux binaries for the cuDNN package:

https://pypi.org/project/nvidia-cudnn-cu11/#files

Installing PyTorch will not help in this case.

I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

Maybe you should double-check the PATH setting. I know it can be tricky to get it right.

You could also look at @Purfview's standalone executable: https://github.com/Purfview/whisper-standalone-win. There you can download the NVIDIA libraries and simply put them in the same directory as the executable.

Thanks, I'll try again to make sure I'm not missing anything.
I just have one question: is the installation of cuda, cudnn "essential" for running faster-whisper on a gpu?
I don't remember installing cudnn (before the windows reinstall), but only because device="cuda" just worked fine.
If they are required to be installed, my memory is probably wrong.

[guillaumekln]

Yes these libraries are required for GPU execution. An error is raised when you try to use the GPU but these libraries cannot be found.

[yangyaofei]

According to this solution, I get the code below:

try:
    import os
    import nvidia.cublas.lib
    import nvidia.cudnn.lib

    cublas_path = os.path.dirname(nvidia.cublas.lib.__file__)
    cudnn_path = os.path.dirname(nvidia.cudnn.lib.__file__)
    os.environ["LD_LIBRARY_PATH"] = f"{cublas_path}:{cudnn_path}"
except ModuleNotFoundError:
    pass

But still get same error, like the PATH is not set.

[guillaumekln]

The environment variable LD_LIBRARY_PATH should be set before starting the Python process.

[yangyaofei]

@guillaumekln oh, thank you to explain this. It seems like the once for all solution is not easy😂.

[Feanix-Fyre]

I'm pretty certain that NVIDIA offers cuDNN files of some sort for Windows at https://developer.nvidia.com/rdp/cudnn-download. I do remember getting faster-whisper to work on the GPU on Windows but I do remember it being quite the hassle, either due to my inexperience or genuine difficulty.

[guillaumekln]

Yes, you can download cuDNN binaries for Windows on the NVIDIA website.

But this issue is about installing cuDNN via PyPI with:

pip install nvidia-cudnn-cu11

This does not work on Windows.

[lugia19]

If you're on Windows, a functioning workaround I've found is to install torch with cuda support, then add the "lib" subfolder to your PATH.

It works since the lib folder contains DLLs for both cuBLAS and cuDNN.

[vadi2]

The trick in #153 (comment) is not working for me, what am I doing wrong?

(faster-whisper) vadi@barbar:~$ pip install nvidia-cublas-cu11 nvidia-cudnn-cu11
Requirement already satisfied: nvidia-cublas-cu11 in ./Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages (11.11.3.6)
Requirement already satisfied: nvidia-cudnn-cu11 in ./Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages (8.9.2.26)
(faster-whisper) vadi@barbar:~$ export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(faster-whisper) vadi@barbar:~$ echo $LD_LIBRARY_PATH 
/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/nvidia/cublas/lib:/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/nvidia/cudnn/lib
(faster-whisper) vadi@barbar:~$ python ~/Downloads/faster-whisper.py 
Traceback (most recent call last):
  File "/home/vadi/Downloads/faster-whisper.py", line 6, in <module>
    model = WhisperModel(model_size, device="cuda", compute_type="float16")
  File "/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 124, in __init__
    self.model = ctranslate2.models.Whisper(
RuntimeError: CUDA failed with error unknown error
(faster-whisper) vadi@barbar:~$ 

Running Nvidia driver 535.54.03 on RTX 4080 on Ubuntu 22.04.