TCPStore is not available

[Z-Diviner]

Hello, when I use vllm3.2 and deepseek coder-33b install to start the service through Docker, the following error is reported. What is the situation?

[DoctorKey]

I have the same problem

[rkooo567]

Can you try eager_force=True?

I think in general, your environment doesn't seem to play well with cupy nccl backend.

[wangzihe1996]

I have the same problem when the version of vllm is greater than 0.3.0. Is there any possible solution?

[rkooo567]

I think that's the version cupy backend has been introduced. It is basically to enable cuda graph (cuz it didn't work with default nccl backends).

[rkooo567]

do you have any other error from logs? For some reasons, your env cannot initialize the cupy backend, but it is difficult to know with information you just posted.

One way to debug is to use this yourself

        from vllm.model_executor.parallel_utils import cupy_utils
        cupy_utils.init_process_group(
            world_size=1
            rank=0,
            host="localhost",
            port=<choose port>,
        )

And see why this fails.

[Ethan-yt]

I can reproduce when I use a spawn process to run vllm.
The TCPStore.run method raised an error: cannot pickle '_thread.lock' object.

There is another problem. Here this catch any exception without logging:

vllm/vllm/executor/ray_gpu_executor.py

Line 338 in 253a980

try:

[Z-Diviner]

do you have any other error from logs? For some reasons, your env cannot initialize the cupy backend, but it is difficult to know with information you just posted.

One way to debug is to use this yourself

        from vllm.model_executor.parallel_utils import cupy_utils
        cupy_utils.init_process_group(
            world_size=1
            rank=0,
            host="localhost",
            port=<choose port>,
        )

And see why this fails.

I used the following method to create an image and run it. Below are the specific configurations:
docker build -t harbor.4pd.io/mlsonar/vllm:cu118-vllm-0.3.2-zw-test3.1 -f /home/common-user/vllm-0.2.5/vllm_docker/Dockerfile . --network=host

[nkwangleiGIT]

same issue here with the latest vllm 0.3.3 version

[rkooo567]

@Z-Diviner can you give me the copy-able docker run command? I can try repro in my local env

[CodeScriptum]

Hello, I have the same issue with version 0.3.3 running on a ray cluster in kubernetes. Everything works fine in a single node with multiple GPU and --tensor-parallel-size enabled but running the same config with an additional worker node results in "TCPStore is not available".

I'll be happy to provide any information to help in the resolution of this issue.

Thank you

[rkooo567]

@CodeScriptum it'd be great if you can provide me a way to reproduce the issue! I'd like to make it clear that this doesn't seem like vllm or Ray issue. It must be cupy is somehow not working with your environment. I can try helping debug in this case, but I'd need to reproduce since I think it only happens in some environment.

[Z-Diviner]

如果您能为我提供一种重现该问题的方法，那就太好了！我想明确指出，这似乎不像 vllm 或 Ray 问题。它一定是 cupy 不知何故不适用于您的环境。在这种情况下，我可以尝试帮助调试，但我需要重现，因为我认为它只发生在某些环境中。

Hello, I'm sorry, I just saw that below is the command for my Docker Run：

    --ipc=host \
    --name ds33b \
    -p 17778:8080 \
    --gpus '"device=3,4,5,6"' \
    -v /mnt/contest_ceph/zhangwei03/deepseek-coder-33b-instruct:/deepseek-coder-33b-instruct \
    harbor.4pd.io/mlsonar/vllm:cu118-vllm-0.3.2-zw-test3.1 \
    --host 0.0.0.0 \
    --port 8080 \
    --served-model-name deepseek-coder-33b-instruct \
    --model /deepseek-coder-33b-instruct \
    --trust-remote-code \
    --max-num-batched-tokens 16384 \
    --max-model-len 16384 \
    --tokenizer-mode auto \
    --tensor-parallel-size 4```
Among them, harpor.4pd.io/mlsonar/vllm: cu118-vllm-0.3.2-zw test3.1 is the vllm image I built myself。

[wangzihe1996]

@CodeScriptum it'd be great if you can provide me a way to reproduce the issue! I'd like to make it clear that this doesn't seem like vllm or Ray issue. It must be cupy is somehow not working with your environment. I can try helping debug in this case, but I'd need to reproduce since I think it only happens in some environment.

I think you can try to inference with multi-process and tensor-parallel-size greater than 1. This way can reproduce the error.

And this error occurs when the version of vllm is greater than 0.3.0. So it should be due to the cupy.

The specific reasons for this can be found below.

I can reproduce when I use a spawn process to run vllm. The TCPStore.run method raised an error: cannot pickle '_thread.lock' object.

There is another problem. Here this catch any exception without logging:

vllm/vllm/executor/ray_gpu_executor.py

Line 338 in 253a980

try: