[Bug] sglang[all]>=0.4.4.post2 installation environment is very confusing

[17Reset]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

I tried to reinstall the latest version of SGLang, but I get all sorts of errors like the following

Reproduction

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ uv pip install --no-cache "sglang[all]>=0.4.4.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
Using Python 3.12.3 environment at: sglang_env
Resolved 129 packages in 9.99s
Prepared 129 packages in 35m 27s
Installed 129 packages in 160ms
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.11.14
 + aiosignal==1.3.2
 + airportsdata==20250224
 + annotated-types==0.7.0
 + anthropic==0.49.0
 + anyio==4.9.0
 + asttokens==3.0.0
 + attrs==25.3.0
 + certifi==2025.1.31
 + cffi==1.17.1
 + charset-normalizer==3.4.1
 + click==8.1.8
 + cloudpickle==3.1.1
 + cuda-bindings==12.8.0
 + cuda-python==12.8.0
 + datasets==3.4.1
 + decorator==5.2.1
 + decord==0.6.0
 + dill==0.3.8
 + diskcache==5.6.3
 + distro==1.9.0
 + executing==2.2.0
 + fastapi==0.115.12
 + filelock==3.18.0
 + flashinfer-python==0.2.3+cu124torch2.5
 + frozenlist==1.5.0
 + fsspec==2024.12.0
 + h11==0.14.0
 + hf-transfer==0.1.9
 + httpcore==1.0.7
 + httpx==0.28.1
 + huggingface-hub==0.29.3
 + idna==3.10
 + importlib-metadata==8.6.1
 + interegular==0.3.3
 + ipython==9.0.2
 + ipython-pygments-lexers==1.1.1
 + jedi==0.19.2
 + jinja2==3.1.6
 + jiter==0.9.0
 + jsonschema==4.23.0
 + jsonschema-specifications==2024.10.1
 + lark==1.2.2
 + litellm==1.64.1
 + llguidance==0.7.10
 + markupsafe==3.0.2
 + matplotlib-inline==0.1.7
 + modelscope==1.24.0
 + mpmath==1.3.0
 + multidict==6.2.0
 + multiprocess==0.70.16
 + nest-asyncio==1.6.0
 + networkx==3.4.2
 + ninja==1.11.1.4
 + numpy==2.2.4
 + nvidia-cublas-cu12==12.4.5.8
 + nvidia-cuda-cupti-cu12==12.4.127
 + nvidia-cuda-nvrtc-cu12==12.4.127
 + nvidia-cuda-runtime-cu12==12.4.127
 + nvidia-cudnn-cu12==9.1.0.70
 + nvidia-cufft-cu12==11.2.1.3
 + nvidia-curand-cu12==10.3.5.147
 + nvidia-cusolver-cu12==11.6.1.9
 + nvidia-cusparse-cu12==12.3.1.170
 + nvidia-nccl-cu12==2.21.5
 + nvidia-nvjitlink-cu12==12.4.127
 + nvidia-nvtx-cu12==12.4.127
 + openai==1.68.2
 + orjson==3.10.16
 + outlines==0.1.11
 + outlines-core==0.1.26
 + packaging==24.2
 + pandas==2.2.3
 + parso==0.8.4
 + pexpect==4.9.0
 + pillow==11.1.0
 + prometheus-client==0.21.1
 + prompt-toolkit==3.0.50
 + propcache==0.3.1
 + psutil==7.0.0
 + ptyprocess==0.7.0
 + pure-eval==0.2.3
 + pyarrow==19.0.1
 + pycountry==24.6.1
 + pycparser==2.22
 + pydantic==2.10.6
 + pydantic-core==2.27.2
 + pygments==2.19.1
 + python-dateutil==2.9.0.post0
 + python-dotenv==1.1.0
 + python-multipart==0.0.20
 + pytz==2025.2
 + pyyaml==6.0.2
 + pyzmq==26.3.0
 + referencing==0.36.2
 + regex==2024.11.6
 + requests==2.32.3
 + rpds-py==0.24.0
 + safetensors==0.5.3
 + sentencepiece==0.2.0
 + setproctitle==1.3.5
 + setuptools==78.1.0
 + sgl-kernel==0.0.5.post3
 + sglang==0.4.4.post2
 + six==1.17.0
 + sniffio==1.3.1
 + soundfile==0.13.1
 + stack-data==0.6.3
 + starlette==0.46.1
 + sympy==1.13.1
 + tiktoken==0.9.0
 + tokenizers==0.21.1
 + torch==2.5.1
 + torchao==0.9.0
 + tqdm==4.67.1
 + traitlets==5.14.3
 + transformers==4.50.0
 + triton==3.1.0
 + typing-extensions==4.13.0
 + tzdata==2025.2
 + urllib3==2.3.0
 + uvicorn==0.34.0
 + uvloop==0.21.0
 + wcwidth==0.2.13
 + xgrammar==0.1.16
 + xxhash==3.5.0
 + yarl==1.18.3
 + zipp==3.21.0

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ python -m sglang.launch_server --model-path /mnt/Model/Qwen --tp-size 4 --reasoning-parser deepseek-r1 --host 0.0.0.0
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/launch_server.py", line 6, in <module>
    from sglang.srt.entrypoints.http_server import launch_server
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py", line 45, in <module>
    from sglang.srt.entrypoints.engine import _launch_subprocesses
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/entrypoints/engine.py", line 40, in <module>
    from sglang.srt.managers.data_parallel_controller import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/data_parallel_controller.py", line 27, in <module>
    from sglang.srt.managers.io_struct import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/io_struct.py", line 25, in <module>
    from sglang.srt.managers.schedule_batch import BaseFinishReason
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/schedule_batch.py", line 43, in <module>
    from sglang.srt.configs.model_config import ModelConfig
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/configs/__init__.py", line 3, in <module>
    from sglang.srt.configs.deepseekvl2 import DeepseekVL2Config
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/configs/deepseekvl2.py", line 7, in <module>
    import torchvision.transforms as T
ModuleNotFoundError: No module named 'torchvision'

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ uv pip install torchvision
Using Python 3.12.3 environment at: sglang_env
Resolved 27 packages in 758ms
Uninstalled 2 packages in 528ms
Installed 4 packages in 105ms
 + nvidia-cusparselt-cu12==0.6.2
 - torch==2.5.1
 + torch==2.6.0
 + torchvision==0.21.0
 - triton==3.1.0
 + triton==3.2.0

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ python -m sglang.launch_server --model-path /mnt/Model/Qwen --tp-size 4 --reasoning-parser deepseek-r1 --host 0.0.0.0
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/launch_server.py", line 6, in <module>
    from sglang.srt.entrypoints.http_server import launch_server
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py", line 45, in <module>
    from sglang.srt.entrypoints.engine import _launch_subprocesses
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/entrypoints/engine.py", line 40, in <module>
    from sglang.srt.managers.data_parallel_controller import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/data_parallel_controller.py", line 27, in <module>
    from sglang.srt.managers.io_struct import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/io_struct.py", line 25, in <module>
    from sglang.srt.managers.schedule_batch import BaseFinishReason
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/managers/schedule_batch.py", line 43, in <module>
    from sglang.srt.configs.model_config import ModelConfig
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 25, in <module>
    from sglang.srt.layers.quantization import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/layers/quantization/__init__.py", line 46, in <module>
    from sglang.srt.layers.quantization.compressed_tensors.compressed_tensors import (
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py", line 9, in <module>
    from compressed_tensors.config import (
ModuleNotFoundError: No module named 'compressed_tensors'

Environment

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ python -m sglang.check_env
Python: 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3: NVIDIA RTX 6000 Ada Generation
GPU 0,1,2,3 Compute Capability: 8.9
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 570.124.06
PyTorch: 2.6.0+cu124
sglang: 0.4.4.post2
sgl_kernel: 0.0.5.post3
flashinfer: Module Not Found
triton: 3.2.0
transformers: 4.50.0
torchao: 0.9.0
numpy: 2.2.4
aiohttp: 3.11.14
fastapi: 0.115.12
hf_transfer: 0.1.9
huggingface_hub: 0.29.3
interegular: 0.3.3
modelscope: 1.24.0
orjson: 3.10.16
outlines: 0.1.11
packaging: 24.2
psutil: 7.0.0
pydantic: 2.10.6
multipart: Module Not Found
zmq: Module Not Found
uvicorn: 0.34.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.16
openai: 1.68.2
tiktoken: 0.9.0
anthropic: 0.49.0
litellm: 1.64.1
decord: 0.6.0
NVIDIA Topology:
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    SYS     SYS     20-29   2               N/A
GPU1    NODE     X      SYS     SYS     20-29   2               N/A
GPU2    SYS     SYS      X      NODE    60-69   6               N/A
GPU3    SYS     SYS     NODE     X      60-69   6               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

ulimit soft: 1024

[bgyoon]

Same issue. Temporarily, uv pip install vllm==0.7.2 seems to fix it for me.

[shuaills]

Please create a clean env and try again. It seems there are problems with the installation of torch.

[anapple-hub]

I use the sglang:latest docker image, and the same error

[17Reset]

I'm using the install from source method and the latest version doesn't have the above problem anymore!

# Use the last release branch
uv venv sglang_env
source sglang_env/bin/activate

git clone --depth 1 https://github.com/sgl-project/sglang.git
cd sglang

uv pip install -e "python[all]" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

[17Reset]

But there is an error loading the GGUF file.

(sglang_env) xlab@xlab:/mnt/Agent/SGLang$ python -m sglang.launch_server --model-path /mnt/Model/Google/Gemma3_27B_BF16.gguf --tp-size 4 --host 0.0.0.0 --mem-fraction-static 0.6 --quantization gguf
[2025-03-28 11:23:51] server_args=ServerArgs(model_path='/mnt/Model/Google/Gemma3_27B_BF16.gguf', tokenizer_path='/mnt/Model/Google/Gemma3_27B_BF16.gguf', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='gguf', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization='gguf', quantization_param_path=None, context_length=None, device='cuda', served_model_name='/mnt/Model/Google/Gemma3_27B_BF16.gguf', chat_template=None, completion_template=None, is_embedding=False, revision=None, host='0.0.0.0', port=30000, mem_fraction_static=0.6, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=4, stream_interval=1, stream_output=False, random_seed=339382140, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend='flashinfer', sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=5, speculative_eagle_topk=4, speculative_num_draft_tokens=8, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998)
Loading a GGUF checkpoint in PyTorch, requires both PyTorch and GGUF>=0.10.0 to be installed. Please see https://pytorch.org/ and https://github.com/ggerganov/llama.cpp/tree/master/gguf-py for installation instructions.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/mnt/Agent/SGLang/sglang/python/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/mnt/Agent/SGLang/sglang/python/sglang/srt/entrypoints/http_server.py", line 673, in launch_server
    tokenizer_manager, scheduler_info = _launch_subprocesses(server_args=server_args)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang/python/sglang/srt/entrypoints/engine.py", line 546, in _launch_subprocesses
    tokenizer_manager = TokenizerManager(server_args, port_args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang/python/sglang/srt/managers/tokenizer_manager.py", line 159, in __init__
    self.model_config = ModelConfig(
                        ^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang/python/sglang/srt/configs/model_config.py", line 64, in __init__
    self.hf_config = get_config(
                     ^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang/python/sglang/srt/hf_transformers_utils.py", line 75, in get_config
    config = AutoConfig.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1096, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/transformers/configuration_utils.py", line 594, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/transformers/configuration_utils.py", line 685, in _get_config_dict
    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)["config"]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Agent/SGLang/sglang_env/lib/python3.12/site-packages/transformers/modeling_gguf_pytorch_utils.py", line 360, in load_gguf_checkpoint
    raise ImportError("Please install torch and gguf>=0.10.0 to load a GGUF checkpoint in PyTorch.")
ImportError: Please install torch and gguf>=0.10.0 to load a GGUF checkpoint in PyTorch.

[GindaChen]

Encountered this, and I did a small hack to make it work:

mkdir ~/envs/
uv venv ~/envs/sglang --python 3.12 --seed

uv pip install --python ~/envs/sglang/ "sglang[all]>=0.4.4.post2"  --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
uv pip install --python ~/envs/sglang/ torchvision torch==2.5.1 compressed_tensors gguf partial_json_parser

Tested with

uv run --python ~/envs/sglang/ python -m sglang.launch_server --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0