Skip to content

Feature/modelscope model download #8083

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhyncs merged 19 commits into from
Aug 1, 2025
Merged

Feature/modelscope model download #8083

Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d7754db
add modelscope model download support in bench_offline_throughput.py …
yrk111222 Jul 16, 2025
a24e25f
restore empty lines
yrk111222 Jul 16, 2025
d9a2f21
add modelscope model download support in bench_offline_throughput.py …
yrk111222 Jul 16, 2025
f29ab45
remove redundant module imports
yrk111222 Jul 16, 2025
34f4103
modify the error reporting logic
yrk111222 Jul 16, 2025
d93e98f
remove redundant code
yrk111222 Jul 16, 2025
7aefb56
remove redundant parameters
yrk111222 Jul 16, 2025
8abb4c9
add local model path judgment
yrk111222 Jul 16, 2025
93c0679
modify judgment logic
yrk111222 Jul 16, 2025
fa09096
Merge branch 'main' into feature/modelscope-model-download
ping1jing2 Jul 28, 2025
2c63a4b
modify code style
yrk111222 Jul 28, 2025
33cfa09
Merge branch 'main' into feature/modelscope-model-download
ping1jing2 Jul 28, 2025
11dd3ad
Merge branch 'main' into feature/modelscope-model-download
Alcanderian Jul 29, 2025
b091f69
Merge branch 'main' into feature/modelscope-model-download
ping1jing2 Jul 29, 2025
a045bdc
Merge branch 'main' into feature/modelscope-model-download
Alcanderian Jul 29, 2025
1fa00db
Merge branch 'main' into feature/modelscope-model-download
ping1jing2 Jul 29, 2025
182cb89
Merge branch 'main' into feature/modelscope-model-download
Alcanderian Jul 30, 2025
4baa756
Merge branch 'main' into feature/modelscope-model-download
Alcanderian Jul 31, 2025
18b14fe
Merge branch 'main' into feature/modelscope-model-download
Alcanderian Aug 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions python/sglang/bench_offline_throughput.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,22 @@ def throughput_test(
ServerArgs.add_cli_args(parser)
BenchArgs.add_cli_args(parser)
args = parser.parse_args()

# handling Modelscope model downloads
Copy link
Copy Markdown

@yingdachen yingdachen Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ModelScope

if os.getenv("SGLANG_USE_MODELSCOPE", "false").lower() in ("true", "1"):
try:
from modelscope import snapshot_download
print(f"Using ModelScope to download model: {args.model_path}")
Copy link
Copy Markdown

@yingdachen yingdachen Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的args.model_path有可能是模型的本地路径吗？是不是要做个判断？


# download the model and replace args.model_path
args.model_path = snapshot_download(
args.model_path,
)
print(f"Model downloaded to: {args.model_path}")
except Exception as e:
print(f"ModelScope download failed: {str(e)}")
raise e

Comment on lines +421 to +440
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

  • Redundant Import: The import os on line 423 is redundant since os is already imported at the top of the file (line 19).
  • Improved Exception Handling: The exception handling can be more specific. Catch ImportError for optional dependencies like modelscope separately to provide a more helpful error message if the package isn't installed.
    # handling Modelscope model downloads
    if os.getenv("SGLANG_USE_MODELSCOPE", "false").lower() in ("true", "1"):
        try:
            from modelscope import snapshot_download
        except ImportError:
            print(
                "Error: modelscope is not installed. "
                "Please install it via `pip install modelscope` to use this feature."
            )
            raise

        try:
            os.environ["TRANSFORMERS_OFFLINE"] = "1"
            os.environ["HF_DATASETS_OFFLINE"] = "1"
            os.environ["HF_EVALUATE_OFFLINE"] = "1"

            print(f"Using ModelScope to download model: {args.model_path}")

            # download the model and replace args.model_path
            args.model_path = snapshot_download(
                args.model_path,
                cache_dir=os.getenv("MODELSCOPE_CACHE"),
                revision=os.getenv("MODEL_REVISION"),
                local_files_only=False,
            )
            print(f"Model downloaded to: {args.model_path}")

            if hasattr(args, "tokenizer_path") and args.tokenizer_path is None:
                args.tokenizer_path = args.model_path
        except Exception as e:
            print(f"ModelScope download failed: {e}")
            raise

server_args = ServerArgs.from_cli_args(args)
bench_args = BenchArgs.from_cli_args(args)

Expand Down