[WIP] Support glm4.1v

[JustinTong0323]

Motivation

Close #7993

Modifications

This pull request introduces support for the GLM-4V model within the sglang framework. It includes the necessary model files, configuration adjustments, and a new conversation template to facilitate interaction with the GLM-4V model. Additionally, it addresses potential issues with MRoPE section mismatches and enhances the flexibility of the get_rope_index function to accommodate various model configurations.

• Model Support: Adds support for the GLM-4V model, including necessary configurations and model files.
• Conversation Template: Registers a new conversation template specifically for GLM-4V, defining system messages, roles, and separators.
• MRoPE Section Adjustment: Implements auto-correction for MRoPE section mismatches in rotary embeddings, ensuring compatibility with different model configurations.
• Vision Start Token ID: Modifies the get_rope_index function to accept a list of vision start token IDs, accommodating models with multiple vision start tokens.
• Conditional Logic: Adds conditional logic to handle specific model types like 'glm4v' in the get_rope_index function.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[gemini-code-assist]

Summary of Changes

Hello @JustinTong0323, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for the GLM-4V model within the sglang framework. It includes the necessary model files, configuration adjustments, and a new conversation template to facilitate interaction with the GLM-4V model. Additionally, it addresses potential issues with MRoPE section mismatches and enhances the flexibility of the get_rope_index function to accommodate various model configurations.

Highlights

• Model Support: Adds support for the GLM-4V model, including necessary configurations and model files.
• Conversation Template: Registers a new conversation template specifically for GLM-4V, defining system messages, roles, and separators.
• MRoPE Section Adjustment: Implements auto-correction for MRoPE section mismatches in rotary embeddings, ensuring compatibility with different model configurations.
• Vision Start Token ID: Modifies the get_rope_index function to accept a list of vision start token IDs, accommodating models with multiple vision start tokens.
• Conditional Logic: Adds conditional logic to handle specific model types like 'glm4v' in the get_rope_index function.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[JustinTong0323]

CI not pass, caused by transformers error:

  File "/usr/local/lib/python3.10/dist-packages/transformers/image_processing_utils_fast.py", line 614, in __call__
    return self.preprocess(images, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/glm4v/image_processing_glm4v_fast.py", line 310, in preprocess
    patches, image_grid_thw = self._preprocess(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/glm4v/image_processing_glm4v_fast.py", line 176, in _preprocess
    grouped_images, grouped_images_index = group_images_by_shape(images)
TypeError: group_images_by_shape() missing 1 required positional argument: 'disable_grouping'

Solved in upstream:

• Fix some bug for finetune and batch infer For GLM-4.1V huggingface/transformers#39090

[JustinTong0323]

Update transformers version first:

• update transformers to 4.53.2 #8029

[zRzRzRzRzRzRzR]

Thank you for your submission and for your efforts integrating GLM-4.1V into the sglang.

Regarding the issue with transformers, it can be resolved by updating to version 4.53.2.

we appreciate your contribution and are currently reviewing the PR. Thanks again!

[mickqian]

nit: we should have some auto mechanism for detecting potential vlm arch, e.g. if it has a vision_config or vision_token

[mickqian]

Or more straight forward, add an attribute to the Model Class

[mickqian]

should we move this to rotary_embedding?

[mickqian]

nit: no pad_input_ids in this model?

[JustinTong0323]

inherit from qwenvl

[lambertwjh]

I found a blocking bug regarding multimodal video input with the OpenAI /v1/chat/completions API:
Even when do_sample_frames: false (and all related param names) are set both at the root and in video_url, the server always returns this error for video input:
{"object":"error","message":"Frame sampling is enabled but no video metadata was found. Please pass in VideoMetadata object per each input video or set do_sample_frames=False","type":"BadRequestError"}
All keys (do_sample_frames, sample_frames, frame_sampling, etc.) are boolean and included, together with a minimal video_metadata dict, and verified in the server-side debug log.
The server still ignores these parameters and refuses to disable frame sampling for video, making it impossible to use video input via OpenAI API.
For text/image input, everything works fine.
Please check if the backend is reading and respecting these parameters for video input in this PR.

[zRzRzRzRzRzRzR]

do_sample_frames = True is the mandatory requirement we set for transformers processors, as our model must sample according to the specified method to function.

[marsh312]

Thank you for your great contribution! However, I noticed that the template for GLM-4V differs significantly from that of Qwen2-VL, including differences in special tokens. You can refer to the following links for details:

https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking/blob/main/chat_template.jinja

https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking/blob/main/config.json

Given this, should the code for registering the template be modified accordingly?
If there's any issue with my suggestion, please feel free to disregard it.

[JustinTong0323]

Processor logic is from https://github.com/vllm-project/vllm/blob/32142b3c62277ac7cb941f2036270decb6b514f4/vllm/model_executor/models/glm4_1v.py#L1082. But it doesn't work well.

[JustinTong0323]

The tests cannot pass, I don't know why the CI has passed.... (PR Test / unit-test-backend-1-gpu (5) (pull_request))

[JustinTong0323]

do_sample_frames = True is the mandatory requirement we set for transformers processors, as our model must sample according to the specified method to function.

Hi @zRzRzRzRzRzRzR, thanks for reviewing. I've solved this issue, but I've encountered some more strange bugs. Could I get in touch with you via Slack, WeChat, or any other contact method to ask for your help? I appreciate your help in advance!

[zRzRzRzRzRzRzR]

We have already communicated on WeChat. Once your image understanding is working properly, we can try to start modifying the video part.

[JustinTong0323]

move to #8798

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!