[Bug] VLM cuda memory leak incur OOM

[WingEdge777]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

When serving a vlm (qwen2.5-vl-3b/7b/32b in my cases). The GPU memory keeps increasing until it gets cuda out of memory for the coming request.

Reproduction

Start server:

python -m sglang.launch_server --model-path Qwen2.5-VL-3B-Instruct --tp 1 --nnodes 1 --node-rank 0 --trust-remote-code --host 0.0.0.0 --port 30000

client code:

import openai


client = openai.Client(api_key="EMPTY", base_url="http://127.0.0.1:30000/v1")
model_name = client.models.list().data[0].id

sampling_params = {"max_tokens": 512, "temperature": 0}

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "please describe this image"},
            {"type": "image_url", "image_url": {"url": url}},
        ]
    }
]

for idx in range(100):
    print(f"{idx} req")
    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
        **sampling_params,
    )
    # time.sleep(0.5)

print(response)

Every 2~5 requests cause a memory occupation bump of 20 MB.

Environment

Python: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3: NVIDIA A10
GPU 0,1,2,3 Compute Capability: 8.6
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.6, V12.6.68
CUDA Driver Version: 470.161.03
PyTorch: 2.7.1+cu126
sglang: 0.4.9.post6
sgl_kernel: 0.2.7
flashinfer_python: 0.2.9rc2
triton: 3.3.1
transformers: 4.53.0
torchao: 0.9.0+cu126
numpy: 2.2.6
aiohttp: 3.12.14
fastapi: 0.116.1
hf_transfer: 0.1.9
huggingface_hub: 0.34.3
interegular: 0.3.3
modelscope: 1.28.0
orjson: 3.10.18
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.0
uvicorn: 0.35.0
uvloop: 0.21.0
vllm: 0.9.0.1
xgrammar: 0.1.21
openai: 1.95.1
tiktoken: 0.9.0
anthropic: 0.57.1
litellm: 1.74.2
decord: 0.6.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 mlx5_0 CPU Affinity NUMA Affinity
GPU0 X PHB SYS SYS SYS 0-117 0
GPU1 PHB X SYS SYS SYS 0-117 0
GPU2 SYS SYS X PHB SYS 118-235 1
GPU3 SYS SYS PHB X SYS 118-235 1
mlx5_0 SYS SYS SYS SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

Hypervisor vendor: KVM

[WingEdge777]

By adding toch.cuda.memory_summary() call at http entry points. We can observe things as follows:

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      |  82321 KiB |  94541 KiB | 442490 KiB | 360169 KiB |
|       from large pool |  82320 KiB |  92610 KiB | 411600 KiB | 329280 KiB |
|       from small pool |      1 KiB |   1932 KiB |  30890 KiB |  30889 KiB |
|---------------------------------------------------------------------------|
| Active memory         |  82321 KiB |  94541 KiB | 442490 KiB | 360169 KiB |
|       from large pool |  82320 KiB |  92610 KiB | 411600 KiB | 329280 KiB |
|       from small pool |      1 KiB |   1932 KiB |  30890 KiB |  30889 KiB |
|---------------------------------------------------------------------------|
| Requested memory      |  82320 KiB |  94539 KiB | 442470 KiB | 360150 KiB |
|       from large pool |  82320 KiB |  92610 KiB | 411600 KiB | 329280 KiB |
|       from small pool |      0 KiB |   1929 KiB |  30870 KiB |  30870 KiB |
|---------------------------------------------------------------------------|
| GPU reserved memory   | 124928 KiB | 124928 KiB | 124928 KiB |      0 B   |
|       from large pool | 122880 KiB | 122880 KiB | 122880 KiB |      0 B   |
|       from small pool |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
|---------------------------------------------------------------------------|
| Non-releasable memory |  42607 KiB |  47752 KiB | 517296 KiB | 474689 KiB |
|       from large pool |  40560 KiB |  45705 KiB | 485002 KiB | 444442 KiB |
|       from small pool |   2047 KiB |   2047 KiB |  32293 KiB |  30246 KiB |
|---------------------------------------------------------------------------|
| Allocations           |      18    |      24    |     164    |     146    |
|       from large pool |      16    |      19    |     112    |      96    |
|       from small pool |       2    |       6    |      52    |      50    |
|---------------------------------------------------------------------------|
| Active allocs         |      18    |      24    |     164    |     146    |
|       from large pool |      16    |      19    |     112    |      96    |
|       from small pool |       2    |       6    |      52    |      50    |
|---------------------------------------------------------------------------|
| GPU reserved segments |       7    |       7    |       7    |       0    |
|       from large pool |       6    |       6    |       6    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |       8    |       9    |      54    |      46    |
|       from large pool |       6    |       7    |      35    |      29    |
|       from small pool |       2    |       3    |      19    |      17    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

After sent some requests

|===========================================================================|
|                  PyTorch CUDA memory summary, device ID 0                 |
|---------------------------------------------------------------------------|
|            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
|===========================================================================|
|        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
|---------------------------------------------------------------------------|
| Allocated memory      | 252106 KiB | 264326 KiB |   1323 MiB |   1077 MiB |
|       from large pool | 252105 KiB | 262395 KiB |   1230 MiB |    984 MiB |
|       from small pool |      1 KiB |   1932 KiB |     92 MiB |     92 MiB |
|---------------------------------------------------------------------------|
| Active memory         | 252106 KiB | 264326 KiB |   1323 MiB |   1077 MiB |
|       from large pool | 252105 KiB | 262395 KiB |   1230 MiB |    984 MiB |
|       from small pool |      1 KiB |   1932 KiB |     92 MiB |     92 MiB |
|---------------------------------------------------------------------------|
| Requested memory      | 252105 KiB | 264324 KiB |   1323 MiB |   1077 MiB |
|       from large pool | 252105 KiB | 262395 KiB |   1230 MiB |    984 MiB |
|       from small pool |      0 KiB |   1929 KiB |     92 MiB |     92 MiB |
|---------------------------------------------------------------------------|
| GPU reserved memory   | 350208 KiB | 350208 KiB | 350208 KiB |      0 B   |
|       from large pool | 348160 KiB | 348160 KiB | 348160 KiB |      0 B   |
|       from small pool |   2048 KiB |   2048 KiB |   2048 KiB |      0 B   |
|---------------------------------------------------------------------------|
| Non-releasable memory |  98102 KiB | 103247 KiB |   1504 MiB |   1408 MiB |
|       from large pool |  96055 KiB | 101200 KiB |   1411 MiB |   1317 MiB |
|       from small pool |   2047 KiB |   2047 KiB |     93 MiB |     91 MiB |
|---------------------------------------------------------------------------|
| Allocations           |      51    |      57    |     494    |     443    |
|       from large pool |      49    |      52    |     343    |     294    |
|       from small pool |       2    |       6    |     151    |     149    |
|---------------------------------------------------------------------------|
| Active allocs         |      51    |      57    |     494    |     443    |
|       from large pool |      49    |      52    |     343    |     294    |
|       from small pool |       2    |       6    |     151    |     149    |
|---------------------------------------------------------------------------|
| GPU reserved segments |      18    |      18    |      18    |       0    |
|       from large pool |      17    |      17    |      17    |       0    |
|       from small pool |       1    |       1    |       1    |       0    |
|---------------------------------------------------------------------------|
| Non-releasable allocs |      19    |      20    |     131    |     112    |
|       from large pool |      17    |      18    |      79    |      62    |
|       from small pool |       2    |       3    |      52    |      50    |
|---------------------------------------------------------------------------|
| Oversize allocations  |       0    |       0    |       0    |       0    |
|---------------------------------------------------------------------------|
| Oversize GPU segments |       0    |       0    |       0    |       0    |
|===========================================================================|

The reserved and non-releasable memory keeps increasing. I used to believe PyTorch should reuse memory. But its not, and I don't know why.

This issue is preventing us from using SGLang in production environments. Without reliability, with nothing. We think this issue should be put top of the to-do list.

BTY, thank you all for your contributions to OSS community.

[Swipe4057]

In our company (around 30,000 people), we also use SGLang on our servers. Unfortunately, this issue with multimodal input causing OOM (Out of Memory) errors has been known for a long time and hasn’t been resolved for several months now. I also believe the community should allocate more attention to this problem, as it is critical for usability—effectively blocking the functionality of multimodal models for users. Let’s create a plan: what hypotheses we need to investigate, which code sections to examine, and what tests to run in order to isolate the bug!

[WingEdge777]

cc @merrymercy @zhyncs . Could you please pay more attention to this? This issue causes significant harm to sglang, since it makes sglang not capable of being used in multimodal scenarios.

[hnyls2002]

For anyone who meets this problem, try this PR #8306. Still in the long term, we have to find out which memory pieces are reserved @JustinTong0323

[JustinTong0323]

For anyone who meets this problem, try this PR #8306. Still in the long term, we have to find out which memory pieces are reserved @JustinTong0323

I tried this, but not work for me. torch.cuda.empty_cache() seems not able to release the extra memory occupied by ViT.

[yannqi]

Same issue.

[jinleic]

Same issue when serving GLM 4.5V

SGLANG_VLM_CACHE_SIZE_MB=8192 CUDA_VISIBLE_DEVICES=4,5,6,7 python3 -m sglang.launch_server --model-path zai-org/GLM-4.5V --tp-size 4 --tool-call-parser glm45 --reasoning-parser glm45 --served-model-name glm-4.5v --port 30456 --host 0.0.0.0 --attention-backend fa3 --mm-attention-backend fa3 --enable-torch-compile --mem-fraction-static 0.8

[JustinTong0323]

@handoku I apologize for the misleading information provided above. I conducted tests with your environment settings, and I believe that the root cause of your OOM issue has been addressed in this PR: #8431

[JustinTong0323]

However, there are other cases that can lead to VLM OOM. We are continuously monitoring and testing. Thank you for reporting the issue to us.

[WingEdge777]

@handoku I apologize for the misleading information provided above. I conducted tests with your environment settings, and I believe that the root cause of your OOM issue has been addressed in this PR: #8431

That pr indeed eases the OOM pain a lot (every 2~5 requests causing a memory bump to every 40~50 requests). And you are right about there being other causes that lead to OOM, which is why I also try my best to make the Team pay more attention to this. In our pressure test, CUDA OOM occurs eventually, even using the latest version of sglang. Neither #8431 nor #7340 fixes this issue. You can see others still encountered the same problem as I.

On the contrary, I hate to say it, but vllm demonstrates remarkable robustness(the CUDA memory occupation remains completely stable, does not even fluctuate at all) and better performance according to our test results. Thus, we still have to maintain two LLM frameworks in the our production environment.

[hnyls2002]

I conducted an experiment based on the reproduction code in this issue (simply sending the same request multiple times), and the result was not satisfactory.

Image Input	Image Processor	Num Reqs	Flush Period	Init Memory	Peak Memory
Yes	CUDA	30K	1000 reqs	67858.88 MB	79056.44 MB
Yes	CPU	30K	1000 reqs	67860.88 MB	78576.88 MB
Yes	CUDA	13K	10 reqs	67858.88 MB	73720.44 MB
No	-	60K	1000 reqs	67858.88 MB	72032.88 MB

There are probably multiple unknown memory leaks, whether using image input or not...

@JustinTong0323 @mickqian You can reproduce the result https://paste.ubuntu.com/p/WzFYcSxvCx/, which shows the peak memory usage increases with time.