Bug: memory leak with vllm_grpo_trainer_modified.py

[zhangyuygss]

When training with vllm_grpo_trainer_modified.py, the memory (system memory, not cuda memory) keeps growing.
It leads to OOM in the middle of training (640G memory machine).
I tried to locate the leak with tracemalloc, it shows that transformers/models/qwen2_vl/image_processing_qwen2_vl.py:455 growing fast during training.
image_processing_qwen2_vl.py:455: pixel_values = np.array(pixel_values)
Seems the pixel_values was never relesed, I tried to namually del related variables like in the trainer, but did not work.
Any idea on this issue?

[chengxin0913]

same question, did you solve it?

[puppynull]

The same issue, but I tested it with images of the same size and found that the average memory consumption of this code remains unchanged, so there should be no memory leak. Have you resolved this issue?

[zhangyuygss]

Try this：vllm-project/vllm#16472 (comment)

same question, did you solve it?