[Bugfix] Missing thumbnail from NVLM-D processor

[ameyanjarlekar]

If num_patches evaluate to 1 for the input image prompt, the existing code does not generate any image token leading to the error "Unable to allocate 256 multimodal tokens to 0 placeholders". For images of particular dimensions eg. 220x229 or 200x229, I received this error.

The code currently breaks the input image prompt into multiple images by cropping certain parts of the image. It then appends all these cropped images plus a thumbnail image which is the resized version of the original image into a list. The number of elements in this list is dependent on the num_patches variable which is calculated based on the size of the original image.

If the num_patches variable is 1, then the code currently does not crop the image and also doesn't add the thumbnail image, therefore adding 0 image placeholders. Therefore, to consider the thumbnail image, which is the original image when num_patches==1 we need to remove the condition num_patches != 1.

To reproduce the bug you can try giving a prompt with this image (https://github.com/user-attachments/assets/7cd2df26-c765-43d8-bab2-9783d33c792f).

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks for fixing! @kylehh can you confirm this fix since I don't have the means to run this model locally right now.

[kylehh]

I can reprod the issue w the shared image and the inference can run through w the bug fix. ( I didn't go through the whole logic of how num_patches works though )

[DarkLight1337]

Merged