[New Model]Donut model

[princepride]

Purpose

FIX #18850.
This PR adds support for Donut-like models, resolving issue #18850.

It implements the Donut model and the structurally similar Dolphin model. Since the Donut model uses a Swin Transformer as its vision backbone, this PR also includes the implementation for the Swin model.

Test Plan

Donut Model Test

Script

python examples/offline_inference/encoder_decoder_multimodal.py -m donut

Result

Dolphin Model Tests

Script

python examples/offline_inference/dolphin.py

Result

[gemini-code-assist]

Code Review

This pull request introduces support for Donut-like models, including Donut, Dolphin, and the Swin Transformer backbone. The changes are extensive, adding new model implementations and example inference scripts. My review focuses on correctness and potential bugs. I've identified a critical bug in the Donut model implementation that would cause a TypeError during image validation, and a high-severity issue in the Dolphin example script that could lead to a ZeroDivisionError. I've provided code suggestions to fix both issues.

[princepride]

@DarkLight1337 @Isotr0py Could you review it? I messed up the previous PR (#23187) during a rebase, so I've created a new one. Thank you.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[Isotr0py]

Let's add examples in examples/offline_inference/encoder_decoder_multimodal.py and examples/offline_inference/vision_language.py instead of adding new files in examples/offline_inference/.

[princepride]

I was looking at the images in the S3 bucket and it seems there aren't any suitable for OCR tasks. This is particularly true for the Dolphin model, whose OCR task is similar to executing a workflow. Different prompts will determine whether to segment or parse the document. At the same time, depending on the parsing tags, it will decide whether to parse text or icons. That's why I've added two example files.

[princepride]

I used fetch_image to load the image and moved donut to encoder_decoder_multimodal.py

[princepride]

I also removed the --task in dolphin

[hmellor]

It looks like the independent dolphin example still got merged. We don't really want to have model specific examples as that will clutter the examples and make it harder for new users to find what they need

[princepride]

@Isotr0py @DarkLight1337 I adjust the example code and update the task plan, please review it. thank you.

[DarkLight1337]

Thanks!