Add support for DeepseekAI's DeepseekVL

[geetu040]

What does this PR do?

Fixes #36110

This PR adds DeepseekAI's DeepseekVL model to Hugging Face Transformers.

DeepseekVL is an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. DeepSeek-VL possesses general multimodal understanding capabilities, capable of processing logical diagrams, web pages, formula recognition, scientific literature, natural images, and embodied intelligence in complex scenarios.

Relevant Links

• Research Paper: DeepSeek-VL: Towards Real-World Vision-Language Understanding
• Authors: Haoyu Lu, Wen Liu, Bo Zhang, et al.
• Implementation: github.com/deepseek-ai/DeepSeek-VL
• Models Weights: huggingface.co/collections/deepseek-ai/deepseek-vl

CC: @Benjamin-eecs, @RERV (github contributors of deepseek-ai/DeepSeek-VL)

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker, @Rocketknight1, @Cyrilvallez, @zucchini-nlp

TODOs

• Documentation
• Import Statements and Auto Modeling
• Modeling
  • VisionModel
  • TextModel
  • AlignerModel
• Tests
• Integration Tests
• Weight Conversion Script
• Model Cards
• Tokenizer/Processor
• Fix CI/CD tests

[geetu040]

@zucchini-nlp , @Rocketknight1, @Cyrilvallez

The Deepseek-VL uses Sam as backbone for encoding high-resolution images.
And to be more specific, the backbone is SamVisionEncoder instead of SamModel, which is not available as a public class. By which I mean that, you can do following with SamModel but not with SamVisionEncoder

from transformers import SamConfig, SamModel
config = SamConfig()
model = SamModel(config)

---

I think that we should rename SamVisionEncoder -> SamVisionModel, inherit it from SamPreTrainedModel and make it accessible to the user. I don't think it breaks backward compatibility in any way.

Otherwise, we would have to copy all the classes that build SamVisionEncoder for deepseek. There is nothing wrong with this either but having a SamVisionModel along with a SamModel makes sense, since it might benefit someone else as well.

If you think having a SamVisionModel makes sense, should that be done in a separate PR?

Btw, final results would look like this

from transformers import SamVisionConfig, SamVisionModel
config = SamVisionConfig()
model = SamVisionModel(config)

and SamVisionConfig is already available publically.

[zucchini-nlp]

@geetu040 we had similar situation with ideficsVision afair. Yes, in that case, we can just make it public and add in the docs. Renaming though would be breaking, imo we can leave name as is

[geetu040]

@zucchini-nlp is it okay to do it in the same PR? or should I create a new one

[zucchini-nlp]

@geetu040 imo a new PR will make it easier for us to iterate and review

[geetu040]

Hi @zucchini-nlp, I am working on the SamVisionEncoder (going to create the PR soon) and I have a quick question.
I realized that SamVisionAttention and SamVisionSdpaAttention produce attn_weights of different shapes when output_attentions=True.

Can you please answer these 2 questions:

1. Is this allowed in transformers for the 2 attentions to produce outputs of different shapes?
2. And lets suppose we do something that changes the shape of output_attentions, does that break backward compatibility?

[zucchini-nlp]

@geetu040 no, that is not expected to have different shapes. Usually using sdpa attention means that no attn_weights are returned, so it should be available only through 'eager' attention modules

I see that the weights are calculated on top of SDPA by manual matmul of key and query, which imo defeats the purpose of using SDPA in the first place. Can you remove the returned attention and raise warning similar to what is done in ViT?

[geetu040]

@zucchini-nlp sure I'll do that.

[zucchini-nlp]

Left a few tiny comments 😄

[geetu040]

@ArthurZucker @zucchini-nlp

Thanks for the reviews! I've addressed all the feedback, this should be ready for another look.

For checkpoints, see this: #36248-issuecomment

[ArthurZucker]

On it!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, deepseek_vl, deepseek_vl_hybrid, janus

[ArthurZucker]

Let's go thanks for the hardwork and good PR! 🚀