You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!--Copyright 2024 The HuggingFace Team. All rights reserved.
2
+
3
+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4
+
the License. You may obtain a copy of the License at
5
+
6
+
http://www.apache.org/licenses/LICENSE-2.0
7
+
8
+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9
+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10
+
specific language governing permissions and limitations under the License.
11
+
12
+
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
13
+
rendered properly in your Markdown viewer.
14
+
15
+
-->
16
+
17
+
# Mllama
18
+
19
+
## Overview
20
+
21
+
The Llama 3.2-Vision collection of multimodal large language models (LLMs) is a collection of pretrained and instruction-tuned image reasoning generative models in 11B and 90B sizes (text \+ images in / text out). The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.
22
+
23
+
**Model Architecture:** Llama 3.2-Vision is built on top of Llama 3.1 text-only model, which is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. To support image recognition tasks, the Llama 3.2-Vision model uses a separately trained vision adapter that integrates with the pre-trained Llama 3.1 language model. The adapter consists of a series of cross-attention layers that feed image encoder representations into the core LLM.
24
+
25
+
## Usage Tips
26
+
27
+
- For image+text and text inputs use `MllamaForConditionalGeneration`.
28
+
- For text-only inputs use `MllamaForCausalLM` for generation to avoid loading vision tower.
29
+
- Each sample can contain multiple images, and the number of images can vary between samples. The processor will pad the inputs to the maximum number of images across samples and to a maximum number of tiles within each image.
30
+
- The text passed to the processor should have the `"<|image|>"` tokens where the images should be inserted.
31
+
- The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as text to the processor.
32
+
33
+
## Usage Example
34
+
35
+
#### Instruct model
36
+
```python
37
+
import requests
38
+
import torch
39
+
fromPILimport Image
40
+
from transformers import MllamaForConditionalGeneration, AutoProcessor
0 commit comments