[Feature] Improve embedding merge implementation #23891
Description
🚀 The feature, motivation and pitch
Currently merging text embeddings and multimodal embedding is done by checking input_ids in the current batch and scatter multimodal embeddings into where the placeholder ids are. See more details here.
An alternative solution here that should be investigated is to gather this information from mm_positions of scheduled requests to form a batch-level mask (similar to grammar_bitmask)
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.