[Frontend] Feature: support transcription API with language detection #13465
Conversation
Signed-off-by: albert <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
NickLucche
left a comment
NickLucche
left a comment
There was a problem hiding this comment.
Thanks for your contribution! :)
I think we should take the chance to do some bookkeeping and organize the code around constants a bit better, left some comments.
Also, we will need appropriate tests for this feature.
| if ( | ||
| "v3" in self.model_config.model.lower() | ||
| and self.model_config.hf_config.model_type.lower() == "whisper" | ||
| ): | ||
| id2token = LANG_ID_TO_LANG_TOKEN["whisper-v3"] | ||
| else: | ||
| return default_lang_token |
There was a problem hiding this comment.
we can't have an approximate check here, I'd rather build a mapping dynamically for all whisper models sharing the same content. See comment below for a cleaner alternative.
There was a problem hiding this comment.
Agree. It is better to wait for official supported language config and id-token mapping.
| "bo": "Tibetan", | ||
| } | ||
|
|
||
| LANG_ID_TO_LANG_TOKEN = { |
There was a problem hiding this comment.
I think we need some bookkeeping with this extra dict here @DarkLight1337 . How about we use the SupportsTranscription interface to query the model class for a mapping of supported languages?
Otherwise this will quickly become a mess if only another model gets added
There was a problem hiding this comment.
If we wanna support more asr models, need a better way to structure those mapping
There was a problem hiding this comment.
How about we use the
SupportsTranscriptioninterface to query the model class for a mapping of supported languages?
Sounds good to me!
| PromptType, | ||
| { | ||
| "encoder_prompt": { | ||
| "prompt": "", | ||
| "multi_modal_data": { | ||
| "audio": audio_data, | ||
| }, | ||
| }, | ||
| "decoder_prompt": "<|startoftranscript|>", | ||
| }, | ||
| ) |
There was a problem hiding this comment.
on second thought, it would be nice to avoid running the encoder twice. Ideally one could start the generation with the same decoder prompt as here, have the model predict a lang token, and then continue by forcing the subsequent tokens to be <|lang|><|transcribe|><|notimestamps|>{prompt}.
But I see how this would be invasive to implement.
We could re-use the encoder kv cache which is already populated here, but how do we make sure it's not evicted, would re-using the same request id here clash with scheduler policy? @DarkLight1337
There was a problem hiding this comment.
That's why It is a temporary fix. The current method does not take full adv of vllm features.
There was a problem hiding this comment.
I suggest not relying on the internals of KV cache for now as it may change significantly in V1.
There was a problem hiding this comment.
I assume this would target V0 though, encoder-decoder models are still far from being supported in V1 afaik #12761
There was a problem hiding this comment.
Oh, I forgot about that. I guess it's fine then.
We could re-use the encoder kv cache which is already populated here, but how do we make sure it's not evicted, would re-using the same request id here clash with scheduler policy?
@robertgshaw2-redhat @ywang96 might have a better idea regarding this.
Signed-off-by: albert <[email protected]>
Signed-off-by: albert <[email protected]>
mru4913
requested review from
robertgshaw2-redhat and
simon-mo
as code owners
Add a simple test case |
|
This pull request has merge conflicts that must be resolved before it can be |
|
Hello, I also needs this feature so that transcription works on multiple languages :) |
|
Hey! |
|
This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you! |
|
This pull request has been automatically closed due to inactivity. Please feel free to reopen if you intend to continue working on it. Thank you! |
5 participants
This is a follow-up to the Whisper transcription API endpoint, contributed by @DarkLight1337 and @NickLucche ([#12909], [#13301]).
The previous code restricted the use of audio with unknown languages, forcing it to be recognized as English. This update, adapted from the transformers library, introduces language detection. This is a temporary fix to support audio with unknown languages; a more robust and general solution will be implemented in the future.
Thanks to @DarkLight1337 @NickLucche and etc.