Adds audio querying to MultimodalQ&A gateway#974
Closed
mhbuehler wants to merge 15 commits intoopea-project:mainfrom
Closed
Adds audio querying to MultimodalQ&A gateway#974mhbuehler wants to merge 15 commits intoopea-project:mainfrom
mhbuehler wants to merge 15 commits intoopea-project:mainfrom
Conversation
Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added in audio dict creation Signed-off-by: okhleif-IL <omar.khleif@intel.com> * separated audio from prompt Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added ASR endpoint Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed ASR endpoints from mm embedding Signed-off-by: okhleif-IL <omar.khleif@intel.com> * edited return logic, fixed function call Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added megaservice to elif Signed-off-by: okhleif-IL <omar.khleif@intel.com> * reworked helper func Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Append audio to prompt Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Reworked handle messages, added metadata Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Moved dictionary logic to right place Signed-off-by: okhleif-IL <omar.khleif@intel.com> * changed logic to rely on message len Signed-off-by: okhleif-IL <omar.khleif@intel.com> * list --> empty str Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: okhleif-IL <omar.khleif@intel.com> Signed-off-by: dmsuehir <dina.s.jones@intel.com>
for more information, see https://pre-commit.ci
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
1 task
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Fixed role bug where enumeration was wrong
for more information, see https://pre-commit.ci
Codecov ReportAttention: Patch coverage is
|
ashahba
reviewed
Dec 5, 2024
Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Adds unit test coverage for audio query
for more information, see https://pre-commit.ci
Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Fix port number placement
mkbhanda
approved these changes
Dec 6, 2024
| return prompt | ||
|
|
||
| def convert_audio_to_text(self, audio): | ||
| # translate audio to text by passing in dictionary to ASR |
Collaborator
There was a problem hiding this comment.
comment quirky! dictionary is a data type here but can get mixed with the English word dictionary (word meanings)
| else: | ||
| input_dict = {"byte_str": audio[0]} | ||
|
|
||
| response = requests.post(self.asr_endpoint, data=json.dumps(input_dict), proxies={"http": None}) |
Collaborator
There was a problem hiding this comment.
should proxies be read from some environment variable for a more general solution?
Contributor
There was a problem hiding this comment.
Why this is setting proxies in the first place, shouldn't those be set well before this point?
eero-t
reviewed
Dec 9, 2024
| import requests | ||
| from fastapi import Request | ||
|
|
||
| os.environ["ASR_SERVICE_PORT"] = "8086" |
Contributor
There was a problem hiding this comment.
Why this overrides environment, instead of taking the value from environment?
4 tasks
Collaborator
|
Labeling this as |
Collaborator
|
Looking at new changes in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds ASR endpoint, speech audio processing, prompt construction, and return of decoded audio in response metadata. This goes with GenAIExamples PR: opea-project/GenAIExamples#1225.
Issues
Part of the MultimodalQnA Audio & Image Enhancements RFC
Type of change
Dependencies
N/A
Tests
Automated tests were added to GenAIExamples.