Build, compose, and doc updates for audio query by okhleif-10 · Pull Request #12 · mhbuehler/GenAIExamples

okhleif-10 · 2024-11-04T23:41:06Z

Description

Updated the compose file and README's for audio query

Issues

Linked with https://github.com/mhbuehler/GenAIComps/pull/8/files

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

N/A

Tests

N/A

Signed-off-by: Melanie Buehler <[email protected]>

* added entrypoint override Signed-off-by: okhleif-IL <[email protected]> * fixed entrypoint container Signed-off-by: okhleif-IL <[email protected]> * removed workdir Signed-off-by: okhleif-IL <[email protected]> * reverted set env vars changes Signed-off-by: okhleif-IL <[email protected]> * added entrypoint for embedding model Signed-off-by: okhleif-IL <[email protected]> * added llava model to readme Signed-off-by: okhleif-IL <[email protected]> * removed stray \n Signed-off-by: okhleif-IL <[email protected]> * added entrypoint override in bridgetower hpu Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <[email protected]>

* changed variable names Signed-off-by: okhleif-IL <[email protected]> * changed gaudi env vars Signed-off-by: okhleif-IL <[email protected]> * changed vars + docs Signed-off-by: okhleif-IL <[email protected]> * verified + updated readme Signed-off-by: okhleif-IL <[email protected]> * updated readme Signed-off-by: okhleif-IL <[email protected]> * updated tests Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

…estion Image ingestion improvements

…e/mm-rag-enhanced

…xamples into dina/audio-query

Signed-off-by: dmsuehir <[email protected]>

Update LVM model for Xeon

Signed-off-by: Melanie Buehler <[email protected]>

Signed-off-by: dmsuehir <[email protected]>

…e/mm-rag-enhanced

Signed-off-by: dmsuehir <[email protected]>

Signed-off-by: Melanie Buehler <[email protected]>

Image+Text Upload

Signed-off-by: Melanie Buehler <[email protected]>

UI Audio Upload

Signed-off-by: Melanie Buehler <[email protected]>

Signed-off-by: okhleif-IL <[email protected]>

Signed-off-by: dmsuehir <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dmsuehir <[email protected]>

Test update: Increase sleep to 3m to wait for LVM microservice to be ready

…xamples into dina/audio-query

dmsuehir

Could you sync your branch? I see that LVM_MODEL_ID fix for xeon but I thought we fixed that already

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md

…ch-audio

Signed-off-by: okhleif-IL <[email protected]>

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md

dmsuehir · 2024-11-07T19:38:47Z

Could you also update the test files: https://github.com/mhbuehler/GenAIExamples/tree/melanie/mm-rag-enhanced/MultimodalQnA/tests
There's a list of which comps to build which will need to be updated to include asr, etc. We can add a test. We should probably also look into updating the mermaid diagram in the README, but we could do that in a separate PR

Signed-off-by: okhleif-IL <[email protected]>

MultimodalQnA/tests/test_compose_on_xeon.sh

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md

MultimodalQnA/docker_compose/intel/hpu/gaudi/README.md

Signed-off-by: okhleif-IL <[email protected]>

…udio

Signed-off-by: okhleif-IL <[email protected]>

mhbuehler

One suggested change. Everything else LGTM.

MultimodalQnA/tests/test_compose_on_xeon.sh

Signed-off-by: okhleif-IL <[email protected]>

dmsuehir · 2024-11-22T23:38:47Z

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md

@@ -357,6 +377,12 @@ curl http://${host_ip}:8888/v1/multimodalqna \
    -d '{"messages": "What is the revenue of Nike in 2023?"}'
 ```



Should we also have something above in the "Validate Microservices" section to test ASR?

dmsuehir · 2024-11-22T23:40:50Z

MultimodalQnA/tests/test_compose_on_xeon.sh


 function build_docker_images() {
    cd $WORKPATH/docker_image_build
+    # TODO: Revert to the clone of opea-project "main" after the merge of https://github.com/opea-project/GenAIComps/pull/852


This comment can be removed?

…udio

Signed-off-by: okhleif-IL <[email protected]>

dmsuehir

LGTM

* Add var for LVM model Signed-off-by: dmsuehir <[email protected]> * README updates and tests Signed-off-by: dmsuehir <[email protected]> * Build, compose, and doc updates for audio query (#12) Signed-off-by: okhleif-IL <[email protected]> * validated, updated tests Signed-off-by: okhleif-IL <[email protected]> * added one more curl test for audio Signed-off-by: okhleif-IL <[email protected]> * fixed typo Signed-off-by: okhleif-IL <[email protected]> * reverted git clone command Signed-off-by: okhleif-IL <[email protected]> * added ASR test Signed-off-by: okhleif-IL <[email protected]> * fixed command with backslashes Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]> Signed-off-by: okhleif-IL <[email protected]> Signed-off-by: dmsuehir <[email protected]> * MMQnA doc update correcting ASR and whisper image names (#24) * MMQnA doc update correcting ASR and whisper image names Signed-off-by: dmsuehir <[email protected]> * Add image tags Signed-off-by: dmsuehir <[email protected]> --------- Signed-off-by: dmsuehir <[email protected]> * Integrate audio query into UI (#22) * Enabled audio query functionality in the MultimodalQnA UI Signed-off-by: Melanie Buehler <[email protected]> * Temporarily redirect clones for tests Signed-off-by: Melanie Buehler <[email protected]> * Doc update and set MAX_IMAGES to default to 1 in compose.yaml Signed-off-by: dmsuehir <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * README update to add a note about MAX_IMAGES Signed-off-by: dmsuehir <[email protected]> * Add services to tests and correct small text error (#26) * Add services to tests and correct small text error Signed-off-by: Melanie Buehler <[email protected]> * Revert unintended changes Signed-off-by: Melanie Buehler <[email protected]> --------- Signed-off-by: Melanie Buehler <[email protected]> * Fixed build.yaml inconsistency Signed-off-by: Melanie Buehler <[email protected]> * Moving note about MAX_IMAGES due to review feedback: Signed-off-by: dmsuehir <[email protected]> * Update repo clones for E2E tests Signed-off-by: Melanie Buehler <[email protected]> * Moved Audio Query Gateway changes to multimodalqna.py (#29) * Moved gateway changes to multimodalqna.py Signed-off-by: okhleif-IL <[email protected]> * reverted port changes Signed-off-by: okhleif-IL <[email protected]> * addressed review comments Signed-off-by: okhleif-IL <[email protected]> * reverted print statement Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]> * Removed proxies arg from translation request (#31) * Moved gateway changes to multimodalqna.py Signed-off-by: okhleif-IL <[email protected]> * reverted port changes Signed-off-by: okhleif-IL <[email protected]> * addressed review comments Signed-off-by: okhleif-IL <[email protected]> * reverted print statement Signed-off-by: okhleif-IL <[email protected]> * removed proxies Signed-off-by: okhleif-IL <[email protected]> --------- Signed-off-by: okhleif-IL <[email protected]> * Add MAX_IMAGES to docs Signed-off-by: dmsuehir <[email protected]> * Moved gateway updates for image query to GenAIExamples Signed-off-by: dmsuehir <[email protected]> * Fix initial input format for TextImageDoc Signed-off-by: dmsuehir <[email protected]> * Fix merge conflict Signed-off-by: dmsuehir <[email protected]> * Fix merge conflict Signed-off-by: dmsuehir <[email protected]> * Add Gaudi tests Signed-off-by: dmsuehir <[email protected]> * Temporary update to use dev branch for test Signed-off-by: dmsuehir <[email protected]> * Temporary change for dockerfile to use dev fork/branch for GenAIComps Signed-off-by: dmsuehir <[email protected]> --------- Signed-off-by: dmsuehir <[email protected]> Signed-off-by: Melanie Buehler <[email protected]> Signed-off-by: okhleif-IL <[email protected]> Co-authored-by: Omar Khleif <[email protected]> Co-authored-by: Melanie Hart Buehler <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <[email protected]>

mhbuehler and others added 30 commits October 14, 2024 16:28

Initial implementation of image ingestion

97f4bc0

Signed-off-by: Melanie Buehler <[email protected]>

Use same endpoint for image and video ingestion

56bd8c3

Signed-off-by: Melanie Buehler <[email protected]>

Update tests and docs

b334dc2

Signed-off-by: Melanie Buehler <[email protected]>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

13c752a

…e/mm-rag-enhanced

Updated env vars and compose.yaml for ASR

97d7789

Signed-off-by: dmsuehir <[email protected]>

added LVM_MODEL_ID var to test file (#4)

271117e

Signed-off-by: okhleif-IL <[email protected]>

Updates tests per feedback

c4e0259

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #2 from mhbuehler/melanie/combined_image_video_ing…

a541140

…estion Image ingestion improvements

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

30c311d

…e/mm-rag-enhanced

Merge branch 'melanie/mm-rag-enhanced' of github.com:mhbuehler/GenAIE…

caa08df

…xamples into dina/audio-query

Update LVM model for Xeon

e83fc44

Signed-off-by: dmsuehir <[email protected]>

Merge pull request #5 from mhbuehler/dina/lvm_model

4b8a5ad

Update LVM model for Xeon

Initial setup for ingest_with_text

a6d826c

Signed-off-by: Melanie Buehler <[email protected]>

Write and send custom caption file

547a139

Signed-off-by: Melanie Buehler <[email protected]>

Update docs and tests

7cfc343

Signed-off-by: Melanie Buehler <[email protected]>

MMQnA doc updates for audio ingestion (#7)

69dbdfc

Signed-off-by: dmsuehir <[email protected]>

Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…

834c668

…e/mm-rag-enhanced

Fix UI request for follow up queries with no image (#8)

a8f8dc9

Signed-off-by: dmsuehir <[email protected]>

Updated for review suggestions

718d02e

Signed-off-by: Melanie Buehler <[email protected]>

Merge branch 'melanie/mm-rag-enhanced' into melanie/images_and_text

d05cfb3

Add audio upload functionality to UI

72591c1

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #6 from mhbuehler/melanie/images_and_text

431e41b

Image+Text Upload

Merge branch 'melanie/mm-rag-enhanced' into melanie/audio_ingest_ui

d535aa7

Minor refactor, improve display text, and suppress PDF tab

39f43fc

Signed-off-by: Melanie Buehler <[email protected]>

Merge pull request #9 from mhbuehler/melanie/audio_ingest_ui

afc3c8a

UI Audio Upload

Small fixes

ddd5dfb

Signed-off-by: Melanie Buehler <[email protected]>

Improve appearance

426c739

Signed-off-by: Melanie Buehler <[email protected]>

added asr endpoints

641b484

Signed-off-by: okhleif-IL <[email protected]>

dmsuehir and others added 7 commits November 6, 2024 16:00

Update MMQnA xeon test to wait for LVM to be ready (#18)

6631601

Signed-off-by: dmsuehir <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

59acc77

for more information, see https://pre-commit.ci

Test update: Increase wait time and add more messages (#19)

9d4cb5f

Signed-off-by: dmsuehir <[email protected]>

Test update: Increase sleep

c1bfe6a

Signed-off-by: dmsuehir <[email protected]>

Merge pull request #20 from mhbuehler/dina/test-update-wait

b88aa3f

Test update: Increase sleep to 3m to wait for LVM microservice to be ready

Merge branch 'main' into melanie/mm-rag-enhanced

6e5a200

Merge branch 'melanie/mm-rag-enhanced' of github.com:mhbuehler/GenAIE…

77ee6b3

…xamples into dina/audio-query

dmsuehir requested changes Nov 7, 2024

View reviewed changes

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md Outdated Show resolved Hide resolved

okhleif-10 added 2 commits November 7, 2024 11:16

Merge remote-tracking branch 'origin/dina/audio-query' into omar/spee…

e629cd2

…ch-audio

formatted curl command

c225371

Signed-off-by: okhleif-IL <[email protected]>

dmsuehir reviewed Nov 7, 2024

View reviewed changes

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md Outdated Show resolved Hide resolved

updated build and env vars

9d236ad

Signed-off-by: okhleif-IL <[email protected]>

okhleif-10 changed the base branch from melanie/mm-rag-enhanced to mmqna-phase2 November 18, 2024 17:46

okhleif-10 added 2 commits November 19, 2024 14:04

merging mmqna-phase2

2e3ba80

Signed-off-by: okhleif-IL <[email protected]>

validated, updated tests

4aa35d9

Signed-off-by: okhleif-IL <[email protected]>

mhbuehler requested changes Nov 20, 2024

View reviewed changes

MultimodalQnA/tests/test_compose_on_xeon.sh Outdated Show resolved Hide resolved

MultimodalQnA/docker_compose/intel/cpu/xeon/README.md Outdated Show resolved Hide resolved

MultimodalQnA/docker_compose/intel/hpu/gaudi/README.md Outdated Show resolved Hide resolved

okhleif-10 added 3 commits November 22, 2024 13:42

added one more curl test for audio

d5e8d7a

Signed-off-by: okhleif-IL <[email protected]>

Merge remote-tracking branch 'origin/mmqna-phase2' into omar/speech-a…

fb89859

…udio

fixed typo

1be60be

Signed-off-by: okhleif-IL <[email protected]>

mhbuehler requested changes Nov 22, 2024

View reviewed changes

MultimodalQnA/tests/test_compose_on_xeon.sh Outdated Show resolved Hide resolved

reverted git clone command

b2dcc3c

Signed-off-by: okhleif-IL <[email protected]>

mhbuehler approved these changes Nov 22, 2024

View reviewed changes

dmsuehir requested changes Nov 22, 2024

View reviewed changes

okhleif-10 added 3 commits December 2, 2024 10:54

Merge remote-tracking branch 'origin/mmqna-phase2' into omar/speech-a…

222c6c4

…udio

added ASR test

f1772dc

Signed-off-by: okhleif-IL <[email protected]>

fixed command with backslashes

229777b

Signed-off-by: okhleif-IL <[email protected]>

dmsuehir approved these changes Dec 2, 2024

View reviewed changes

okhleif-10 merged commit 3d22cf5 into mmqna-phase2 Dec 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build, compose, and doc updates for audio query#12

Build, compose, and doc updates for audio query#12
okhleif-10 merged 76 commits intommqna-phase2from
omar/speech-audio

okhleif-10 commented Nov 4, 2024 •

edited

Loading

Uh oh!

dmsuehir left a comment

Uh oh!

Uh oh!

Uh oh!

dmsuehir commented Nov 7, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhbuehler left a comment

Uh oh!

Uh oh!

dmsuehir Nov 22, 2024

Uh oh!

dmsuehir Nov 22, 2024

Uh oh!

dmsuehir left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -357,6 +377,12 @@ curl http://${host_ip}:8888/v1/multimodalqna \
		-d '{"messages": "What is the revenue of Nike in 2023?"}'
		```

Conversation

okhleif-10 commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

dmsuehir left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dmsuehir commented Nov 7, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mhbuehler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dmsuehir Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

dmsuehir Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

dmsuehir left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

okhleif-10 commented Nov 4, 2024 •

edited

Loading