MultimodalQnA image query, pdf, and dynamic ports by mhbuehler · Pull Request #1134 · opea-project/GenAIComps

mhbuehler · 2025-01-10T22:14:00Z

Description

According to the RFC's Phase 2 plan, this PR adds image query support, PDF ingestion support, and dynamic ports to the microservices used by MultimodalQnA. This PR goes with this one in GenAIExamples.

Issues

RFC

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Others (enhancement, documentation, validation, etc.)

Dependencies

pymupdf is new for the dataprep microservice, but it's not new to GenAIComps.

Tests

Tests were added to the following scripts:

tests/dataprep/test_dataprep_multimodal_redis_langchain.sh
tests/embeddings/test_embeddings_multimodal.sh
tests/lvms/test_lvms_llava.sh
tests/lvms/test_lvms_tgi-llava_on_intel_hpu.sh
tests/retrievers/test_retrievers_multimodal_redis_langchain.sh
tests/retrievers/test_retrievers_redis.sh

* Backend enhancements for image query capabilities for MultimodalQnA * Fix model name var Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Remove space at end of prompt Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Add env var for the max number of images sent to the LVM Signed-off-by: dmsuehir <dina.s.jones@intel.com> * README update for the MAX_IMAGES env var Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Remove prints Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Audio query functionality to multimodal backend (#8) Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added in audio dict creation Signed-off-by: okhleif-IL <omar.khleif@intel.com> * separated audio from prompt Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added ASR endpoint Signed-off-by: okhleif-IL <omar.khleif@intel.com> * removed ASR endpoints from mm embedding Signed-off-by: okhleif-IL <omar.khleif@intel.com> * edited return logic, fixed function call Signed-off-by: okhleif-IL <omar.khleif@intel.com> * added megaservice to elif Signed-off-by: okhleif-IL <omar.khleif@intel.com> * reworked helper func Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Append audio to prompt Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Reworked handle messages, added metadata Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Moved dictionary logic to right place Signed-off-by: okhleif-IL <omar.khleif@intel.com> * changed logic to rely on message len Signed-off-by: okhleif-IL <omar.khleif@intel.com> * list --> empty str Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: okhleif-IL <omar.khleif@intel.com> Signed-off-by: dmsuehir <dina.s.jones@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixed role bug where i never was > 0 Signed-off-by: okhleif-IL <omar.khleif@intel.com> * Fix after merge Signed-off-by: dmsuehir <dina.s.jones@intel.com> * removed whitespace Signed-off-by: okhleif-IL <omar.khleif@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix call to get role labels Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Gateway test updates images within the conversation Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Adds unit test coverage for audio query Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Update test to check the returned b64 types Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Update test since we don't expect images from the assistant Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Port number fix Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Formatting Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixed place where port number is set Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Remove old comment and added more accurate description Signed-off-by: dmsuehir <dina.s.jones@intel.com> * add comment in code about MAX_IMAGES Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Add Gaudi support for image query Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Fix to pass the retrieved image last Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Revert out gateway and gateway test code, due to its move to GenAIExamples Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Fix retriever test for checking for b64_img_str in the result Signed-off-by: dmsuehir <dina.s.jones@intel.com> --------- Signed-off-by: dmsuehir <dina.s.jones@intel.com> Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Signed-off-by: okhleif-IL <omar.khleif@intel.com> Co-authored-by: Omar Khleif <omar.khleif@intel.com> Co-authored-by: Melanie Hart Buehler <melanie.h.buehler@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>

…ge-query

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…nv file (#17) * changed all hardcoded ports to getenv with defaults instead Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Initial implementation of PDF ingestion Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * PDF ingestion fixes Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Adds a test for dataprep microservice Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Improved comments, variable name, and a docstring Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> * Updated for review feedback Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> --------- Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…ge-query Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…ge-query

for more information, see https://pre-commit.ci

codecov · 2025-01-10T22:39:10Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/proto/docarray.py	`99.43% <100.00%> (ø)`

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

comps/dataprep/multimodal/redis/langchain/prepare_videodoc_redis.py

comps/asr/src/integrations/dependency/whisper/whisper_server.py

comps/asr/src/opea_asr_microservice.py

* Fixing Multimodal Retriever Redis tests Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Code cleanup Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Remove debug changes Signed-off-by: dmsuehir <dina.s.jones@intel.com> * Formatting Signed-off-by: dmsuehir <dina.s.jones@intel.com> --------- Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…ge-query

…nto mmqna-image-query

for more information, see https://pre-commit.ci

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

…ge-query Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…nto mmqna-image-query

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

…ge-query

Per the proposed changes in this [RFC](https://github.com/opea-project/docs/blob/main/community/rfcs/24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA.md)'s Phase 2 plan, this PR adds support for image queries, PDF ingestion and display, and dynamic ports. There are also some bug fixes. This PR goes with [this one in GenAIComps](opea-project/GenAIComps#1134). Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Co-authored-by: Liang Lv <liang1.lv@intel.com>

According to the RFC's Phase 2 plan, this PR adds image query support, PDF ingestion support, and dynamic ports to the microservices used by MultimodalQnA. This PR goes with this one in GenAIExamples. Signed-off-by: dmsuehir <dina.s.jones@intel.com> Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…roject#1381) Per the proposed changes in this [RFC](https://github.com/opea-project/docs/blob/main/community/rfcs/24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA.md)'s Phase 2 plan, this PR adds support for image queries, PDF ingestion and display, and dynamic ports. There are also some bug fixes. This PR goes with [this one in GenAIComps](opea-project/GenAIComps#1134). Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Co-authored-by: Liang Lv <liang1.lv@intel.com> Signed-off-by: Chingis Yundunov <YundunovCN@sibedge.com>

According to the RFC's Phase 2 plan, this PR adds image query support, PDF ingestion support, and dynamic ports to the microservices used by MultimodalQnA. This PR goes with this one in GenAIExamples. Signed-off-by: dmsuehir <dina.s.jones@intel.com> Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

…roject#1381) Per the proposed changes in this [RFC](https://github.com/opea-project/docs/blob/main/community/rfcs/24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA.md)'s Phase 2 plan, this PR adds support for image queries, PDF ingestion and display, and dynamic ports. There are also some bug fixes. This PR goes with [this one in GenAIComps](opea-project/GenAIComps#1134). Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com> Co-authored-by: Liang Lv <liang1.lv@intel.com> Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>

dmsuehir and others added 6 commits December 16, 2024 10:02

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

36de7cf

…ge-query

Merge branch 'main' into mmqna-image-query

89655e7

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

Replaced hard coded ports with dynamically assigned values from set_e…

20a79e1

…nv file (#17) * changed all hardcoded ports to getenv with defaults instead Signed-off-by: okhleif-IL <omar.khleif@intel.com> --------- Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

fa61505

…ge-query Signed-off-by: dmsuehir <dina.s.jones@intel.com>

mhbuehler requested review from Spycsh, XinyaoWa, XinyuYe-Intel, chensuyue, ftian1, letonghan, lkk12014402 and lvliang-intel as code owners January 10, 2025 22:14

ashahba added this to the v1.2 milestone Jan 10, 2025

ashahba added the WIP label Jan 10, 2025

mhbuehler mentioned this pull request Jan 10, 2025

MultimodalQnA image query, pdf, dynamic ports, and UI updates opea-project/GenAIExamples#1381

Merged

3 tasks

ashahba added the r1.2 label Jan 10, 2025

dmsuehir and others added 2 commits January 10, 2025 14:35

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

a4db261

…ge-query

[pre-commit.ci] auto fixes from pre-commit.com hooks

cc4f41f

for more information, see https://pre-commit.ci

Fixed download location to permanent link (#18)

7b045f6

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

tileintel reviewed Jan 12, 2025

View reviewed changes

comps/dataprep/multimodal/redis/langchain/prepare_videodoc_redis.py Outdated Show resolved Hide resolved

Spycsh reviewed Jan 13, 2025

View reviewed changes

comps/asr/src/integrations/dependency/whisper/whisper_server.py Outdated Show resolved Hide resolved

Spycsh reviewed Jan 13, 2025

View reviewed changes

comps/asr/src/opea_asr_microservice.py Show resolved Hide resolved

dmsuehir and others added 5 commits January 13, 2025 11:20

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

cce77c9

…ge-query

Merge branch 'mmqna-image-query' of github.com:mhbuehler/GenAIComps i…

21a31b7

…nto mmqna-image-query

[pre-commit.ci] auto fixes from pre-commit.com hooks

10c0101

for more information, see https://pre-commit.ci

Improved local variable names (#20)

fc378d6

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

okhleif-10 and others added 2 commits January 13, 2025 15:12

reverted configurable asr port (#21)

20d6302

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

Merge branch 'main' into mmqna-image-query

284526f

This was referenced Jan 14, 2025

Update compose.yaml to use fixed internal port for the whisper server mhbuehler/GenAIExamples#42

Merged

Revert configurable internal whisper port mhbuehler/GenAIComps#22

Merged

ZePan110 and others added 2 commits January 15, 2025 10:26

Merge branch 'main' into mmqna-image-query

96a63ad

Merge branch 'main' into mmqna-image-query

90ff144

ashahba removed the WIP label Jan 16, 2025

dmsuehir added 3 commits January 16, 2025 10:49

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

d00d336

…ge-query Signed-off-by: dmsuehir <dina.s.jones@intel.com>

Merge branch 'mmqna-image-query' of github.com:mhbuehler/GenAIComps i…

eb4707a

…nto mmqna-image-query

Revert env var used for internal whisper port (#22)

b1c9c76

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

joshuayao requested review from Spycsh, dmsuehir and tileintel January 17, 2025 03:13

ashahba and others added 4 commits January 16, 2025 21:38

Merge branch 'main' into mmqna-image-query

195991f

Merge branch 'main' into mmqna-image-query

3cfe004

Merge branch 'main' of github.com:mhbuehler/GenAIComps into mmqna-ima…

f8f8ac6

…ge-query

Merge branch 'main' into mmqna-image-query

9b12915

dmsuehir approved these changes Jan 17, 2025

View reviewed changes

chensuyue approved these changes Jan 19, 2025

View reviewed changes

lkk12014402 approved these changes Jan 19, 2025

View reviewed changes

chensuyue merged commit ee0c11e into opea-project:main Jan 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MultimodalQnA image query, pdf, and dynamic ports#1134

MultimodalQnA image query, pdf, and dynamic ports#1134
chensuyue merged 25 commits intoopea-project:mainfrom
mhbuehler:mmqna-image-query

mhbuehler commented Jan 10, 2025 •

edited by ashahba

Loading

Uh oh!

codecov bot commented Jan 10, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

mhbuehler commented Jan 10, 2025 • edited by ashahba Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Type of change

Dependencies

Tests

Uh oh!

codecov bot commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mhbuehler commented Jan 10, 2025 •

edited by ashahba

Loading

codecov bot commented Jan 10, 2025 •

edited

Loading