[Data Layer] Add data layer and embedding component#7

Merged

lvliang-intel merged 14 commits intomainfrom

letong/data_embedding

Apr 26, 2024

Collaborator

letonghan commented Apr 26, 2024

Type of Change

feature

Description

Add _docarray.py for data transfer of microservices.
Add embedding microservice.
Add requirements.txt for embedding service.

How has this PR been tested?

Local tested on SPR.

Dependency Change?

None

letonghan and others added 14 commits

April 26, 2024 16:44


          add data layer for microservices, add embedding example service

334b5fb

Signed-off-by: letonghan <letong.han@intel.com>


          Merge branch 'main' of https://github.com/opea-project/GenAIComps int…

cc3cf8d

…o letong/data_embedding

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          refine the code structure

845f3ac

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

4c9cbeb

for more information, see https://pre-commit.ci


          fix import path

87fae0f

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          Merge branch 'main' of https://github.com/opea-project/GenAIComps int…

1788f84

…o letong/data_embedding

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          refine code structure

e977f7a

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

de2ff23

for more information, see https://pre-commit.ci


          add ut

b3b9c82

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          Merge branch 'letong/data_embedding' of https://github.com/opea-proje…

1d6bf02

…ct/GenAIComps into letong/data_embedding

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          remove scheduler

20f4145

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          [pre-commit.ci] auto fixes from pre-commit.com hooks

536c3e4

for more information, see https://pre-commit.ci


          Merge branch 'letong/data_embedding' of https://github.com/opea-proje…

e90ac20

…ct/GenAIComps into letong/data_embedding

Signed-off-by: lvliang-intel <liang1.lv@intel.com>


          fix __init__.py

3423cc2

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

lvliang-intel approved these changes

View reviewed changes

lvliang-intel merged commit 23a7288 into main

lkk12014402 pushed a commit that referenced this pull request


          Merge pull request #7 from opea-project/add_ci

c033459

add template

lkk12014402 pushed a commit that referenced this pull request


          Merge pull request #7 from intel/letong/chat_history

9451e45

Support chat history for ChatQnA chat_stream

chensuyue deleted the letong/data_embedding branch

August 19, 2024 02:16

Spycsh pushed a commit that referenced this pull request


          MultimodalQnA Image and Audio Support Phase 1 (#852)

29ef642

* Adds an endpoint for image ingestion

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Combined image and video endpoint

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add test and update README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* fixed variable name for embedding model (#1)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Fixed test script

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Remove redundant function

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* get_videos, delete_videos --> get_files, delete_files (#3)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Updates test per review feedback

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Fixed test

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add support for audio files multimodal data ingestion (#4)

* Add support for audio files multimodal data ingestion

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Update function name

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Change videos_with_transcripts to ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add image support to video ingestion with transcript functionality

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Update test and README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Updated for review suggestions

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add two tests for ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* LVM TGI Gaudi update for prompts without images (#7)

* LVM Gaudi TGI update for prompts without images

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Wording

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Add a test

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change dummy image to be b64 encoded instead of the url (#9)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Updates based on review feedback (#10)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Test fix (#11)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>

madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request


          [Data Layer] Add data layer and embedding component (opea-project#7)

d06e08e

* add data layer for microservices, add embedding example service

Signed-off-by: letonghan <letong.han@intel.com>

* refine the code structure

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix import path

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* refine code structure

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add ut

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* remove scheduler

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix __init__.py

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: lvliang-intel <liang1.lv@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request


          MultimodalQnA Image and Audio Support Phase 1 (opea-project#852)

e4d2f1c

* Adds an endpoint for image ingestion

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Combined image and video endpoint

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add test and update README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* fixed variable name for embedding model (opea-project#1)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Fixed test script

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Remove redundant function

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* get_videos, delete_videos --> get_files, delete_files (opea-project#3)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Updates test per review feedback

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Fixed test

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add support for audio files multimodal data ingestion (opea-project#4)

* Add support for audio files multimodal data ingestion

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Update function name

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Change videos_with_transcripts to ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add image support to video ingestion with transcript functionality

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Update test and README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Updated for review suggestions

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add two tests for ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* LVM TGI Gaudi update for prompts without images (opea-project#7)

* LVM Gaudi TGI update for prompts without images

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Wording

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Add a test

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change dummy image to be b64 encoded instead of the url (opea-project#9)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Updates based on review feedback (opea-project#10)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Test fix (opea-project#11)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>

WenjiaoYue pushed a commit that referenced this pull request


          Merge pull request #7 from opea-project/add_ci

f21113c

add template

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet