Skip to content

Add DAG flow#9

Merged
lvliang-intel merged 3 commits intomainfrom
genaicomps_flow
Apr 26, 2024
Merged

Add DAG flow#9
lvliang-intel merged 3 commits intomainfrom
genaicomps_flow

Conversation

@Spycsh
Copy link
Copy Markdown
Collaborator

@Spycsh Spycsh commented Apr 26, 2024 •
edited
Loading

Type of Change

feature

Description

Microservices dataflow is organized in a DAG, we expose two ways: Python API or Yaml config to define that. Further features are to be done including Ingress/Egress, megaservice API etc.
usage e.g.

Yaml

    - (s1, s5) >> s2
    - s2 >> (s3, s4)
    - s3 >> s4

equivalent to

Python

service_builder = ServiceBuilder(port=1234, hostfile=None)
s1 = BaseService(id="s1", endpoint="http://localhost:8081/v1/add")
s2 = BaseService(id="s2", endpoint="http://localhost:8082/v1/add")
s3 = BaseService(id="s3", endpoint="http://localhost:8083/v1/add")
s4 = BaseService(id="s4", endpoint="http://localhost:8084/v1/add")
s5 = BaseService(id="s5", endpoint="http://localhost:8085/v1/add")
service_builder.add(s1).add(s2).add(s3).add(s4).add(s5)
service_builder.flow_to(s1, s2)
service_builder.flow_to(s5, s2)
service_builder.flow_to(s2, s3)
service_builder.flow_to(s2, s4)
service_builder.flow_to(s3, s4)

How has this PR been tested?

local test

Dependency Change?

None

@lvliang-intel lvliang-intel merged commit b3f83dd into main Apr 26, 2024
lkk12014402 pushed a commit that referenced this pull request Jul 22, 2024
@changwangss
sperate main.py to lm_eval and bigcode_eval (#9)
36be820 
Signed-off-by: changwangss <chang1.wang@intel.com>
lkk12014402 pushed a commit that referenced this pull request Aug 8, 2024
@lvliang-intel
Support GuardRails (#9)
ed594d7 
* add guardrails

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix dockerfile

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* fix integration issues

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

* add package lock json file

Signed-off-by: lvliang-intel <liang1.lv@intel.com>

---------

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
@chensuyue chensuyue deleted the genaicomps_flow branch August 19, 2024 02:16
Spycsh pushed a commit that referenced this pull request Nov 8, 2024
@mhbuehler @dmsuehir
@okhleif-10 @pre-commit-ci @ashahba
MultimodalQnA Image and Audio Support Phase 1 (#852)
29ef642 
* Adds an endpoint for image ingestion

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Combined image and video endpoint

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add test and update README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* fixed variable name for embedding model (#1)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Fixed test script

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Remove redundant function

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* get_videos, delete_videos --> get_files, delete_files (#3)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Updates test per review feedback

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Fixed test

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add support for audio files multimodal data ingestion (#4)

* Add support for audio files multimodal data ingestion

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Update function name

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Change videos_with_transcripts to ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add image support to video ingestion with transcript functionality

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Update test and README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Updated for review suggestions

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add two tests for ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* LVM TGI Gaudi update for prompts without images (#7)

* LVM Gaudi TGI update for prompts without images

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Wording

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Add a test

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change dummy image to be b64 encoded instead of the url (#9)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Updates based on review feedback (#10)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Test fix (#11)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request May 12, 2025
@Spycsh @pre-commit-ci
Add DAG flow (opea-project#9)
bb54ea0 
* add basic DAG flow

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* comment test code

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request May 12, 2025
@mhbuehler @dmsuehir
@okhleif-10 @pre-commit-ci @ashahba
MultimodalQnA Image and Audio Support Phase 1 (opea-project#852)
e4d2f1c 
* Adds an endpoint for image ingestion

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Combined image and video endpoint

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add test and update README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* fixed variable name for embedding model (opea-project#1)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Fixed test script

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Remove redundant function

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* get_videos, delete_videos --> get_files, delete_files (opea-project#3)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>

* Updates test per review feedback

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Fixed test

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add support for audio files multimodal data ingestion (opea-project#4)

* Add support for audio files multimodal data ingestion

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Update function name

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Change videos_with_transcripts to ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add image support to video ingestion with transcript functionality

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Update test and README

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Updated for review suggestions

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* Add two tests for ingest_with_text

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>

* LVM TGI Gaudi update for prompts without images (opea-project#7)

* LVM Gaudi TGI update for prompts without images

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Wording

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Add a test

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change dummy image to be b64 encoded instead of the url (opea-project#9)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Updates based on review feedback (opea-project#10)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

* Test fix (opea-project#11)

Signed-off-by: dmsuehir <dina.s.jones@intel.com>

---------

Signed-off-by: Melanie Buehler <melanie.h.buehler@intel.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: dmsuehir <dina.s.jones@intel.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
WenjiaoYue pushed a commit that referenced this pull request Jul 7, 2025
@changwangss
sperate main.py to lm_eval and bigcode_eval (#9)
491fbee 
Signed-off-by: changwangss <chang1.wang@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lvliang-intel lvliang-intel lvliang-intel approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Spycsh @lvliang-intel