Skip to content

enable mysql db for sql agent#1431

Merged
chensuyue merged 8 commits intoopea-project:mainfrom
cheehook:add-mysql-connector-for-sql-agent
Apr 15, 2025
Merged

enable mysql db for sql agent#1431
chensuyue merged 8 commits intoopea-project:mainfrom
cheehook:add-mysql-connector-for-sql-agent

Conversation

@cheehook
Copy link
Copy Markdown
Contributor

Description

Add mysql connector to allow mysql db connection from SQL agent
Before this, sql agent is only enabled and validated with sqlitedb.
This change is needed to support GenAIStudio v1.3 release.

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

  • New feature (non-breaking change which adds new functionality)

Dependencies

  • mysql-connector-python

Tests

Tested by pointing db_path and db_url to AWS RDS MySQL database when running the docker container

Logs in SQL agent container before adding mysql connector
[2025-03-17 05:41:30,596] [ INFO] - comps-react-agent - ========initiating agent============ [2025-03-17 05:41:30,596] [ INFO] - comps-react-agent - args: Namespace(stream=False, port=9096, agent_name='OPEA_Default_Agent', strategy='sql_agent_llama', role_description='LLM enhanced agent', tools=None, recursion_limit=12, require_human_feedback=False, debug=False, model='gpt-4o-mini-2024-07-18', llm_engine='openai', llm_endpoint_url='', max_new_tokens=4096, top_k=10, top_p=0.95, temperature=0.1, repetition_penalty=1.03, return_full_text=False, custom_prompt=None, with_memory=False, memory_type='checkpointer', timeout=60, db_path='mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', db_name='dbname', use_hints=False, hints_file=None) Traceback (most recent call last): File "/home/user/comps/agent/src/agent.py", line 43, in <module> agent_inst = instantiate_agent(args) env_config: ['--port', '9096', '--strategy', 'sql_agent_llama', '--llm_endpoint_url', '', '--llm_engine', 'openai', '--model', 'gpt-4o-mini-2024-07-18', '--recursion_limit', '12', '--stream', 'false', '--max_new_tokens', '4096', '--temperature', '0.1', '--with_memory', 'false', '--db_path', 'mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', '--db_name', 'dbname', '--use_hints', 'false'] ==========sys_args==========: Namespace(stream=False, port=9096, agent_name='OPEA_Default_Agent', strategy='sql_agent_llama', role_description='LLM enhanced agent', tools=None, recursion_limit=12, require_human_feedback=False, debug=False, model='gpt-4o-mini-2024-07-18', llm_engine='openai', llm_endpoint_url='', max_new_tokens=4096, top_k=10, top_p=0.95, temperature=0.1, repetition_penalty=1.03, return_full_text=False, custom_prompt=None, with_memory=False, memory_type='checkpointer', timeout=60, db_path='mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', db_name='dbname', use_hints=False, hints_file=None) Initializing SQL Agent Llama Registered tools: [] @@@@ user defined tools: [] @@@@ Tools: ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/comps/agent/src/integrations/agent.py", line 44, in instantiate_agent return SQLAgentLlama(args, with_memory, custom_prompt=custom_prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/comps/agent/src/integrations/strategy/sqlagent/planner.py", line 130, in __init__ sql_tool = get_sql_query_tool(args.db_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/comps/agent/src/integrations/strategy/sqlagent/sql_tools.py", line 22, in get_sql_query_tool db = connect_to_db(db_path) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/comps/agent/src/integrations/strategy/sqlagent/sql_tools.py", line 9, in connect_to_db db = SQLDatabase.from_uri(db_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain_community/utilities/sql_database.py", line 139, in from_uri return cls(create_engine(database_uri, **_engine_args), **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<string>", line 2, in create_engine File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned return fn(*args, **kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine dbapi = dbapi_meth(**dbapi_args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/mysql/mysqlconnector.py", line 83, in import_dbapi from mysql import connector ModuleNotFoundError: No module named 'mysql'

Logs in SQL agent container after adding mysql connector

[2025-03-19 05:04:13,792] [    INFO] - comps-react-agent - ========initiating agent============
[2025-03-19 05:04:13,792] [    INFO] - comps-react-agent - args: Namespace(stream=False, port=9096, agent_name='OPEA_Default_Agent', strategy='sql_agent', role_description='LLM enhanced agent', tools=None, recursion_limit=12, require_human_feedback=False, debug=False, model='gpt-4o-mini-2024-07-18', llm_engine='openai', llm_endpoint_url='', max_new_tokens=4096, top_k=10, top_p=0.95, temperature=0.1, repetition_penalty=1.03, return_full_text=False, custom_prompt=None, with_memory=False, memory_type='checkpointer', timeout=60, db_path='mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', db_name='dbname', use_hints=False, hints_file=None)
/home/user/comps/agent/src/integrations/strategy/sqlagent/sql_tools.py:29: LangChainDeprecationWarning: The class `QuerySQLDataBaseTool` was deprecated in LangChain 0.3.12 and will be removed in 1.0. An updated version of the class exists in the :class:`~langchain-community package and should be used instead. To use it run `pip install -U :class:`~langchain-community` and import as `from :class:`~langchain_community.tools import QuerySQLDatabaseTool``.
  db_query_tool = QuerySQLDataBaseTool(db=db, name="sql_db_query", description=query_sql_database_tool_description)
[2025-03-19 05:04:25,412] [    INFO] - Base service - CORS is enabled.
[2025-03-19 05:04:25,412] [    INFO] - Base service - Setting up HTTP server
[2025-03-19 05:04:25,413] [    INFO] - Base service - Uvicorn server setup on port 9096
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9096 (Press CTRL+C to quit)
[2025-03-19 05:04:25,416] [    INFO] - Base service - HTTP server setup successful
[2025-03-19 05:24:42,744] [    INFO] - comps-react-agent - -----------NOT STREAMING-------------
env_config:  ['--port', '9096', '--strategy', 'sql_agent', '--llm_endpoint_url', '', '--llm_engine', 'openai', '--model', 'gpt-4o-mini-2024-07-18', '--recursion_limit', '12', '--stream', 'false', '--max_new_tokens', '4096', '--temperature', '0.1', '--with_memory', 'false', '--db_path', 'mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', '--db_name', 'dbname', '--use_hints', 'false']
==========sys_args==========:
 Namespace(stream=False, port=9096, agent_name='OPEA_Default_Agent', strategy='sql_agent', role_description='LLM enhanced agent', tools=None, recursion_limit=12, require_human_feedback=False, debug=False, model='gpt-4o-mini-2024-07-18', llm_engine='openai', llm_endpoint_url='', max_new_tokens=4096, top_k=10, top_p=0.95, temperature=0.1, repetition_penalty=1.03, return_full_text=False, custom_prompt=None, with_memory=False, memory_type='checkpointer', timeout=60, db_path='mysql+mysqlconnector://admin:password@some_db_instance.us-east-1.rds.amazonaws.com.com:3306/dbname', db_name='dbname', use_hints=False, hints_file=None)
Initializing SQL Agent
Registered tools:  []
SQL Query Tool Created:  description='Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. ' db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x7f1f09698890>
@@@@ ALL Tools:  [QuerySQLDataBaseTool(description='Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. ', db=<langchain_community.utilities.sql_database.SQLDatabase object at 0x7f1f09698890>)]
@@@ Initial State:  {'messages': [HumanMessage(content='How many projects are there in the Easydata platform?', additional_kwargs={}, response_metadata={})], 'is_last_step': False, 'hint': ''}
================================ Human Message =================================

How many projects are there in the Easydata platform?
----------Call Agent Node----------
================================== Ai Message ==================================

To determine how many projects are in the Easydata platform, I will query the `project` table, which contains information about the projects. 

Let's execute a SQL query to count the total number of projects in the `project` table. 

Here is the SQL query I will use:

sql
SELECT COUNT(*) AS total_projects FROM project;


Now, I will execute this query.
Tool Calls:
  sql_db_query (call_qObpvz37SoGmG9v72O2mmaDt)
 Call ID: call_qObpvz37SoGmG9v72O2mmaDt
  Args:
    query: SELECT COUNT(*) AS total_projects FROM project;
@@@@ Called Tool:  sql_db_query
@@@@ Going to Query Fixer
================================= Tool Message =================================
Name: sql_db_query

[(3722,)]
----------Call Query Fixer Node----------
@@@@ Executed SQL Query:  SELECT COUNT(*) AS total_projects FROM project;
@@@@ Execution Result:  [(3722,)]

Signed-off-by: cheehook <chee.hoo.kok@intel.com>
@chensuyue chensuyue merged commit 2736462 into opea-project:main Apr 15, 2025
12 checks passed
aMahanna pushed a commit to arangoml/GenAIComps1.3 that referenced this pull request Apr 15, 2025
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
aMahanna pushed a commit to arangoml/GenAIComps1.3 that referenced this pull request Apr 18, 2025
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
rbrugaro added a commit that referenced this pull request Apr 18, 2025
* Fix image build issue (#1553)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Unified default port number for the same service in text2graph and text2sql (#1554)

Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoDataprep` (#2)

* new: `third_parties/arangodb`

* new: `OpeaArangoDataprep`

* cleanup

* fix: `vllm` instead of `tgi`

* fix: dataprep compsoe

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoRetriever` (#3)

* new: `OpeaArangoRetriever`

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: deps

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix typo: `test_retrievers_arango.sh`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* updated retriever-arango compose file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add json-repair to dataprep-arango requirements

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix network error, change WORKPATH

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extra time for health check retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extended retriever healthcheck 90secs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update arangodb.py

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Removing hugging face token requirement from test file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update test_dataprep_arango with network tests and additional logs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Running CI after docker rate limit

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Base case remove HF_token, no additional tests

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Adding VLLM check and logs, currently VLLM not working in CI/CD

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: compose.yaml

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb healthcheck

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: retriever test

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* rem: unused vars

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: indent

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* temp: swap vllm healthcheck with sleep

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: component name typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: support `EmbedDoc` for retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: `getattr`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: CURL command

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert 6061484

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update xtune file and change DDP paramter (#1552)

Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add N/A option (#1561)

Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Test latest gaudi docker container (#1477)

Update base gaudi container into the latest version, docker pull vault.habana.ai/gaudi-docker/1.20.1/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest, https://docs.habana.ai/en/latest/Installation_Guide/Additional_Installation/Docker_Installation.html#use-intel-gaudi-containers

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix audioqna male voice setting (#1559)

Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* added error handling for lvm (#1556)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* enable mysql db for sql agent (#1431)

Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Enlarge DocSum prompt buffer (#1567)

* Enlarge DocSum prompt buffer
Follow PR #1471

Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update vLLM parameter max-seq-len-to-capture (#1565)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: missing import

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: healtcheck for dataprep-arangodb

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb readmes

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT2)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT3)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: whitespace

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove Transformers versions from requirements.txt file (#1547)

* Remove Transformers versions from requirements.txt file

Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove index_names from files for dataprep-get request  (#1569)

* remove index_names from files fot get request

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation check for 'all' as an index_name

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* fix for readme file

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Upgrade Optimum Habana version to fix security check issue (#1571)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Make llamaguard compatible with both TGI and vLLM (#1581)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix Dockerfile error and add CI test for IPEX (#1585)

* Fix Dockerfile error and add CI teat

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Reduce multilang tts docker image size (#1587)

* fix audioqna male voice setting

* reduce multilang tts docker image size

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* unset OPENAI_KEY in CI test (#1586)

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Add AWS Credentials for CD test (#1588)

* Fix CD test issue

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: shorten ingest_dataprep.txt

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: a4d943e

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `DataprepRequest` model (#1525)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: #1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: #1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* Fix dataprep request class issue of Redis (#1)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: #1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* fix dataprep request class of redis

Signed-off-by: letonghan <letong.han@intel.com>

* revert change in redis.py

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Anthony Mahanna <43019056+aMahanna@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert: `DataprepRequest` for multimodal

* revert: `DataprepRequest` for multimodal (PT2)

* fix: conditionally fetch unique `DataprepRequest` attributes

* fix bugs in dataprep util script

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert change of pgvector

Signed-off-by: letonghan <letong.han@intel.com>

* fix indices bug for redis

Signed-off-by: letonghan <letong.han@intel.com>

* minor fix for redis

Signed-off-by: letonghan <letong.han@intel.com>

* ingest file into rag_redis_test

Signed-off-by: letonghan <letong.han@intel.com>

* update indice name

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: letonghan <letong.han@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: bc4445c

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: d17f6aa

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Revert "new: `DataprepRequest` model (#1525)" (#1592)

This reverts commit 88947ab.

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add hyperlinks

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: 4eb9ec4f

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: ArangoDBDataprepRequest

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: delete_files

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* remove: env mutation

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: move openai key env var to top of file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Co-authored-by: lasyasn <lasyan640@gmail.com>
Co-authored-by: Ajay Kallepalli <ajay.r.kallepalli@gmail.com>
Co-authored-by: jilongW <109333127+jilongW@users.noreply.github.com>
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: Spycsh <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: cheehook <chee.hoo.kok@intel.com>
Co-authored-by: XinyaoWa <xinyao.wang@intel.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: letonghan <letong.han@intel.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/GenAIComps that referenced this pull request Apr 29, 2025
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>
chyundunovDatamonsters pushed a commit to chyundunovDatamonsters/GenAIComps that referenced this pull request Apr 29, 2025
)

* Fix image build issue (opea-project#1553)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Unified default port number for the same service in text2graph and text2sql (opea-project#1554)

Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoDataprep` (opea-project#2)

* new: `third_parties/arangodb`

* new: `OpeaArangoDataprep`

* cleanup

* fix: `vllm` instead of `tgi`

* fix: dataprep compsoe

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoRetriever` (opea-project#3)

* new: `OpeaArangoRetriever`

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: deps

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix typo: `test_retrievers_arango.sh`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* updated retriever-arango compose file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add json-repair to dataprep-arango requirements

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix network error, change WORKPATH

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extra time for health check retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extended retriever healthcheck 90secs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update arangodb.py

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Removing hugging face token requirement from test file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update test_dataprep_arango with network tests and additional logs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Running CI after docker rate limit

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Base case remove HF_token, no additional tests

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Adding VLLM check and logs, currently VLLM not working in CI/CD

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: compose.yaml

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb healthcheck

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: retriever test

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* rem: unused vars

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: indent

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* temp: swap vllm healthcheck with sleep

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: component name typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: support `EmbedDoc` for retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: `getattr`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: CURL command

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert 6061484

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update xtune file and change DDP paramter (opea-project#1552)

Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add N/A option (opea-project#1561)

Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Test latest gaudi docker container (opea-project#1477)

Update base gaudi container into the latest version, docker pull vault.habana.ai/gaudi-docker/1.20.1/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest, https://docs.habana.ai/en/latest/Installation_Guide/Additional_Installation/Docker_Installation.html#use-intel-gaudi-containers

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix audioqna male voice setting (opea-project#1559)

Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* added error handling for lvm (opea-project#1556)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* enable mysql db for sql agent (opea-project#1431)

Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Enlarge DocSum prompt buffer (opea-project#1567)

* Enlarge DocSum prompt buffer
Follow PR opea-project#1471

Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update vLLM parameter max-seq-len-to-capture (opea-project#1565)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: missing import

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: healtcheck for dataprep-arangodb

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb readmes

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT2)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT3)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: whitespace

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove Transformers versions from requirements.txt file (opea-project#1547)

* Remove Transformers versions from requirements.txt file

Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove index_names from files for dataprep-get request  (opea-project#1569)

* remove index_names from files fot get request

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation check for 'all' as an index_name

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* fix for readme file

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Upgrade Optimum Habana version to fix security check issue (opea-project#1571)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Make llamaguard compatible with both TGI and vLLM (opea-project#1581)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix Dockerfile error and add CI test for IPEX (opea-project#1585)

* Fix Dockerfile error and add CI teat

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Reduce multilang tts docker image size (opea-project#1587)

* fix audioqna male voice setting

* reduce multilang tts docker image size

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* unset OPENAI_KEY in CI test (opea-project#1586)

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Add AWS Credentials for CD test (opea-project#1588)

* Fix CD test issue

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: shorten ingest_dataprep.txt

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: a4d943e

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `DataprepRequest` model (opea-project#1525)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* Fix dataprep request class issue of Redis (opea-project#1)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* fix dataprep request class of redis

Signed-off-by: letonghan <letong.han@intel.com>

* revert change in redis.py

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Anthony Mahanna <43019056+aMahanna@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert: `DataprepRequest` for multimodal

* revert: `DataprepRequest` for multimodal (PT2)

* fix: conditionally fetch unique `DataprepRequest` attributes

* fix bugs in dataprep util script

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert change of pgvector

Signed-off-by: letonghan <letong.han@intel.com>

* fix indices bug for redis

Signed-off-by: letonghan <letong.han@intel.com>

* minor fix for redis

Signed-off-by: letonghan <letong.han@intel.com>

* ingest file into rag_redis_test

Signed-off-by: letonghan <letong.han@intel.com>

* update indice name

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: letonghan <letong.han@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: bc4445c

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: d17f6aa

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Revert "new: `DataprepRequest` model (opea-project#1525)" (opea-project#1592)

This reverts commit 88947ab.

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add hyperlinks

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: 4eb9ec4f

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: ArangoDBDataprepRequest

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: delete_files

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* remove: env mutation

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: move openai key env var to top of file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Co-authored-by: lasyasn <lasyan640@gmail.com>
Co-authored-by: Ajay Kallepalli <ajay.r.kallepalli@gmail.com>
Co-authored-by: jilongW <109333127+jilongW@users.noreply.github.com>
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: Spycsh <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: cheehook <chee.hoo.kok@intel.com>
Co-authored-by: XinyaoWa <xinyao.wang@intel.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: letonghan <letong.han@intel.com>
Signed-off-by: Chingis Yundunov <c.yundunov@datamonsters.com>
madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request May 12, 2025
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
madison-evans pushed a commit to SAPD-Intel/GenAIComps that referenced this pull request May 12, 2025
)

* Fix image build issue (opea-project#1553)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Unified default port number for the same service in text2graph and text2sql (opea-project#1554)

Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoDataprep` (opea-project#2)

* new: `third_parties/arangodb`

* new: `OpeaArangoDataprep`

* cleanup

* fix: `vllm` instead of `tgi`

* fix: dataprep compsoe

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `OpeaArangoRetriever` (opea-project#3)

* new: `OpeaArangoRetriever`

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: deps

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix typo: `test_retrievers_arango.sh`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* updated retriever-arango compose file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add json-repair to dataprep-arango requirements

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix network error, change WORKPATH

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extra time for health check retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* extended retriever healthcheck 90secs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* correction

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update arangodb.py

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Removing hugging face token requirement from test file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update test_dataprep_arango with network tests and additional logs

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Running CI after docker rate limit

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Base case remove HF_token, no additional tests

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Adding VLLM check and logs, currently VLLM not working in CI/CD

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: compose.yaml

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb healthcheck

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: retriever test

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* rem: unused vars

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: indent

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* temp: swap vllm healthcheck with sleep

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: component name typo

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: support `EmbedDoc` for retriever

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: `getattr`

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: CURL command

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert 6061484

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update xtune file and change DDP paramter (opea-project#1552)

Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add N/A option (opea-project#1561)

Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Test latest gaudi docker container (opea-project#1477)

Update base gaudi container into the latest version, docker pull vault.habana.ai/gaudi-docker/1.20.1/ubuntu22.04/habanalabs/pytorch-installer-2.6.0:latest, https://docs.habana.ai/en/latest/Installation_Guide/Additional_Installation/Docker_Installation.html#use-intel-gaudi-containers

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix audioqna male voice setting (opea-project#1559)

Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* added error handling for lvm (opea-project#1556)

Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* enable mysql db for sql agent (opea-project#1431)

Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Enlarge DocSum prompt buffer (opea-project#1567)

* Enlarge DocSum prompt buffer
Follow PR opea-project#1471

Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Update vLLM parameter max-seq-len-to-capture (opea-project#1565)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: missing import

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: healtcheck for dataprep-arangodb

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: arangodb readmes

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT2)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: test_dataprep_arango.sh (PT3)

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: test_dataprep_arango.sh

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: whitespace

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove Transformers versions from requirements.txt file (opea-project#1547)

* Remove Transformers versions from requirements.txt file

Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Remove index_names from files for dataprep-get request  (opea-project#1569)

* remove index_names from files fot get request

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* update the tests

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation check for 'all' as an index_name

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* fix for readme file

Signed-off-by: Mustafa <mustafa.cetin@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Upgrade Optimum Habana version to fix security check issue (opea-project#1571)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Make llamaguard compatible with both TGI and vLLM (opea-project#1581)

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Fix Dockerfile error and add CI test for IPEX (opea-project#1585)

* Fix Dockerfile error and add CI teat

Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Reduce multilang tts docker image size (opea-project#1587)

* fix audioqna male voice setting

* reduce multilang tts docker image size

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* unset OPENAI_KEY in CI test (opea-project#1586)

Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Add AWS Credentials for CD test (opea-project#1588)

* Fix CD test issue

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* update: shorten ingest_dataprep.txt

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: a4d943e

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: `DataprepRequest` model (opea-project#1525)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* Fix dataprep request class issue of Redis (opea-project#1)

* new: `DataprepRequest`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: docstrings

* rem: `ingest_from_graphDB`

* new: dep injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: verbose `input` processing

* attempt: replace `kwargs` with params

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rem: `db_type`

ref: opea-project#1525 (comment)

* attempt: require `base`

* Revert "attempt: require `base`"

This reverts commit 620ca6b.

* fix dataprep request class of redis

Signed-off-by: letonghan <letong.han@intel.com>

* revert change in redis.py

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Anthony Mahanna <43019056+aMahanna@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert: `DataprepRequest` for multimodal

* revert: `DataprepRequest` for multimodal (PT2)

* fix: conditionally fetch unique `DataprepRequest` attributes

* fix bugs in dataprep util script

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* revert change of pgvector

Signed-off-by: letonghan <letong.han@intel.com>

* fix indices bug for redis

Signed-off-by: letonghan <letong.han@intel.com>

* minor fix for redis

Signed-off-by: letonghan <letong.han@intel.com>

* ingest file into rag_redis_test

Signed-off-by: letonghan <letong.han@intel.com>

* update indice name

Signed-off-by: letonghan <letong.han@intel.com>

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: letonghan <letong.han@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: bc4445c

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: d17f6aa

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* Revert "new: `DataprepRequest` model (opea-project#1525)" (opea-project#1592)

This reverts commit 2cc9559.

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* add hyperlinks

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* revert: 4eb9ec4f

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* new: ArangoDBDataprepRequest

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: lint

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* cleanup: delete_files

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* remove: env mutation

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

* fix: move openai key env var to top of file

Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>

---------

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Anthony Mahanna <anthony.mahanna@arangodb.com>
Signed-off-by: Yao, Qing <qing.yao@intel.com>
Signed-off-by: jilongwa <jilong.wang@intel.com>
Signed-off-by: ZhangJianyu <zhang.jianyu@outlook.com>
Signed-off-by: okhleif-IL <omar.khleif@intel.com>
Signed-off-by: cheehook <chee.hoo.kok@intel.com>
Signed-off-by: XinyaoWa <xinyao.wang@intel.com>
Signed-off-by: lvliang-intel <liang1.lv@intel.com>
Signed-off-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Rita Brugarolas <rita.brugarolas.brufau@intel.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Co-authored-by: lasyasn <lasyan640@gmail.com>
Co-authored-by: Ajay Kallepalli <ajay.r.kallepalli@gmail.com>
Co-authored-by: jilongW <109333127+jilongW@users.noreply.github.com>
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
Co-authored-by: ZhangJianyu <zhang.jianyu@outlook.com>
Co-authored-by: Spycsh <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: Omar Khleif <omar.khleif@intel.com>
Co-authored-by: cheehook <chee.hoo.kok@intel.com>
Co-authored-by: XinyaoWa <xinyao.wang@intel.com>
Co-authored-by: Liang Lv <liang1.lv@intel.com>
Co-authored-by: Abolfazl Shahbazi <12436063+ashahba@users.noreply.github.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: rbrugaro <rita.brugarolas.brufau@intel.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: letonghan <letong.han@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants