Feat rdb summary wide table

[FOkvj]

Description

When the relational database table is a wide table, there are a lot of fields. In addition to many redundant fields, retrieving the full number of fields may also exceed the maximum sequence length accepted by the embedding model when performing summary embedding. As a result, the generated embedding cannot accurately reflect the semantic information of the summary. Therefore, for wide tables, I split the fields and the basic information of the table. If the number of fields in the table is too large, the fields will be divided into multiple chunks during summary, and the length of a chunk does not exceed the maximum sequence length of the embedding model. If the table is not wide, then the summary is the same as the original, and the table name and the table description and fields are in the same chunk. In the retrieval, the table name is retrieved first, then the table name (id) is used as filter, and the query is used for vector retrieval, and then the table name and table description are assembled with the field as the final result.

How Has This Been Tested?

Test summary of wide table and retrieve respectively in dbgpt/rag/assembler/tests/test_db_struct_assembler.py and dbgpt/rag/assembler/tests/test_embedding_assembler.py

Snapshots:

Checklist:

• My code follows the style guidelines of this project
• I have already rebased the commits and make the commit message conform to the project standard.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• Any dependent changes have been merged and published in downstream modules

[fangyinc]

@Aries-ckt Please review it.

[intfish123]

@Aries-ckt 有什么进展吗，啥时候能合进去，我目前也遇到了同样的问题

[FOkvj]

@Aries-ckt 有什么进展吗，啥时候能合进去，我目前也遇到了同样的问题

先用该分支试试看

[Aries-ckt]

你好, 这边pr功能没啥问题，主要有个顾虑，可能老用户的表schema 向量数据就找不到了

[FOkvj]

你好, 这边pr功能没啥问题，主要有个顾虑，可能老用户的表schema 向量数据就找不到了

针对该问题，已对代码作出了修改，可保证用户原有的向量数据正常被检索

[Aries-ckt]

@FOkvj , hi why do you remove field_vector_connector

[FOkvj]

@Aries-ckt here，I make it created by default, so I think those code is not necessary.

[Aries-ckt]

But when get_summary -> _similarity_search -> _retrieve_field it will meet self._field_vector_store_connector = None error.

[FOkvj]

@Aries-ckt fixed

[Aries-ckt]

Test Success

[Aries-ckt]

LGTM.

[Aries-ckt]

@FOkvj Are you in our WeChat group? and what's your WeChat alias?

[FOkvj]

@Aries-ckt Yep，I go by Wooop！ 😊

[csunny]

LGTM~