feat: Enhance the triplets extraction in the knowledge graph by the batch size

[Appointat]

Description

It calls the async function to accelerate the process of triplets extraction from the chunk text. The config can be set in the .env TRIPLET_EXTRACTION_BATCH_SIZE (default to 20).

How Has This Been Tested?

I have run the app server by set the value of TRIPLET_EXTRACTION_BATCH_SIZE differently. (1, 5, 100), and the running time varies.

Snapshots:

        batch_size = self._triplet_extraction_batch_size

        for i in range(0, len(chunks), batch_size):
            batch_chunks = chunks[i : i + batch_size]

            extraction_tasks = [
                self._graph_extractor.extract(chunk.content) for chunk in batch_chunks
            ]
            async_graphs: List[List[MemoryGraph]] = await asyncio.gather(
                *extraction_tasks
            )

            for chunk, graphs in zip(batch_chunks, async_graphs):
                for graph in graphs:
                    if document_graph_enabled:
                        # append the chunk id to the edge
                        for edge in graph.edges():
                            edge.set_prop("_chunk_id", chunk.chunk_id)
                            graph.append_edge(edge=edge)

                    # upsert the graph
                    self._graph_store_apdater.upsert_graph(graph)

                    # chunk -> include -> entity
                    if document_graph_enabled:
                        for vertex in graph.vertices():
                            self._graph_store_apdater.upsert_chunk_include_entity(
                                chunk=chunk, entity=vertex
                            )

Checklist:

• My code follows the style guidelines of this project
• I have already rebased the commits and make the commit message conform to the project standard.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• Any dependent changes have been merged and published in downstream modules

[Appointat]

@Aries-ckt @fanzhidongyzby Could you please review it and add some tags? thanks

[Aries-ckt]

LGTM.

[fanzhidongyzby]

The read and write order of chunk_history in _graph_extractor needs to be adjusted, otherwise it will lead to inconsistency between text block recall and serial semantics.

[Appointat]

The read and write order of chunk_history in _graph_extractor needs to be adjusted, otherwise it will lead to inconsistency between text block recall and serial semantics.

Thank you for your comment, I fixed it just now.

[fanzhidongyzby]

refine the code by following comments

[Appointat]

[fanzhidongyzby]

lgtm

[Aries-ckt]

LGTM