new: DataprepRequest model

[aMahanna]

Description

Introduces the ability to customize Dataprep input parameters by way of subclassing the DataprepRequest pydantic model.

Avoids having to introduce parameters unique to 1 or 2 Dataprep integrations across all Dataprep providers.

Similar discussion & solution here: #1466 (comment)

Feedback welcomed.

Issues

#1516

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

N/A

Tests

N/A

[codecov]

Codecov Report

Attention: Patch coverage is 41.17647% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
comps/cores/proto/api_protocol.py	41.17%	10 Missing ⚠️

Files with missing lines	Coverage Δ
comps/cores/proto/api_protocol.py	94.26% <41.17%> (-1.93%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aMahanna]

Looking for some assistance regarding the CI failures - what changes are we missing? it seems as if the models are not being recognized as valid input

[letonghan]

Hi @aMahanna , thanks for your great contribution!
For the data class design and CI issues, here are some comments:

1. Parse failure:
   Fastapi will parse pydantic model to json object, which will cause the failure of parsing files when uploading using multipart type. (reference)

2. Use dependency injection:
   Suggested method is to use dependency injection of fastapi to define dataprep request class. The common class is as below:

class DataprepRequest:
    def __init__(
        self,
        db_type: str = Form(None),
        files: Optional[Union[UploadFile, List[UploadFile]]] = File(None),
        link_list: Optional[str] = Form(None),
        xx: str = Form(None)
    ):
        self.db_type = db_type
        self.files = files
        self.link_list = link_list
        self.xx = xx

Redis class should be like:

  class RedisDataprepRequest(DataprepRequest):
    def __init__(
        index_name: Optional[str] = Form(None),
        **kwargs
    ):
        super().__init__(**kwargs)
        self.index_name = index_name

3. Problem of union class of endpoint input
   The usage of input: Union[DataprepRequest, RedisDataprepRequest, Neo4jDataprepRequest] only works for JSON type of request (application/json).
   For the multipart type of request, we need to add a param db_type in DataprepRequest class as above.
   In microservice.py, the input should be like:

async def ingest_files(
    base: Annotated[Optional[DataprepRequest], Depends()] = None,
    redis: Annotated[Optional[RedisDataprepRequest], Depends()] = None,
    neo4j: Annotated[Optional[Neo4jDataprepRequest], Depends()] = None,
):
    request = redis or neo4j or base

    if request is None:
        raise HTTPException(400, detail="Invalid request")

    # DO STH...

[aMahanna]

@letonghan

Thanks for you reply.

I'm confused why this is required for Dataprep if the Retriever supports the usage of the Union class for endpoint input: https://github.com/opea-project/GenAIComps/blob/main/comps/retrievers/src/opea_retrievers_microservice.py#L53-L63

@register_microservice(
    name="opea_service@retrievers",
    service_type=ServiceType.RETRIEVER,
    endpoint="/v1/retrieval",
    host="0.0.0.0",
    port=7000,
)
@register_statistics(names=["opea_service@retrievers"])
async def retrieve_docs(
    input: Union[EmbedDoc, EmbedMultimodalDoc, RetrievalRequest, ChatCompletionRequest]
) -> Union[SearchedDoc, SearchedMultimodalDoc, RetrievalResponse, ChatCompletionRequest]:

Can you please clarify?

[rbrugaro]

@aMahanna from the links in @letonghan answer I think is a parsing issue with different requests types. In dataprep because we do file ingestion it uses multipart/form-data and retriever uses application/json.

curl -X POST
*-H "Content-Type: multipart/form-data" *
-F "files=@./your_file.pdf"
-F "process_table=true"
-F "table_strategy=hq"
http://localhost:6007/v1/dataprep/ingest

curl http://${your_ip}:7000/v1/retrieval
-X POST
-d "{"text":"What is the revenue of Nike in 2023?","embedding":${your_embedding}}"
-H 'Content-Type: application/json'

[aMahanna]

Ah, understood. Thank you @rbrugaro and @letonghan. Addressed here: 7eabcfa

Follow-up question to @letonghan:

For the multipart type of request, we need to add a param db_type in DataprepRequest class as above.

This has been added, but what exactly would be the purpose of now having a db_type property? Would it be required in every API request?

i.e is db_type required?

curl -X POST
*-H "Content-Type: multipart/form-data" *
-F "files=@./your_file.pdf"
-F "process_table=true"
-F "table_strategy=hq"
-F "db_type=redis"
http://localhost:6007/v1/dataprep/ingest

[letonghan]

This has been added, but what exactly would be the purpose of now having a db_type property? Would it be required in every API request?

i.e is db_type required?

curl -X POST
*-H "Content-Type: multipart/form-data" *
-F "files=@./your_file.pdf"
-F "process_table=true"
-F "table_strategy=hq"
-F "db_type=redis"
http://localhost:6007/v1/dataprep/ingest

@aMahanna your are right, the db_type is not required. That's from my original thoughts and not needed anymore.
Thanks for making these updates!

[rbrugaro]

LGTM

[aMahanna]

thank you! I don't have permission to merge, so feel free to do so

[rbrugaro]

@aMahanna let me revert the merge, i just noticed quadrant milvus didn't pass? i thought i saw all passing.

[chensuyue]

@aMahanna let me revert the merge, i just noticed quadrant milvus didn't pass? i thought i saw all passing.

No, it's all pass with the second retry test, https://github.com/opea-project/GenAIComps/actions/runs/14517733895, what you see not pass is the results from the first round.