[WIP] Add support for OPEA LLMs in Llama-Index

[logan-markewich]

Description

This PR adds a llama-index wrapper for LLMs hosted by OPEA's GenAIComps library

A few unknowns/notes from my part

• Will any hosted LLM work with only chat messages? (if so, then I can keep the implementation as is!)
• I still need to try and spin up an OPEA LLM micro service to test this
• It seems like OPEA LLMs try to bundle some RAG functionality into the LLM micro service, but I skipped that, since llama-index is driving the prompting

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• TODO: I added new unit tests to cover this change

[ftian1]

thanks for this PR. I am one of contributors in OPEA. I have a comment here: such code should have an assumption that the OPEA microservice has been deployed in somewhere, either in local or in a cloud env. Then user could pass down the service ip to access. I would suggest to have words in document to mention this prerequisite.

[rbrugaro]

Thanks @logan-markewich!

I tested your PR with OPEA microservices. No issues with llm but there is an issue w embeddings.

If you want to try yourself you can launch the opea microservice following one of the options of this README: https://github.com/opea-project/GenAIComps/tree/main/comps/embeddings/tei/llama_index

Here are some checks after launching OPEA embedding microservice:
Healthcheck
$ curl [http://localhost:6000/v1/health_check](http://localhost:6000/v1/health_check/)
-X GET
-H 'Content-Type: application/json'
{"Service Title":"opea_service@embedding_tei_langchain/MicroService","Service Description":"OPEA Microservice Infrastructure"}

test with TextDoc input
$ curl [http://localhost:6000/v1/embeddings](http://localhost:6000/v1/embeddings/)
-X POST
-d '{"text":"Hello, world!"}'
-H 'Content-Type: application/json'
{"id":"99f4c65b5d0400e6e7872c1c40ccca22","text":"Hello, world!","embedding":[0.0072799586,0.03128243,0.042269558,-0.0063485163,-0.0077530267,0.04410854,0.0590988,0.045488138,-0.02227289,-0.037855495,-0.0067130174,-0.003363003,-0.08337993,-0.0051587764,0.018290782,0.05117798,0.031379085,0.013451039,0.006205124,0.03658391,-0.034309622,0.004967953,0.03619922,0.013945511,0.02123839

test with EmbeddingRequest input
$ curl http://localhost:6000/v1/embeddings
-H "Content-Type: application/json"
-d '{
"input": "The food was delicious and the waiter...",
"model": "BAAI/bge-base-en-v1.5",
"encoding_format": "float"
}'
{"object":"list","model":null,"data":[{"index":0,"object":"embedding","embedding":[-0.047599554,-0.0055408874,0.018720696,-0.02567272,0.06591436,0.010764824,-0.017667083,0.05037338,-0.0014907457,-0.015081286,0.017558929,-0.013840443,-0.008849931,0.020852309,0.0034780612,0.008818663,0.0742632,-0.031206904,0.039376773,-0.037804432,-0.022693759,-0.014756503,-0.011191109,0.012695486,-0.014381637,-0.022319226,-0.007750338,0.024972478,-0.031080812,-0.044731475,0.050228924,0.031221831,-0.013695359,-0.030721791,-0.012819336,-0.0092149945,0.04743948,-0.0103669595,-0.03255411,0.03777905,-0.0052547073,0.00875531,0.024079613,0.041530963,-0.

Input in embeddingRequest is String
this is input model='BAAI/bge-base-en-v1.5' input='The food was delicious and the waiter...' encoding_format='float' dimensions=None user=None request_type='embedding' and type <class 'comps.cores.proto.api_proto
col.EmbeddingRequest'>

Using your code:

from llama_index.embeddings.opea import OPEAEmbedding

embed_model = OPEAEmbedding(
    model_name="BAAI/bge-base-en-v1.5",
    api_base="http://localhost:6000/v1",
    embed_batch_size=10,
)

embeddings = embed_model.get_text_embedding("text")

Input of embeddingRequest is converting the string to a list
[2024-11-06 05:55:00,090] [ INFO] - embedding_tei_langchain - this is input model='BAAI/bge-base-en-v1.5' input=['text'] encoding_format='base64' dimensions=None user=None request_type='embedding' and type <class 'comps.cores.proto.api_protocol.EmbeddingRequest'>

To make it work I had to remove the square brackets in the openai get_embedding function:

llama_index/llama-index-integrations/embeddings/llama-index-embeddings-openai/llama_index/embeddings/openai/base.py

Line 137 in 5b581a3

client.embeddings.create(input=[text], model=engine, **kwargs).data[0].embedding

Please have a check!

[logan-markewich]

Thanks for testing @rbrugaro -- taking a look

[logan-markewich]

@rbrugaro it seems like the OPEA embeddings are not fully openai-compatible?

Only being able to embed a single piece of text at a time is probably not ideal 😅 -- I think I raised a slightly related issue in the opea repo (mentioned above)

[rbrugaro]

@logan-markewich I have tested with OPEA embedding microservice form here and all the issues brought up earlier have been resolved on the OPEA side. Could you please merge if it looks good on your side?