cld2labs/HybridSearch

[arpannookala-12]

This is a straightforward initial addition of the full HybridSearch sample — five microservices (gateway, embedding, retrieval, ingestion, LLM), a Streamlit UI, docker-compose, and supporting scripts/docs.

[arpannookala-12]

Addressed all 4 review comments:

1. Repo URL — Updated clone URL to opea-project/Enterprise-Inference with correct cd path
2. Model config — Aligned model names with .env.example (bge-base-en-v1.5, bge-reranker-base, Qwen3-4B-Instruct-2507) and added a "Required Models" prerequisite section
3. docker compose — Changed docker-compose to docker compose for consistency
4. Per-service logs — Added individual log commands for all services (gateway, embedding, retrieval, llm, ingestion, ui)

[alexsin368]

@arpannookala-12 please resolve my comments when you address them. Also, I'm encountering an error with the ingestion service even though my embedding model is deployed and functional. I deployed EI with Keycloak. It seems there is an issue with the URL to the model endpoint, which should be https://api.example.com/bge-base-en-v1.5/v1/embeddings

[arpannookala-12]

@alexsin368 sure will be sure to resolve the comments as I address them, working on understanding the issue with the keycloak EI deployment, will get back on this

[arpannookala-12]

@alexsin368 I added some config support and api_client changes and I tested again by setting the following values

Please note that the reranker model was not deployed and the value for that is a placeholder, but the conclusion is that the embedding model worked with this effort

[alexsin368]

Functional with EI is deployed with GenAI Gateway. For keycloak deployment, still having an issue with the reranker. It seems the data it looks for is "texts", instead of "documents" because the bge-reranker-base model is deployed with vLLM on Gaudi. A fix would be to look for "texts" if EI is deployed with keycloak.

[arpannookala-12]

@alexsin368 — thanks for the findings on Gaudi + GenAI Gateway. Based on your testing, here's the full picture we've now confirmed:

Enterprise Inference	Hardware	Reranker config required	Code change needed
GenAI Gateway	Xeon	Yes (LiteLLM model update)	None — /v1/rerank works
GenAI Gateway	Gaudi	No	INFERENCE_BACKEND=tei
Keycloak / APISIX	Xeon	No	None
Keycloak / APISIX	Gaudi	No	None

Root cause: Gaudi uses the TEI (Text Embeddings Inference) backend which serves endpoints without the /v1 prefix (/embeddings, /rerank). Xeon uses vLLM which requires the /v1 prefix.

Fix added in this commit: A new INFERENCE_BACKEND env var (vllm by default, set to tei for Gaudi). When tei, all three services (embedding, retrieval, LLM) drop the /v1 prefix from their endpoint URLs.

For Gaudi + GenAI Gateway users, add this to .env:

INFERENCE_BACKEND=tei

All other deployments (Xeon, Keycloak/APISIX) keep the default INFERENCE_BACKEND=vllm and are unaffected.

[arpannookala-12]

Reranker fixes: batching + token overflow (latest commit a7d599e)

Two root causes were found and fixed for the 500 errors seen when uploading large documents:

---

1. Batch size overflow (413 Payload Too Large)

TOP_K_FUSION=50 was sending all 50 fusion candidates in a single rerank request. bge-reranker-base has a max batch size of 32 — anything over that returned a 413.

Fix: Added RERANKER_MAX_BATCH_SIZE config (default 32) to retrieval/config.py and docker-compose.yml. rerank_pairs() now loops over batches of that size, tracking index offsets so scores are written back to the correct positions in the full result list.

---

2. Token length overflow (500 EngineCore)

After the batch fix, batch 1 (32 docs) succeeded but batch 2 consistently returned 500 EngineCore encountered an issue. Root cause: bge-reranker-base has a 512-token max sequence length for query + document combined. Technical document chunks (code, numbers, punctuation) tokenize at ~2 chars/token in the worst case — at 1000-char truncation, batch 2 docs were pushing past the limit.

Fix: Truncate each document to 500 chars (~125 tokens) before sending to the reranker. This leaves safe headroom for the query and worst-case tokenization while staying within the model's limit. Tested at 1000 chars — batch 2 still failed. 500 chars — both batches return 200 OK consistently.

On quality: 500 chars captures the leading context of each chunk, which is what cross-encoders weight most heavily for relevance scoring. The alternative (intermittent 500s on certain document types) is a harder quality regression.

---

Both fixes are tested end-to-end on Xeon + GenAI Gateway and Xeon + Keycloak with an 8MB document upload.

[alexsin368]

suggested changes for user to adjust embedding batch size for embedding and ingestion services. ensures same value is used in both

[alexsin368]

@arpannookala-12 all security scans passed. you may remove code-scans.yaml.

[arpannookala-12]

@alexsin368 done, the code-scans file has been removed