Bump: ChatQnA Example with OpenAI-Compatible Endpoint#2091

Merged

joshuayao merged 56 commits intoopea-project:mainfrom

edlee123:chatqna_w_endpoints

Oct 29, 2025

Contributor

edlee123 commented Jun 24, 2025 •

edited

Loading

Description

Allows ChatQnA to be used with hundreds of OpenAI-like endpoints e.g. OpenRouter.ai, Hugging Face, Denvr, and improve the developer experience to use OPEA quickly even on low resource environments.

Key Changes Made:

Created ChatQnA/docker_compose/intel/cpu/xeon/README_endpoint_openai.md: instructions to spin up example.
Created ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.yaml: replaces vLLM with an opeai-like endpoint

Also:

Fixed align_generator function to properly detect and skip chunks where content is null in open-ai like endpoints. Previously it'd show the null json in the UI.
Added better error handling and debug logging for easier troubleshooting of endpoint issues.

Issues

N/A

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

N/A

Tests

OpenRouter.ai: anthropic/claude-3.7-sonnet
Denvr: meta-llama/Llama-3.1-70B-Instruct
Hugging Face Inference Endpoint: microsoft/phi-4

edlee123 and others added 30 commits

June 24, 2025 18:08


          Compose file for ChatQnA example with openai-like endpoint

985cddd

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Adding README.md for ChatQnA + endpoint

eec42f2

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          In chatqna.py handle null openai api response since UI would show sho…

39962f6

…w null json. Also improved exception handling and logging

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Update ChatQnA/docker_compose/intel/cpu/xeon/compose_endpoint_openai.…

d227878

…yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Add tests for different input formats (opea-project#2006)

139972c

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Fix security issues in workflows (opea-project#1977)

9fc1235

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Integrate MultimodalQnA set_env to ut scripts. (opea-project#1965)

70db6c5

Integrate MultimodalQnA set_env to ut scripts.
Add README.md for UT scripts.

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Optimize benchmark scripts (opea-project#1949)

7ada28f

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Fix permissions error. (opea-project#2008)

8c80b08

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Build comps-base:ci for AgentQnA test (opea-project#2010)

a943345

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Stop CI test on rocm due to lack of test machine (opea-project#2017)

b63bdb3

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Fix workflow permission issues. (opea-project#2018)

2a8f3fb

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Refine the README, folder/file hierarchy and test file for FinanceAge…

34d40dc

…nt (opea-project#1996)

Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Add code owners. (opea-project#2022)

a0f7ea0

Signed-off-by: Yi Yao <yi.a.yao@intel.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Fix MultimodalQnA UT issues (opea-project#2011)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for AgentQnA. (opea-project#2023)

31cd99f

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for AudioQnA. (opea-project#2024)

9b089dd

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for AvatarChatbot and DBQnA. (opea-project#…

313f671

…2030)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for ChatQnA. (opea-project#2029)

39b53a2

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for CodeGen and CodeTrans (opea-project#2031)

d18bc9b

Signed-off-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          [DocSum] Aligned the output format (opea-project#1948)

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for DocIndexRetriever. (opea-project#2035)

229f2b1

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for EdgeCraftRag, FinanceAgent, GraphRAG an…

4d0b5c4

…d HybridRAG (opea-project#2037)

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for ProductivitySuite, RerankFinetuning, Se…

fdbb0bf

…archQnA and Translation (opea-project#2038)

update secrets token name for ProductivitySuite, RerankFinetuning, SearchQnA and Translation
Fix shellcheck issue

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for InstructionTuning, MultimodalQnA and Wo…

b27a6d3

…rkflowExecAgent (opea-project#2039)

update secrets token name for InstructionTuning, MultimodalQnA and WorkflowExecAgent
Fix shellcheck issue

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for DocSum. (opea-project#2036)

a797945

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          update secrets token name for VideoQnA and VisualQnA (opea-project#2040)

81a8841

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Fix shellcheck issues and update secrets TOKEN name (opea-project#2043)

d91de60

Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          add new feature for EC-RAG (opea-project#2013)

1e20459

Signed-off-by: Yongbozzz <yongbo.zhu@intel.com>
Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          [CodeGen] Aligned the output format and fixed acc benchmark issues. (o…

09ceb36

…pea-project#1981)

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>

edlee123 added 3 commits

July 22, 2025 13:10


          Merge branch 'main' into chatqna_w_endpoints

204702e


          Updated compose_endpoint_openai.yaml to use newer text-embeddings-inf…

…erence:cpu-1.7 from 1.5

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>


          Merge branch 'chatqna_w_endpoints' of github.com:edlee123/GenAIExampl…

f000373

…es into chatqna_w_endpoints

joshuayao added this to the v1.4 milestone

joshuayao added this to OPEA

joshuayao moved this to In progress in OPEA


          Merge branch 'main' into chatqna_w_endpoints

855409f

joshuayao requested a review from XinyuYe-Intel

August 20, 2025 03:44


          Merge branch 'main' into chatqna_w_endpoints

9a059d2

edlee123 changed the title ~~ChatQnA Example with OpenAI-Compatible Endpoint~~ Bump: ChatQnA Example with OpenAI-Compatible Endpoint

Collaborator

CICD-at-OPEA commented Sep 25, 2025

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

CICD-at-OPEA added the Stale label

emtl97 commented Sep 26, 2025

Appreciate a review pls.

letonghan approved these changes

View reviewed changes

CICD-at-OPEA removed the Stale label

Collaborator

CICD-at-OPEA commented Oct 26, 2025

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

CICD-at-OPEA added the Stale label

emtl97 commented Oct 27, 2025

Appreciate another approval pls. This works with a few open AI compatible endpoints not just vllm.

CICD-at-OPEA removed the Stale label

yao531441 approved these changes

View reviewed changes


          Merge branch 'main' into chatqna_w_endpoints

b6a2a55

Collaborator

joshuayao commented Oct 28, 2025 •

edited

Loading

Appreciate another approval pls. This works with a few open AI compatible endpoints not just vllm.

@emtl97 @edlee123 Thanks for your contribution. We will merge the PR if the CI tests complete successfully.

joshuayao modified the milestones: v1.4, v1.5

joshuayao merged commit 4f620f0 into opea-project:main

19 of 20 checks passed

github-project-automation bot moved this from In progress to Done in OPEA

Collaborator

joshuayao commented Oct 29, 2025

Appreciate another approval pls. This works with a few open AI compatible endpoints not just vllm.

@emtl97 @edlee123 Thanks for your contribution. We will merge the PR if the CI tests complete successfully.

@emtl97 @edlee123 Merged.

yao531441 added a commit that referenced this pull request


          Bump: ChatQnA Example with OpenAI-Compatible Endpoint (#2091)

65513e8

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Yi Yao <yi.a.yao@intel.com>
Signed-off-by: Yongbozzz <yongbo.zhu@intel.com>
Signed-off-by: CICD-at-OPEA <CICD@opea.dev>
Signed-off-by: Ezequiel Lanza <ezequiel.lanza@gmail.com>
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yi Yao <yi.a.yao@intel.com>
Co-authored-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Co-authored-by: Zhu Yongbo <yongbo.zhu@intel.com>
Co-authored-by: CICD-at-OPEA <cicd@opea.dev>
Co-authored-by: Eze Lanza (Eze) <40643766+ezelanza@users.noreply.github.com>
Co-authored-by: xiguiw <111278656+xiguiw@users.noreply.github.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Signed-off-by: Yao, Qing <qing.yao@intel.com>

cogniware-devops pushed a commit to Cogniware-Inc/GenAIExamples that referenced this pull request


          Bump: ChatQnA Example with OpenAI-Compatible Endpoint (opea-project#2091

939078c

)

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Yi Yao <yi.a.yao@intel.com>
Signed-off-by: Yongbozzz <yongbo.zhu@intel.com>
Signed-off-by: CICD-at-OPEA <CICD@opea.dev>
Signed-off-by: Ezequiel Lanza <ezequiel.lanza@gmail.com>
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yi Yao <yi.a.yao@intel.com>
Co-authored-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Co-authored-by: Zhu Yongbo <yongbo.zhu@intel.com>
Co-authored-by: CICD-at-OPEA <cicd@opea.dev>
Co-authored-by: Eze Lanza (Eze) <40643766+ezelanza@users.noreply.github.com>
Co-authored-by: xiguiw <111278656+xiguiw@users.noreply.github.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>

cogniware-devops pushed a commit to Cogniware-Inc/GenAIExamples that referenced this pull request


          Bump: ChatQnA Example with OpenAI-Compatible Endpoint (opea-project#2091

7f8b212

)

Signed-off-by: Ed Lee <16417837+edlee123@users.noreply.github.com>
Signed-off-by: ZePan110 <ze.pan@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Mustafa <mustafa.cetin@intel.com>
Signed-off-by: Yi Yao <yi.a.yao@intel.com>
Signed-off-by: Yongbozzz <yongbo.zhu@intel.com>
Signed-off-by: CICD-at-OPEA <CICD@opea.dev>
Signed-off-by: Ezequiel Lanza <ezequiel.lanza@gmail.com>
Signed-off-by: Wang, Xigui <xigui.wang@intel.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: ZePan110 <ze.pan@intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Mustafa <109312699+MSCetin37@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Yi Yao <yi.a.yao@intel.com>
Co-authored-by: Zhenzhong Xu <zhenzhong.xu@intel.com>
Co-authored-by: Zhu Yongbo <yongbo.zhu@intel.com>
Co-authored-by: CICD-at-OPEA <cicd@opea.dev>
Co-authored-by: Eze Lanza (Eze) <40643766+ezelanza@users.noreply.github.com>
Co-authored-by: xiguiw <111278656+xiguiw@users.noreply.github.com>
Co-authored-by: Yao Qing <Qing.Yao@intel.com>
Signed-off-by: cogniware-devops <ambarish.desai@cogniware.ai>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

yao531441 yao531441 approved these changes

letonghan letonghan approved these changes

lvliang-intel Awaiting requested review from lvliang-intel lvliang-intel is a code owner

XinyuYe-Intel Awaiting requested review from XinyuYe-Intel

Labels

None yet