Add FAQGen gateway in core to support FAQGen Example, Adds FAQGen LLM service

[yogeshmpandey]

Description

Added components to support FAQGen Example.

• FAQGen Gateway
• FAQGen LLM Endpoint

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.

n/a

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Added tests to the tests folder.

[codecov]

Codecov Report

Attention: Patch coverage is 20.00000% with 16 lines in your changes missing coverage. Please review.

Files	Coverage Δ
comps/cores/mega/constants.py	98.14% <100.00%> (+0.03%)	⬆️
comps/cores/mega/gateway.py	25.49% <15.78%> (-1.00%)	⬇️

... and 1 file with indirect coverage changes

[Spycsh]

Hi @yogeshmpandey , thanks for contribution. This microservice works perfectly. Please check the comments in the gateway.py and make sure the example also works with the latest code.

[ftian1]

I am thinking if it's needed to introduce a new FAQGen type under comps/llm folder. the most of code is same with the one in comps/llm/summarization, just prompt_template is different, right? maybe a more appropriate way is to have a unique code with an extra template folder in which it contains many different templates to allow user to setup when starting the LLM docker.

[yogeshmpandey]

I am thinking if it's needed to introduce a new FAQGen type under comps/llm folder. the most of code is same with the one in comps/llm/summarization, just prompt_template is different, right? maybe a more appropriate way is to have a unique code with an extra template folder in which it contains many different templates to allow user to setup when starting the LLM docker.

Hi @ftian1

I did not want to introduce any breaking change to other components hence added a new LLM endpoint,

I agree with this approach, the user can choose the specific prompt template defined by the environment variable. These templates can be read from a template file.

Option - 2
Another approach could be to allow users to provide a custom prompt. We can update the ChatCompletionRequest with an Optional custom_prompt field, enabling users to set their desired prompt. With this approach, the same backend can handle multiple tasks like summarization, FAQs, meeting summaries, etc. Additionally, this would provide an option for users to override the default custom prompt for summarization tasks.

[ftian1]

merge it at first. we will unify the whole LLM instances together in the next release.

PS：we would implement option 1 and 2 together. so either there is a default system prompt when the service is running, or user can override it during generation with user's query.