Support multiple LitAPIs for inference process and endpoints

[aniketmaurya]

What does this PR do?

Users will be able to run multiple APIs in a single server.

Part of #492

Example

from transformers import pipeline
import litserve as ls

class SentimentAnalysisAPI(ls.LitAPI):
    def setup(self, device):
        self.model = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model", device=device)

    def decode_request(self, request: dict):
        return request["text"]

    def predict(self, text):
        return self.model(text)[0]

class TextGenerationAPI(ls.LitAPI):
    def setup(self, device):
        self.generator = pipeline("text-generation", model="gpt2", device=device)

    def decode_request(self, request: dict):
        return request["prompt"]

    def predict(self, prompt):
        return self.generator(prompt)[0]["generated_text"]

if __name__ == "__main__":
    sentiment_api = SentimentAnalysisAPI(api_path="/classify-text")
    chat_api = TextGenerationAPI(api_path="/v1/chat/completions")

    server = ls.LitServer([sentiment_api, chat_api])
    server.run(port=8000)

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[ddiddi]

Thank you for your efforts! @aniketmaurya @bhimrazy

[Copilot]

Pull Request Overview

Supports running multiple LitAPI instances per server by introducing a connector class, refactoring how timeouts, logging, and endpoints are wired, and updating tests to match the new method signatures.

• Added _LitAPIConnector to centralize pre-setup, timeout checks, and logger queue assignment for one or more LitAPI instances.
• Refactored LitServer to delegate setup, worker launching, and endpoint registration through the connector, and updated queue/timeout handling.
• Updated tests in several modules to align with the new signatures (e.g., removing lit_spec args, changing launch_inference_worker parameters).

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_openai_embedding.py	Updated LitServer instantiation to pass spec into the API.
tests/test_loops.py	Removed deprecated lit_spec and stream parameters from loops.
tests/test_litapi.py	Swapped launch_inference_worker(1) for _init_manager + new arg.
tests/test_lit_server.py	Adjusted assertions to expect launch_inference_worker(server.lit_api).
tests/test_batch.py	Dropped redundant lit_api_mock from run_batched_loop call.
src/litserve/utils.py	Reworked wrap_litserve_start to use the new manager/queue setup.
src/litserve/specs/openai_embedding.py	Replaced self._server references with direct spec fields.
src/litserve/specs/openai.py	Unified stream vs. non-stream endpoint queue access via spec fields.
src/litserve/specs/base.py	Initialized response_buffer, request_queue, response_queue_id on specs.
src/litserve/server.py	Introduced _LitAPIConnector, overhauled pre-setup, worker launch, endpoint registration.
src/litserve/loops/streaming_loops.py	Adjusted loop signatures to remove external lit_spec arg.
src/litserve/loops/simple_loops.py	Moved lit_spec defaulting inside loop logic and removed stream arg.
src/litserve/loops/loops.py	Updated inference_worker to derive lit_spec and remove old args.
src/litserve/loggers.py	Changed logger setup to use litapi_connector.set_logger_queue.
src/litserve/api.py	Added default pass in setup, adjusted spec type hints.

Comments suppressed due to low confidence (2)

tests/test_lit_server.py:249

• [nitpick] There’s no test covering the multi-API scenario. Add a test that initializes LitServer with a list of LitAPI instances to verify all are launched correctly.

server.launch_inference_worker.assert_called_with(server.lit_api)

src/litserve/loops/streaming_loops.py:39

• [nitpick] The lit_spec parameter is positioned at the end here but appeared earlier in other loop signatures. For consistency, align parameter ordering across loop methods.

def run_streaming_loop(self, lit_api: LitAPI, request_queue: Queue, transport: MessageTransport, callback_runner: CallbackRunner, lit_spec: Optional[LitSpec] = None):

[codecov]

Codecov Report

Attention: Patch coverage is 95.73171% with 7 lines in your changes missing coverage. Please review.

Project coverage is 86%. Comparing base (dbec61b) to head (630cffe).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #513   +/-   ##
===================================
  Coverage    86%    86%           
===================================
  Files        37     37           
  Lines      2418   2505   +87     
===================================
+ Hits       2080   2165   +85     
- Misses      338    340    +2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Borda]

TODOs:

• Detect duplicate endpoint paths
• Don't allow streaming with non-stream

Is it for this PR or next one?

[aniketmaurya]

there were in TODO in this PR itself @Borda! I just updated to resolve them.

[Copilot]

Pull Request Overview

This PR refactors the LitServe codebase to support running multiple LitAPI instances on a single server, improving how endpoints are registered and how inference workers are launched. Key changes include updating test cases to pass specs via the LitAPI constructor, centralizing API endpoint management using a new _LitAPIConnector, and removing redundant parameters in loop methods to rely directly on lit_api.spec.

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

File	Description
tests/test_openai_embedding.py	Updated LitServer instantiation to pass spec via LitAPI constructor.
tests/test_multiple_endpoints.py	Added tests for handling multiple endpoints with distinct api paths.
tests/test_loops.py	Adjusted loop function calls to remove deprecated spec parameters.
src/litserve/*	Refactored server, API, and loop components to support multiple LitAPIs and improve endpoint registration.

Comments suppressed due to low confidence (1)

src/litserve/api.py:92

• [nitpick] The introduction of the _api_path attribute along with the api_path property (which defers to spec.api_path when available) could lead to ambiguity. Consider consolidating the source of truth for the API path to avoid confusion during future maintenance.

self._api_path = api_path

[ywh-my]

Thanks for your hard work! I can not find the branch "endpoint-2" . Are you still developing it ？

[aniketmaurya]

hi @ywh-my, it has been merged into the main branch. You can install the latest version of LitServe to use this feature. Here is the docs for it - https://lightning.ai/docs/litserve/features/multiple-endpoints

[ywh-my]

Thank you !!!!