-
Notifications
You must be signed in to change notification settings - Fork 529
Fix LiteLLM dual init issue #206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the global state management in agentlightning/llm_proxy.py to eliminate reliance on global variables and make dependencies explicit. The core change moves initialization logic from a global initialize() function into an instance method LLMProxy.initialize(), requiring explicit LightningStore injection rather than falling back to a global store.
Key changes:
- Removes global
_initializedflag and_global_store, replacing them with explicit dependency injection - Moves middleware and callback setup into
LLMProxy.initialize()instance method - Adds helpers to preserve original user middleware and LiteLLM callbacks for idempotent re-initialization
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
agentlightning/llm_proxy.py
Outdated
|
|
||
| Idempotent. Installs: | ||
| def _ensure_global_user_middlewares(app: FastAPI) -> List[Any]: | ||
| """Preserve a copy of `app.user_middlewares` before AGL's middlewares is installed.""" |
Copilot
AI
Oct 23, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected 'middlewares' to 'middleware' in the docstring.
| """Preserve a copy of `app.user_middlewares` before AGL's middlewares is installed.""" | |
| """Preserve a copy of `app.user_middleware` before AGL's middleware is installed.""" |
This pull request refactors how global state and middleware are managed in
agentlightning/llm_proxy.py, with a focus on improving initialization and dependency handling for the proxy, middleware, and telemetry. The changes are to prepare for the coming tinker custom LLM, making llm proxy restarting more robust and emit fewer warnings.Here's what's on my mind:
Now here's the most tricky part (test).