Skip to content

[Bugfix][Elastic EP] Clear scaling flag on failure to prevent permane… - #50778

Open
gagandhakrey wants to merge 2 commits into
vllm-project:mainfrom
gagandhakrey:fix/elastic-ep-scaling-flag-latch
Open

[Bugfix][Elastic EP] Clear scaling flag on failure to prevent permane…#50778
gagandhakrey wants to merge 2 commits into
vllm-project:mainfrom
gagandhakrey:fix/elastic-ep-scaling-flag-latch

Conversation

@gagandhakrey

Copy link
Copy Markdown
Contributor

Purpose

Issue

ScalingMiddleware returns 503 for all HTTP requests while the process-wide
_scaling_elastic_ep flag is set.

AsyncLLM._scale_elastic_ep() only cleared this flag on the success path. If
engine_core.commit_elastic_ep() raised an exception or the task was cancelled,
the flag remained permanently set, causing every endpoint—including
/scale_elastic_ep and /health—to return 503 until the process restarted.

Fix

Wrap the scaling operation in a single try/finally so
_scaling_elastic_ep is always cleared, regardless of success, exceptions, or
cancellation. Remove the redundant cleanup from
_drain_requests_for_elastic_ep(), leaving a single owner for the flag's
lifecycle.

Post-fix Impact

  • Prevents the scaling flag from becoming permanently latched.
  • Failed scaling operations are now retryable without restarting the server.
  • No behavior change to the successful scaling path.

…nt 503s

Signed-off-by: Gagan Dhakrey <gagandhakrey@gmail.com>
@gagandhakrey
gagandhakrey requested a review from njhill as a code owner August 2, 2026 22:31

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the bug Something isn't working label Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant