Skip to content

fix(streaming): avoid LeaseBasedQueueBalancer timer disposal race during shutdown - #10286

Merged
ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-queue-balancer-timer-disposal
Jul 21, 2026
Merged

ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-queue-balancer-timer-disposal

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jul 20, 2026 •

Copy link
Copy Markdown
Member

Fixes #10280.\n\nShutdown now cancels and joins the lease maintenance loops before draining queued responsibility updates and disposing their timers. The final executor operation still releases owned leases, and its failures are observed.\n\nA deterministic regression test blocks lease acquisition while shutdown begins, verifies that shutdown waits for queued work, and confirms that the acquired lease is released.

Microsoft Reviewers: Open in CodeFlow

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a shutdown ordering race in LeaseBasedQueueBalancer where queued membership/responsibility updates could run after PeriodicTimer instances were disposed, leading to ObjectDisposedException during shutdown.

Changes:

  • Reorders shutdown to cancel/join background loops and drain queued executor work before disposing lease timers.
  • Adds a cancellation-aware WaitForNextTick helper for the lease maintenance/acquisition loops.
  • Adds a regression test which blocks lease acquisition during shutdown and verifies shutdown waits and releases the acquired lease.
Show a summary per file
File Description
src/Orleans.Streaming/QueueBalancer/LeaseBasedQueueBalancer.cs Reorders shutdown to avoid timer disposal races; adds cancellation-aware timer tick helper; ensures lease release runs during shutdown.
test/Orleans.Streaming.Tests/OrleansRuntime/Streams/LeaseBasedQueueBalancerTests.cs Adds a deterministic test to validate shutdown drains queued work and releases leases during shutdown.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 3

@ReubenBond ReubenBond changed the title Fix LeaseBasedQueueBalancer timer disposal race fix(streaming): avoid LeaseBasedQueueBalancer timer disposal race during shutdown Jul 20, 2026
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-queue-balancer-timer-disposal branch from 4a20194 to ea953db Compare July 21, 2026 05:21
ReubenBond and others added 3 commits July 20, 2026 22:27
Cancel timer waits and drain queued responsibility updates before disposing the lease balancer timers. Preserve final lease release during shutdown and cover the race with a deterministic regression test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
…ng fails

Wrap the lease-drain/release step in Shutdown() with try/finally so
_leaseMaintenanceTimer and _leaseAcquisitionTimer are always disposed,
even if the final executor drain or ReleaseLeasesToMeetResponsibility
throws (e.g. a transient failure from ILeaseProvider.Release). Without
this, a failure during shutdown would leak both PeriodicTimer instances
and propagate the exception before cleanup completed.

$Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
…cer shutdown test

Replace NullLoggerFactory with a log-recording ILoggerFactory and
TimeProvider.System with FakeTimeProvider so the regression test can
deterministically assert that shutdown does not log an
ObjectDisposedException (or any error), which was the original
symptom reported in dotnet#10280.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-queue-balancer-timer-disposal branch from ea953db to 042ce50 Compare July 21, 2026 05:27
@ReubenBond
ReubenBond merged commit dce1796 into dotnet:main Jul 21, 2026
61 of 62 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-fix-queue-balancer-timer-disposal branch July 21, 2026 13:02
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix LeaseBasedQueueBalancer timer disposal race during shutdown

2 participants