Repository navigation
fix(streaming): avoid LeaseBasedQueueBalancer timer disposal race during shutdown - #10286
Merged
ReubenBond merged 3 commits intoJul 21, 2026
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a shutdown ordering race in LeaseBasedQueueBalancer where queued membership/responsibility updates could run after PeriodicTimer instances were disposed, leading to ObjectDisposedException during shutdown.
Changes:
- Reorders shutdown to cancel/join background loops and drain queued executor work before disposing lease timers.
- Adds a cancellation-aware
WaitForNextTickhelper for the lease maintenance/acquisition loops. - Adds a regression test which blocks lease acquisition during shutdown and verifies shutdown waits and releases the acquired lease.
Show a summary per file
| File | Description |
|---|---|
src/Orleans.Streaming/QueueBalancer/LeaseBasedQueueBalancer.cs |
Reorders shutdown to avoid timer disposal races; adds cancellation-aware timer tick helper; ensures lease release runs during shutdown. |
test/Orleans.Streaming.Tests/OrleansRuntime/Streams/LeaseBasedQueueBalancerTests.cs |
Adds a deterministic test to validate shutdown drains queued work and releases leases during shutdown. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 3
ReubenBond
force-pushed
the
reubenbond-fix-queue-balancer-timer-disposal
branch
from
July 21, 2026 05:21
4a20194 to
ea953db
Compare
Cancel timer waits and drain queued responsibility updates before disposing the lease balancer timers. Preserve final lease release during shutdown and cover the race with a deterministic regression test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
…ng fails Wrap the lease-drain/release step in Shutdown() with try/finally so _leaseMaintenanceTimer and _leaseAcquisitionTimer are always disposed, even if the final executor drain or ReleaseLeasesToMeetResponsibility throws (e.g. a transient failure from ILeaseProvider.Release). Without this, a failure during shutdown would leak both PeriodicTimer instances and propagate the exception before cleanup completed. $Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
…cer shutdown test Replace NullLoggerFactory with a log-recording ILoggerFactory and TimeProvider.System with FakeTimeProvider so the regression test can deterministically assert that shutdown does not log an ObjectDisposedException (or any error), which was the original symptom reported in dotnet#10280. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d66a7a6a-81e4-4714-8f99-709484361438
ReubenBond
force-pushed
the
reubenbond-fix-queue-balancer-timer-disposal
branch
from
July 21, 2026 05:27
ea953db to
042ce50
Compare
This was referenced Jul 22, 2026
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
Merged
This was referenced Aug 18, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #10280.\n\nShutdown now cancels and joins the lease maintenance loops before draining queued responsibility updates and disposing their timers. The final executor operation still releases owned leases, and its failures are observed.\n\nA deterministic regression test blocks lease acquisition while shutdown begins, verifies that shutdown waits for queued work, and confirms that the acquired lease is released.
Microsoft Reviewers: Open in CodeFlow