Repository navigation
fix(runtime): stop retrying dead directory members - #10261
Merged
ReubenBond merged 2 commits intoJul 14, 2026
Merged
ReubenBond merged 2 commits into
ReubenBond merged 2 commits into
Conversation
Use the latest cluster membership snapshot to abandon recovery calls to unavailable silos, and apply randomized exponential backoff to transient failures. Refresh normal directory calls immediately when their owner is unavailable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fecd969e-9e7a-4506-9a69-fca2cf39d594
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Improves Orleans distributed grain directory resiliency by avoiding retries against cluster members which are already known unavailable, reducing sustained message floods during directory recovery and snapshot transfer operations.
Changes:
- Use the latest cluster membership snapshot to stop invoking directory operations on unavailable silos and to refresh/reroute when owners are known unavailable.
- Replace fixed-interval retry loops with randomized exponential backoff (capped) for cluster-member invocations during partition operations.
- Add unit coverage for the “can invoke member” availability predicate across silo statuses.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryPartitionTests.cs | Adds tests validating which SiloStatus values are considered invokable for directory operations. |
| src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs | Updates cluster-member invocation retry logic to consult latest membership and use exponential backoff. |
| src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs | Exposes latest membership snapshot, refreshes/reroutes on known-unavailable owners, and centralizes member-invocation eligibility. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 1
Allow partition shutdown to interrupt membership refreshes in the cluster-member retry loop. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fecd969e-9e7a-4506-9a69-fca2cf39d594
This was referenced Jul 22, 2026
This was referenced Jul 29, 2026
This was referenced Jul 29, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Directory recovery could repeatedly send
RecoverRegisteredActivationsrequests to a silo which the messaging layer already knew was dead. The recovery loop consulted a lagging directory membership view and retried failures indefinitely at a fixed 100 ms interval, producing sustained message floods.Use the latest cluster membership snapshot before recovery and snapshot-transfer calls so unavailable targets are abandoned immediately. Transient failures retain retries, but now use randomized exponential backoff capped at five seconds. Regular directory calls also refresh and reroute immediately when their current owner is known unavailable.
Microsoft Reviewers: Open in CodeFlow