Skip to content

fix(runtime): stop retrying dead directory members - #10261

Merged
ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-directory-recovery-backoff
Jul 14, 2026
Merged

ReubenBond merged 2 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-directory-recovery-backoff

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jul 14, 2026 •

Copy link
Copy Markdown
Member

Directory recovery could repeatedly send RecoverRegisteredActivations requests to a silo which the messaging layer already knew was dead. The recovery loop consulted a lagging directory membership view and retried failures indefinitely at a fixed 100 ms interval, producing sustained message floods.

Use the latest cluster membership snapshot before recovery and snapshot-transfer calls so unavailable targets are abandoned immediately. Transient failures retain retries, but now use randomized exponential backoff capped at five seconds. Regular directory calls also refresh and reroute immediately when their current owner is known unavailable.

Microsoft Reviewers: Open in CodeFlow

Use the latest cluster membership snapshot to abandon recovery calls to unavailable silos, and apply randomized exponential backoff to transient failures. Refresh normal directory calls immediately when their owner is unavailable.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fecd969e-9e7a-4506-9a69-fca2cf39d594
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves Orleans distributed grain directory resiliency by avoiding retries against cluster members which are already known unavailable, reducing sustained message floods during directory recovery and snapshot transfer operations.

Changes:

  • Use the latest cluster membership snapshot to stop invoking directory operations on unavailable silos and to refresh/reroute when owners are known unavailable.
  • Replace fixed-interval retry loops with randomized exponential backoff (capped) for cluster-member invocations during partition operations.
  • Add unit coverage for the “can invoke member” availability predicate across silo statuses.
Show a summary per file
File Description
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryPartitionTests.cs Adds tests validating which SiloStatus values are considered invokable for directory operations.
src/Orleans.Runtime/GrainDirectory/GrainDirectoryPartition.cs Updates cluster-member invocation retry logic to consult latest membership and use exponential backoff.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs Exposes latest membership snapshot, refreshes/reroutes on known-unavailable owners, and centralizes member-invocation eligibility.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Allow partition shutdown to interrupt membership refreshes in the cluster-member retry loop.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fecd969e-9e7a-4506-9a69-fca2cf39d594
@ReubenBond
ReubenBond merged commit 5e18016 into dotnet:main Jul 14, 2026
62 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-fix-directory-recovery-backoff branch July 14, 2026 20:02
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants