Skip to content

fix(membership): reevaluate stale peers while joining - #11431

Merged
ReubenBond merged 6 commits into
dotnet:mainfrom
ReubenBond:rb-11417-fix-joining-stale-active-member
Oct 8, 2026
Merged

ReubenBond merged 6 commits into
dotnet:mainfrom
ReubenBond:rb-11417-fix-joining-stale-active-member

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

A joining silo monitors stale and suspected peers, but previously recalculated that set only when a membership snapshot arrived. An unreachable Active peer with a fresh heartbeat could become stale during the same join attempt while the table version and heartbeat values remained unchanged, leaving connectivity validation waiting for a peer which health monitoring had never selected.

Reevaluate monitoring candidates at the configured TableRefreshTimeout interval. The membership-processing loop serializes timer ticks with ordered snapshot updates, and snapshot-based eligibility governs monitoring for both Joining and Active silos. Existing per-peer monitors and their probe counters are reused. Selected peers pass through the usual probe-failure threshold, connection-liveness checks, suspicion votes, quorum calculation, and optimistic membership writes. Connectivity validation then observes the resulting membership change and completes normally.

Fake-time regressions cover an unchanged snapshot crossing the stale threshold, successful probes of stale peers, active-silo reevaluation preserving existing monitors, recovery through normal death voting in a single-silo restart, quorum preservation with healthy peers, startup cancellation, and membership-update failure propagation. Configuration validation requires a finite positive refresh interval while preserving sub-millisecond and long intervals. The cluster membership documentation describes periodic reevaluation and startup recovery.

Fixes #11417.

Microsoft Reviewers: Open in CodeFlow

Copilot AI balanced review requested due to automatic review settings October 8, 2026 01:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

PeriodicTimer rejects some previously accepted TableRefreshTimeout values, causing startup failures.

1 open finding
What changed in this PR

Reevaluates stale peers during silo startup so joining silos can recover from unreachable members without new membership snapshots.

Changes:

  • Serializes periodic reevaluation with membership updates.
  • Adds fake-time regression coverage.
  • Documents startup recovery behavior.
File Description
ClusterHealthMonitor.cs Adds periodic stale-peer reevaluation.
ClusterHealthMonitorTests.cs Covers monitoring, voting, cancellation, and failures.
cluster-management.md Documents joining-silo recovery.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request
Lines 83.33% (118,109 / 141,728)
Branches 72.86% (35,062 / 48,124)

Report-only conclusion: current-main baseline stale.

The newest successful coverage run tested ad8cf7c, not current main ca862d8.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI balanced review requested due to automatic review settings October 8, 2026 03:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A zero refresh timeout causes the new joining loop to spin continuously.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 05:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The newly supported infinite refresh interval triggers the membership manager’s fatal-error path during startup.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 09:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Infinite timeout support remains incompatible with arithmetic consumers that can produce invalid negative timeouts.

2 open findings

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 15:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The concurrency-sensitive membership and failure-detection changes warrant final human validation despite comprehensive regression coverage.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 18:35
@ReubenBond
ReubenBond force-pushed the rb-11417-fix-joining-stale-active-member branch from 609fd6b to 7ca0a38 Compare October 8, 2026 18:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation addresses unchanged-snapshot staleness while preserving ordered processing, probe state, configuration safety, and broad regression coverage.

0 open findings

🧠 Review effort: Balanced

@ReubenBond
ReubenBond merged commit ab85a77 into dotnet:main Oct 8, 2026
91 checks passed
@ReubenBond
ReubenBond deleted the rb-11417-fix-joining-stale-active-member branch October 8, 2026 19:12
This was referenced Oct 29, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Joining silo can remain blocked when an unreachable Active member becomes stale without a membership update

2 participants