Repository navigation
fix(membership): reevaluate stale peers while joining - #11431
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
PeriodicTimer rejects some previously accepted TableRefreshTimeout values, causing startup failures.
1 open finding
What changed in this PR
Reevaluates stale peers during silo startup so joining silos can recover from unreachable members without new membership snapshots.
Changes:
- Serializes periodic reevaluation with membership updates.
- Adds fake-time regression coverage.
- Documents startup recovery behavior.
| File | Description |
|---|---|
ClusterHealthMonitor.cs |
Adds periodic stale-peer reevaluation. |
ClusterHealthMonitorTests.cs |
Covers monitoring, voting, cancellation, and failures. |
cluster-management.md |
Documents joining-silo recovery. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Code coverage
Report-only conclusion: current-main baseline stale. The newest successful coverage run tested ad8cf7c, not current main ca862d8. Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
🟡 Changes recommended
A zero refresh timeout causes the new joining loop to spin continuously.
1 open finding
1 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
There was a problem hiding this comment.
🟡 Changes recommended
The newly supported infinite refresh interval triggers the membership manager’s fatal-error path during startup.
1 open finding
1 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
There was a problem hiding this comment.
🟡 Changes recommended
Infinite timeout support remains incompatible with arithmetic consumers that can produce invalid negative timeouts.
2 open findings
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
There was a problem hiding this comment.
🔵 Needs a closer look
The concurrency-sensitive membership and failure-detection changes warrant final human validation despite comprehensive regression coverage.
0 open findings
2 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
609fd6b to
7ca0a38
Compare


A joining silo monitors stale and suspected peers, but previously recalculated that set only when a membership snapshot arrived. An unreachable Active peer with a fresh heartbeat could become stale during the same join attempt while the table version and heartbeat values remained unchanged, leaving connectivity validation waiting for a peer which health monitoring had never selected.
Reevaluate monitoring candidates at the configured
TableRefreshTimeoutinterval. The membership-processing loop serializes timer ticks with ordered snapshot updates, and snapshot-based eligibility governs monitoring for both Joining and Active silos. Existing per-peer monitors and their probe counters are reused. Selected peers pass through the usual probe-failure threshold, connection-liveness checks, suspicion votes, quorum calculation, and optimistic membership writes. Connectivity validation then observes the resulting membership change and completes normally.Fake-time regressions cover an unchanged snapshot crossing the stale threshold, successful probes of stale peers, active-silo reevaluation preserving existing monitors, recovery through normal death voting in a single-silo restart, quorum preservation with healthy peers, startup cancellation, and membership-update failure propagation. Configuration validation requires a finite positive refresh interval while preserving sub-millisecond and long intervals. The cluster membership documentation describes periodic reevaluation and startup recovery.
Fixes #11417.
Microsoft Reviewers: Open in CodeFlow