Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,13 @@ Compare the snapshot version with the required <xref:Orleans.Runtime.MembershipV

A starting silo writes its row, becomes `Joining`, and validates two-way connectivity with active members before becoming `Active`. This prevents a partitioned process from silently joining one side of a cluster.

The periodic `IAmAlive` value is not the peer heartbeat. It is a timestamp written to the membership row for diagnostics and startup disaster recovery. A sufficiently stale active row can be ignored during the joining connectivity check, allowing a cluster to recover after all processes were lost without cleanly declaring each other dead.
`IAmAliveTime` is a membership-row liveness timestamp used for diagnostics and startup disaster recovery. A joining silo monitors suspected and stale peers and reevaluates monitoring candidates at the <xref:Orleans.Configuration.ClusterMembershipOptions.TableRefreshTimeout?displayProperty=nameWithType> interval. It compares the latest snapshot's timestamps with current time, so a peer whose timestamp becomes stale during startup enters monitoring even when the membership view remains unchanged.

Those monitors apply the usual failed-probe threshold, connection-liveness checks, and death-vote protocol. Initial connectivity validation retries for up to <xref:Orleans.Configuration.ClusterMembershipOptions.MaxJoinAttemptTime?displayProperty=nameWithType>, completing when each active peer responds or membership records its departure. This allows a replacement silo to recover after an ungraceful process exit and become `Active` through the normal membership protocol.

## Failure detection and death votes <a name="the-membership-protocol"></a>

Active silos monitor peers selected from the membership view. `ClusterHealthMonitor` sends probes over silo-to-silo messaging, tracks consecutive failures, and can use indirect probes to distinguish a failed target from an unhealthy observer. A failed monitor writes a timestamped vote into the target's membership row.
Active silos monitor hash-ring-selected peers plus suspected and stale peers. Joining and active silos reevaluate that selection on membership updates and at the `TableRefreshTimeout` interval, preserving existing monitors and their probe counters. `ClusterHealthMonitor` sends probes over silo-to-silo messaging, tracks consecutive failures, and can use indirect probes to distinguish a failed target from an unhealthy observer. A failed monitor writes a timestamped vote into the target's membership row.

Each observer maintains a [Phi Accrual failure detector](https://paperhub.s3.amazonaws.com/f516fdfa940caa08c679d3946b273128.pdf) for each peer. The detector models successful direct-probe round-trip times and estimates the timeout at which the probability of a later response is sufficiently low. The timeout starts at <xref:Orleans.Configuration.ClusterMembershipOptions.ProbeTimeout?displayProperty=nameWithType> and adapts after enough observations. Failures are excluded because they only show that the response exceeded the current timeout, while indirect results are excluded because they measure a different observer's network path.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ public TimeSpan MaxProbeTimeout
/// <summary>
/// Gets or sets the period between fetching updates from the membership table.
/// </summary>
/// <remarks>
/// Use a positive interval for periodic membership refreshes and peer-monitoring reevaluation.
/// </remarks>
/// <value>The membership table is refreshed every 60 seconds by default.</value>
public TimeSpan TableRefreshTimeout { get; set; } = TimeSpan.FromSeconds(60);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,9 +95,13 @@ public void ValidateConfiguration()
failureDetectionTimeoutTicks = maxProbeCycleTime.Ticks * clusterMembershipOptions.NumMissedProbesLimit;
}

if (clusterMembershipOptions.TableRefreshTimeout <= TimeSpan.Zero)
{
throw new OrleansConfigurationException($"{nameof(ClusterMembershipOptions)}.{nameof(ClusterMembershipOptions.TableRefreshTimeout)} ({clusterMembershipOptions.TableRefreshTimeout}) must be greater than 0.");
}

var tableRefreshTimeoutTicks = clusterMembershipOptions.TableRefreshTimeout.Ticks;
if (tableRefreshTimeoutTicks > 0
&& tableRefreshTimeoutTicks > (TimeSpan.MaxValue.Ticks - failureDetectionTimeoutTicks) / 2)
if (tableRefreshTimeoutTicks > (TimeSpan.MaxValue.Ticks - failureDetectionTimeoutTicks) / 2)
{
throw new OrleansConfigurationException($"The failure detection timeout plus twice {nameof(ClusterMembershipOptions)}.{nameof(ClusterMembershipOptions.TableRefreshTimeout)} ({clusterMembershipOptions.TableRefreshTimeout}) must not exceed {TimeSpan.MaxValue}.");
}
Expand Down
71 changes: 54 additions & 17 deletions src/Orleans.Runtime/MembershipService/ClusterHealthMonitor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -86,30 +86,67 @@ private async Task ProcessMembershipUpdates()
try
{
LogDebugStartingToProcessMembershipUpdates(log);
await foreach (var tableSnapshot in this.membershipManager.MembershipUpdates.WithCancellation(this.shutdownCancellation.Token))
using var processingCancellation = CancellationTokenSource.CreateLinkedTokenSource(this.shutdownCancellation.Token);
var cancellationToken = processingCancellation.Token;
await using var updates = this.membershipManager.MembershipUpdates.GetAsyncEnumerator(cancellationToken);
var reevaluationPeriod = this.clusterMembershipOptions.CurrentValue.TableRefreshTimeout;
var membershipUpdate = updates.MoveNextAsync().AsTask();
var reevaluation = this.timeProvider.DelayAsync(reevaluationPeriod, cancellationToken);
Comment thread
ReubenBond marked this conversation as resolved.
Resolved
var tableSnapshot = this.membershipManager.CurrentSnapshot;
try
{
var utcNow = this.timeProvider.GetUtcNow().UtcDateTime;
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
await Task.WhenAny(membershipUpdate, reevaluation);
var isMembershipUpdate = membershipUpdate.IsCompleted;
if (reevaluation.IsCompleted)
{
await reevaluation;
reevaluation = this.timeProvider.DelayAsync(reevaluationPeriod, cancellationToken);
}

if (isMembershipUpdate)
{
if (!await membershipUpdate)
{
break;
}

var newMonitoredSilos = this.UpdateMonitoredSilos(tableSnapshot, this.monitoredSilos, utcNow);
tableSnapshot = updates.Current;
}

if (this.clusterMembershipOptions.CurrentValue.EvictWhenMaxJoinAttemptTimeExceeded)
{
await this.EvictStaleStateSilos(tableSnapshot, utcNow);
}
var utcNow = this.timeProvider.GetUtcNow().UtcDateTime;
var newMonitoredSilos = this.UpdateMonitoredSilos(tableSnapshot, this.monitoredSilos, utcNow);

foreach (var pair in this.monitoredSilos)
{
if (!newMonitoredSilos.ContainsKey(pair.Key))
if (isMembershipUpdate && this.clusterMembershipOptions.CurrentValue.EvictWhenMaxJoinAttemptTimeExceeded)
{
using var cancellation = new CancellationTokenSource(
this.clusterMembershipOptions.CurrentValue.MaxProbeTimeout,
this.timeProvider);
await pair.Value.StopAsync(cancellation.Token);
await this.EvictStaleStateSilos(tableSnapshot, utcNow);
}
}

this.monitoredSilos = newMonitoredSilos;
this.observedMembershipVersion = tableSnapshot.Version;
foreach (var pair in this.monitoredSilos)
{
if (!newMonitoredSilos.ContainsKey(pair.Key))
{
using var cancellation = new CancellationTokenSource(
this.clusterMembershipOptions.CurrentValue.MaxProbeTimeout,
this.timeProvider);
await pair.Value.StopAsync(cancellation.Token);
}
}

this.monitoredSilos = newMonitoredSilos;
this.observedMembershipVersion = tableSnapshot.Version;
if (isMembershipUpdate)
{
membershipUpdate = updates.MoveNextAsync().AsTask();
}
}
}
finally
{
processingCancellation.Cancel();
await Task.WhenAll(membershipUpdate, reevaluation).SuppressThrowing();
}
}
catch (OperationCanceledException) when (shutdownCancellation.IsCancellationRequested)
Expand Down
Loading
Loading