Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The membership system provides a **canonical membership view**: a versioned view

`MembershipTableManager` publishes snapshots through `ClusterMembershipService` with monotonically advancing canonical membership views. Directory ownership, gateway discovery, and failure recovery rely on this guarantee.

Per-silo `IAmAliveTime` is tracked independently of the canonical membership view. Periodic <xref:Orleans.IMembershipTable.UpdateIAmAlive*> writes leave the view version unchanged. Snapshot updates retain the maximum observed timestamp for each silo, so local liveness timestamps advance monotonically as table reads and peer snapshots arrive. At the same version, merging retains the accepted versioned fields.
Per-silo `IAmAliveTime` is tracked independently of the canonical membership view. Periodic <xref:Orleans.IMembershipTable.UpdateIAmAliveAsync*> writes leave the view version unchanged. Snapshot updates retain the maximum observed timestamp for each silo, so local liveness timestamps advance monotonically as table reads and peer snapshots arrive. At the same version, merging retains the accepted versioned fields.

Snapshots can prune previously `Dead` rows at the same version while retaining every non-Dead row. Pruning preserves the versioned fields and maximum `IAmAliveTime` of each retained entry.

Expand Down Expand Up @@ -101,15 +101,17 @@ These values are protocol parameters, not independent timers: indirect probing,

## Membership-table contract <a name="membership-table"></a>

An <xref:Orleans.IMembershipTable> implementation is more than a list of endpoints. It must support:
An <xref:Orleans.IMembershipTable> implementation coordinates canonical membership changes through:

- insertion of a new silo row;
- optimistic, conditional update of a silo row;
- atomic advancement of the table version with a row mutation;
- reads which return rows and the corresponding version;
- atomic full-table snapshots containing rows and their corresponding version;
- periodic `IAmAlive` updates; and
- durable availability appropriate for cluster coordination.

<xref:Orleans.IMembershipTable.ReadAllAsync*> obtains a coherent snapshot, and <xref:Orleans.MembershipTableData.TryGet*> selects a silo's entry and row ETag within that view. Conditional mutations return their success status. Snapshot refreshes provide the current table state and ETag for subsequent updates.

Table unavailability favors safety over liveness. Existing silos can continue processing calls, but they cannot durably admit a member or declare a failed member dead. A provider must not synthesize successful updates when its backing store is unavailable.

Official providers adapt transactions, ETags, lightweight transactions, or compare-and-swap primitives to this contract. Provider selection and operational setup belong in the [deployment documentation](../deployment/index.md); the extension architecture is covered by [provider authoring](provider-authoring.md).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,16 +73,22 @@ Ownership must be explicit. If the application supplies a keyed SDK client, the

The persistent stream provider illustrates staged lifecycle composition: it creates the adapter during initialization, starts pulling agents at the active stage, then stops agents before closing. See <xref:Orleans.Providers.Streams.Common.PersistentStreamProvider.Participate*?displayProperty=nameWithType> and its [implementation](https://github.com/dotnet/orleans/blob/main/src/Orleans.Streaming/PersistentStreams/PersistentStreamProvider.cs).

Membership callers use the `Async`-suffixed, cancellation-aware methods of <xref:Orleans.IMembershipTable> for initialization, reads, and writes. Built-in providers forward tokens to backend APIs which support cancellation and bound waits on tokenless SDK operations while observing late faults. Custom providers can implement these methods directly; their default implementations adapt existing tokenless providers by canceling the caller's wait while the operation completes. The original tokenless method names remain as obsolete compatibility entry points. Cancellation can race with a committed write, so conditional writes and table versions continue to govern subsequent updates.
Membership callers use the cancellation-aware methods of <xref:Orleans.IMembershipTable> for initialization, reads, and writes. Built-in providers forward tokens to backend APIs which support cancellation and bound waits on tokenless SDK operations while observing late faults. Custom providers implement cancellation according to their backend's operation and resource lifetimes. Cancellation can race with a committed write, so conditional writes and table versions govern subsequent updates.

Return completed backend results, and observe cancellation before starting further I/O. Mapping an already-returned result preserves that operation's outcome.

Shared membership refreshes live until the membership manager is disposed. Each caller owns its wait, while periodic maintenance and its queued cleanup requests stop with the silo lifecycle. This keeps membership reads available during shutdown.

Membership table RPCs retain their existing operation aliases and application-argument payloads, with cancellation propagated separately. The original generated request types remain available for calls through obsolete tokenless methods. During rolling upgrades, each receiver uses its implementation's cancellation behavior.
Membership table RPCs propagate cancellation separately from application arguments. Each receiver applies its implementation's cancellation behavior.

When a lifecycle callback must execute its cancellation or cleanup logic, schedule it with <xref:System.Threading.Tasks.Task.Run*> and pass the cancellation token to the operation inside the callback. The callback then owns how cancellation completes its work.

## Membership snapshots

Use <xref:Orleans.IMembershipTable.ReadAllAsync*> to obtain the membership rows and their corresponding table version in one atomic snapshot. <xref:Orleans.MembershipTableData.TryGet*> selects a silo's entry and row ETag from that snapshot.

Conditional writes use <xref:Orleans.TableVersion.Next*> from the observed snapshot and return their success status. Refresh the snapshot to obtain the current table ETag before selecting another mutation. Row ETags follow each provider's metadata policy and can change during heartbeat-only activity. Canonical updates require an existing row and a matching table ETag; additional row conditions remain heartbeat-neutral.

## Testing a provider

Contract tests should cover more than successful round trips:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -171,38 +171,14 @@ await Task.WhenAll(toDelete.BatchIEnumerable(MAX_BATCH_SIZE)
}
}

[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress siloAddress) => ReadRowAsync(siloAddress, CancellationToken.None);

public async Task<MembershipTableData> ReadRowAsync(SiloAddress siloAddress, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress siloAddress, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
try
{
var siloEntryKeys = new Dictionary<string, AttributeValue>
{
{ $"{SiloInstanceRecord.DEPLOYMENT_ID_PROPERTY_NAME}", new AttributeValue(this.clusterId) },
{ $"{SiloInstanceRecord.SILO_IDENTITY_PROPERTY_NAME}", new AttributeValue(SiloInstanceRecord.ConstructSiloIdentity(siloAddress)) }
};

var versionEntryKeys = new Dictionary<string, AttributeValue>
{
{ $"{SiloInstanceRecord.DEPLOYMENT_ID_PROPERTY_NAME}", new AttributeValue(this.clusterId) },
{ $"{SiloInstanceRecord.SILO_IDENTITY_PROPERTY_NAME}", new AttributeValue(SiloInstanceRecord.TABLE_VERSION_ROW) }
};

var entries = await storage.GetEntriesTxAsync(this.options.TableName,
new[] { siloEntryKeys, versionEntryKeys }, ParseRecord, cancellationToken);

MembershipTableData data = Convert(entries.ToList());
LogTraceReadMyEntry(siloAddress, data);
return data;
}
catch (Exception exc)
{
LogWarningIntermediateErrorReadingSiloEntry(exc, siloAddress, this.options.TableName);
throw;
}
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

[Obsolete("Use ReadAllAsync instead.")]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,23 +70,15 @@ public async Task InitializeMembershipTableAsync(bool tryInitTableVersion, Cance
}

/// <inheritdoc />
[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress key) => ReadRowAsync(key, CancellationToken.None);

/// <inheritdoc />
public async Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
LogTraceReadRow(key);
try
{
return await orleansQueries.MembershipReadRowAsync(this.clusterId, key, cancellationToken);
}
catch (Exception ex)
{
LogDebugReadRowFailed(ex);
throw;
}
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

/// <inheritdoc />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,24 +78,14 @@ public Task CleanupDefunctSiloEntriesAsync(DateTimeOffset beforeDate, Cancellati
return tableManager.CleanupDefunctSiloEntries(beforeDate, cancellationToken);
}

[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress key) => ReadRowAsync(key, CancellationToken.None);

public async Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
try
{
var entries = await tableManager.FindSiloEntryAndTableVersionRow(key, cancellationToken);
MembershipTableData data = Convert(entries);
LogDebugReadMyEntry(key, data);
return data;
}
catch (Exception exc)
{
LogWarningIntermediateErrorReadingSiloEntry(exc, key, tableManager.TableName);
throw;
}
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

[Obsolete("Use ReadAllAsync instead.")]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -174,25 +174,14 @@ await _container.DeleteItemAsync<SiloEntity>(
}
}

[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress key) => ReadRowAsync(key, CancellationToken.None);

public async Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
var id = ConstructSiloEntityId(key);

try
{
var snapshot = await ReadMembershipSnapshot(id, cancellationToken).ConfigureAwait(false);
return ConvertToTableData(snapshot.Silos, snapshot.Version);
}
catch (Exception exc) when (exc is not OperationCanceledException)
{
LogWarningFailureReadingSiloEntry(exc, key, _clusterId);
WrappedException.CreateAndRethrow(exc);
throw;
}
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

[Obsolete("Use ReadAllAsync instead.")]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -258,12 +258,14 @@ public async Task<MembershipTableData> ReadAllAsync(CancellationToken cancellati
return await ReadConsistentAsync(null, cancellationToken);
}

[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
Task<MembershipTableData> IMembershipTable.ReadRow(SiloAddress key) => ReadRowAsync(key, CancellationToken.None);

public async Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
{
return await ReadConsistentAsync(key, cancellationToken);
cancellationToken.ThrowIfCancellationRequested();
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

private async Task<MembershipTableData> ReadConsistentAsync(SiloAddress? key, CancellationToken cancellationToken)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -120,42 +120,14 @@ await this._storage.ExecuteTransaction(async transaction =>
}
}

[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress key) => ReadRowAsync(key, CancellationToken.None);

public async Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress key, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
try
{
var collection = this._storage.GetCollection();
var data = await this._storage.ExecuteTransaction(async transaction =>
{
// The transaction binds the version and row to one snapshot across the streamed batch response.
var snapshots = await transaction.GetAllSnapshotsAsync(
[collection.Document(this._partitionId), collection.Document(key.ToParsableString())],
transaction.CancellationToken);
var versionSnapshot = snapshots[0];
var siloSnapshot = snapshots[1];
if (!versionSnapshot.Exists)
throw new KeyNotFoundException($"Could not find cluster version entry for {this._partitionId}");

var silos = siloSnapshot.Exists
? new[] { siloSnapshot.ConvertTo<SiloInstanceEntity>() }
: Array.Empty<SiloInstanceEntity>();
return (silos, versionSnapshot.ConvertTo<ClusterVersionEntity>());
}, cancellationToken);
var table = Convert(data);

LogReadEntry(key, table);

return table;
}
catch (Exception exc) when (exc is not OperationCanceledException)
{
LogReadEntryError(exc, key);
throw;
}
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

[Obsolete("Use ReadAllAsync instead.")]
Expand Down
9 changes: 4 additions & 5 deletions src/Orleans.Clustering.Consul/ConsulBasedMembershipTable.cs
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,15 @@ public async Task InitializeMembershipTableAsync(bool tryInitTableVersion, Cance
}

/// <inheritdoc />
[Obsolete("Use ReadRowAsync instead.")]
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRow(SiloAddress siloAddress) => ReadRowAsync(siloAddress, CancellationToken.None);

/// <inheritdoc />
public async Task<MembershipTableData> ReadRowAsync(SiloAddress siloAddress, CancellationToken cancellationToken = default)
[Obsolete("Use ReadAllAsync and MembershipTableData.TryGet instead.")]
public Task<MembershipTableData> ReadRowAsync(SiloAddress siloAddress, CancellationToken cancellationToken = default)
{
cancellationToken.ThrowIfCancellationRequested();
var (siloRegistration, tableVersion) = await GetConsulSiloRegistration(siloAddress, cancellationToken);

return AssembleMembershipTableData(tableVersion, siloRegistration);
return Task.FromException<MembershipTableData>(new NotSupportedException("Use ReadAllAsync and MembershipTableData.TryGet instead."));
}

/// <inheritdoc />
Expand Down
7 changes: 6 additions & 1 deletion src/Orleans.Clustering.TestKit/MembershipTableModel.cs
Original file line number Diff line number Diff line change
Expand Up @@ -369,8 +369,13 @@ internal async Task<MembershipModelResult> ExecuteAsync(MembershipRequest reques
break;
case MembershipOperationKind.ReadPresentRow:
case MembershipOperationKind.ReadAbsentRow:
// Retain generated operation identities while selecting from one full snapshot.
var snapshot = await writer.ReadAllAsync(_ct);
var selected = snapshot.TryGet(input.SiloAddress);
MembershipTableTestRunner.Equal(before.Select(input.SiloAddress),
ClusteringMembershipSnapshot.Capture(await writer.ReadRowAsync(input.SiloAddress, _ct)));
ClusteringMembershipSnapshot.Capture(selected is null
? new MembershipTableData(snapshot.Version)
: new MembershipTableData(selected, snapshot.Version)));
break;
case MembershipOperationKind.InsertNew:
case MembershipOperationKind.StartSuccessor:
Expand Down
Loading
Loading