Repository navigation
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to improve Orleans Event Hubs streaming resiliency by allowing the provider to recover from stale/invalid persisted checkpoints by resetting/deleting the stored offset and resuming from a fresh position.
Changes:
- Adds a new
Reset()capability toIStreamQueueCheckpointer<TCheckpoint>(defaulting toNotSupportedException). - Implements
Reset()for the Event Hubs Azure Table checkpointer by deleting the persisted checkpoint row. - Attempts to trigger checkpoint reset when Event Hubs reads fail with an
ArgumentException.
Show a summary per file
| File | Description |
|---|---|
| src/Orleans.Streaming/PersistentStreams/IStreamQueueCheckpointer.cs | Adds optional Reset() operation to the streaming checkpointer abstraction. |
| src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/IEventHubReceiver.cs | Formatting-only change in receiver proxy constructor. |
| src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubCheckpointer.cs | Adds Event Hubs checkpoint delete/reset implementation. |
| src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs | Invokes checkpoint reset on read failures to recover from stale offsets. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
4580e7a to
b3a069f
Compare
There was a problem hiding this comment.
Review details
Suppressed comments (1)
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:244
- ResetReceiver() nulls
cachebefore nullingflowController. SinceTryPurgeFromCache()callsIsUnderPressure()(which readsflowController) and then dereferencesthis.cache!, there is a race window where another thread can see a non-nullflowController(returning > 0) but a nullcache, causing a NullReferenceException. ClearingflowControllerfirst ensuresIsUnderPressure()returns true during reset and avoids dereferencing a null cache.
var checkpointer = Interlocked.Exchange(ref this.checkpointer, null);
var receiver = Interlocked.Exchange(ref this.receiver, null);
var cache = Interlocked.Exchange(ref this.cache, null);
Interlocked.Exchange(ref this.flowController, null);
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Lite
b3f61e5 to
dcf3d75
Compare
There was a problem hiding this comment.
Review details
Suppressed comments (1)
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:556
LogWarningRetryingInitializationOfEventHubPartitionis usingOrleansEventHubErrorCode.FailedPartitionReadfor its EventId, butOrleansEventHubErrorCodedefines a dedicatedRetryReceiverInitvalue. Using the dedicated ID makes log queries/alerts unambiguous.
[LoggerMessage(
Level = LogLevel.Warning,
EventId = (int)OrleansEventHubErrorCode.FailedPartitionRead,
Message = "Retrying initialization of EventHub partition {EventHubName}-{Partition}."
)]
private partial void LogWarningRetryingInitializationOfEventHubPartition(string eventHubName, string partition);
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
Code coverage
Report-only conclusion: mixed. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
Review details
Suppressed comments (4)
Previously missed (2) — in code that hasn't changed since the last review.
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:498
- Cursor construction assumes owner.cache is non-null (using the null-forgiving operator). If GetCacheCursor is called after Shutdown (or during a reset/reinit window where cache is temporarily cleared), this can throw a NullReferenceException. Consider throwing ObjectDisposedException (or otherwise handling null) to make the failure mode deterministic and easier to diagnose.
This issue also appears on line 522 of the same file.
this.owner = owner;
this.streamId = streamId;
this.cache = owner.cache!;
this.cursor = this.cache.GetCursor(streamId, token);
}
src/Azure/Orleans.Streaming.EventHubs/Providers/EventDataGeneratorStreamProvider/NoOpCheckpointer.cs:10
- The public singleton Instance field is mutable (not readonly), so external code can reassign it, which is surprising for a singleton and makes behavior harder to reason about. Making it readonly prevents accidental reassignment.
This issue also appears on line 23 of the same file.
public static NoOpCheckpointerFactory Instance = new NoOpCheckpointerFactory();
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:526
- Cursor.Refresh() rebinds to owner.cache using the null-forgiving operator. If the receiver has been shut down (owner.cache == null) or the cache is being recreated, this can throw NullReferenceException. It would be safer to explicitly handle a null cache (eg, throw ObjectDisposedException) before rebinding.
public void Refresh(StreamSequenceToken? token)
{
if (!ReferenceEquals(this.cache, this.owner.cache))
{
this.cache = this.owner.cache!;
src/Azure/Orleans.Streaming.EventHubs/Providers/EventDataGeneratorStreamProvider/NoOpCheckpointer.cs:23
- The public singleton Instance field is mutable (not readonly), so external code can reassign it. Making it readonly prevents accidental reassignment and better matches singleton intent.
public static NoOpCheckpointer Instance = new NoOpCheckpointer();
- Files reviewed: 15/15 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
5 open issues
The recovery gate relies onReferenceEqualswhen matchingearliestSubscriptionTokenagainst… · New LikeStreamQueueCheckpointer, thisReset(CancellationToken)queues reset work even if the token… · New The comment claims to preserve “exact legacy exception and cursor behavior”, but this new explicit… · NewQueueCacheCursorResultKind.NotSupportedis handled by thedefaultbranch and throws `"The… · NewReset(CancellationToken)mutates state and schedules work even whencancellationTokenis… · New
Review effort: Lite
| if (this.recoveredCursorProgress is { } recoveredProgress) | ||
| { | ||
| if (this.recoveryPendingCursors is { Count: > 0 } | ||
| || !recoveredProgress.Values.Any( | ||
| progress => ReferenceEquals(progress.DeliveredToken, earliestSubscriptionToken) | ||
| || ReferenceEquals(progress.ResumeToken, earliestSubscriptionToken))) | ||
| { | ||
| return; | ||
| } |
| public Task Reset(CancellationToken cancellationToken) | ||
| { | ||
| var completion = new TaskCompletionSource( | ||
| TaskCreationOptions.RunContinuationsAsynchronously); | ||
| lock (_lock) | ||
| { | ||
| _pendingResetCount++; | ||
| _throttleSavesUntilUtc = DateTime.MaxValue; | ||
| var inProgressSave = _inProgressSave; | ||
| _inProgressSave = completion.Task; | ||
| RunReset( | ||
| completion, | ||
| inProgressSave, | ||
| _inProgressSave, | ||
| _latestCheckpoint, | ||
| _updateGeneration, | ||
| cancellationToken).Ignore(); | ||
| completion.Task.Ignore(); | ||
| } | ||
|
|
||
| return completion.Task.WaitAsync(cancellationToken); | ||
| } |
| lock (this.cacheLock) | ||
| { | ||
| var cache = this.cache | ||
| ?? throw new InvalidOperationException("The Event Hub receiver is recovering."); | ||
| #pragma warning disable CS0618 // Preserve the exact legacy exception and cursor behavior. | ||
| var cursor = cache.GetCursor(streamId, token); | ||
| #pragma warning restore CS0618 | ||
| return new Cursor( | ||
| this, | ||
| streamId, | ||
| cache, | ||
| cursor, | ||
| ReferenceEquals(cache, this.recoveryCache) ? token : null); | ||
| } |
| var result = cache.TryGetCursor(this.streamId, token); | ||
| switch (result.Kind) | ||
| { | ||
| case QueueCacheCursorResultKind.Success: | ||
| this.owner.ClearRecoveredDeliveryToken(this); | ||
| this.cache = cache; | ||
| this.cursor = result.Cursor; | ||
| this.current = null; | ||
| this.recoveryResumeToken = | ||
| ReferenceEquals(cache, this.owner.recoveryCache) ? token : null; | ||
| this.pendingMoveResult = null; | ||
| break; | ||
| case QueueCacheCursorResultKind.CacheMiss: | ||
| this.owner.ClearRecoveredDeliveryToken(this); | ||
| this.cache = cache; | ||
| this.cursor = null; | ||
| this.current = null; | ||
| this.recoveryResumeToken = null; | ||
| this.pendingMoveResult = QueueCacheCursorMoveResult.FromCacheMiss(result.CacheMiss!.Value); | ||
| break; | ||
| default: | ||
| throw new InvalidOperationException("The cursor result is not initialized."); | ||
| } |
| public Task Reset(CancellationToken cancellationToken) | ||
| { | ||
| var completion = new TaskCompletionSource( | ||
| TaskCreationOptions.RunContinuationsAsynchronously); | ||
| lock (_lock) | ||
| { | ||
| _pendingResetCount++; | ||
| _throttleSavesUntilUtc = DateTime.MaxValue; | ||
| var inProgressSave = _inProgressSave; | ||
| _inProgressSave = completion.Task; | ||
| RunReset( | ||
| completion, | ||
| inProgressSave, | ||
| _inProgressSave, | ||
| _latestCheckpoint, | ||
| _updateGeneration, | ||
| cancellationToken).Ignore(); | ||
| completion.Task.Ignore(); | ||
| } | ||
|
|
||
| return completion.Task.WaitAsync(cancellationToken); | ||
| } |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical lifecycle and cursor-progress findings, plus moderate cleanup and contention issues, require changes before approval.
7 open issues
Reject stale initialization after timed-out shutdown · New Allow idle recovered cursors to advance checkpoints · New The recovery gate relies onReferenceEqualswhen matchingearliestSubscriptionTokenagainst…Reset(CancellationToken)mutates state and schedules work even whencancellationTokenis…QueueCacheCursorResultKind.NotSupportedis handled by thedefaultbranch and throws `"The… The comment claims to preserve “exact legacy exception and cursor behavior”, but this new explicit… LikeStreamQueueCheckpointer, thisReset(CancellationToken)queues reset work even if the token…
Review effort: Lite
| { | ||
| lock (this.cacheLock) | ||
| { | ||
| this.receiverState = ReceiverRunning; |
| this.recoveredCursorProgress = new Dictionary<Cursor, RecoveredCursorProgress>(ReferenceEqualityComparer.Instance); | ||
| this.recoveryPendingCursors = new HashSet<Cursor>(this.cursors, ReferenceEqualityComparer.Instance); |
There was a problem hiding this comment.
🟡 Changes recommended
Two critical active issues remain: failed shutdowns can allow overlapping receiver lifetimes and checkpoint races, while the offset-error classifier is too broad and contradicts the added test’s expected behavior.
8 open issues
Match the documented service offset error format · New Allow idle recovered cursors to advance checkpoints Reject stale initialization after timed-out shutdown The recovery gate relies onReferenceEqualswhen matchingearliestSubscriptionTokenagainst…Reset(CancellationToken)mutates state and schedules work even whencancellationTokenis…QueueCacheCursorResultKind.NotSupportedis handled by thedefaultbranch and throws `"The… The comment claims to preserve “exact legacy exception and cursor behavior”, but this new explicit… LikeStreamQueueCheckpointer, thisReset(CancellationToken)queues reset work even if the token…
Review effort: Lite
| => exception is ArgumentException | ||
| && exception.Message.StartsWith("The supplied offset", StringComparison.OrdinalIgnoreCase) | ||
| && exception.Message.Contains(" is invalid.", StringComparison.OrdinalIgnoreCase); |
There was a problem hiding this comment.
🟡 Changes recommended
Approval is blocked by multiple active critical correctness and lifecycle issues in Event Hubs recovery, checkpoint progression, cache lifetime handling, and formatting.
11 open issues
Add the formatter-required blank line before the class · New Keep the old cache alive until returned batches are admitted or discarded · New Complete recovery when there are no cursors to track · New Match the documented service offset error format Allow idle recovered cursors to advance checkpoints Reject stale initialization after timed-out shutdown The recovery gate relies onReferenceEqualswhen matchingearliestSubscriptionTokenagainst…Reset(CancellationToken)mutates state and schedules work even whencancellationTokenis…QueueCacheCursorResultKind.NotSupportedis handled by thedefaultbranch and throws `"The… The comment claims to preserve “exact legacy exception and cursor behavior”, but this new explicit… LikeStreamQueueCheckpointer, thisReset(CancellationToken)queues reset work even if the token…
Review effort: Lite
| } | ||
| /// <summary> |
| List<StreamPosition> messageStreamPositions; | ||
| lock (this.cacheLock) | ||
| { | ||
| if (!ReferenceEquals(receiver, this.receiver) | ||
| || !ReferenceEquals(cache, this.cache) | ||
| || cache is null) | ||
| { | ||
| return batches; | ||
| } | ||
|
|
||
| messageStreamPositions = cache.Add(messages, dequeueTimeUtc); | ||
| } |
| if (this.recoveryPendingCursors is { Count: > 0 } | ||
| || !recoveredProgress.Values.Any( | ||
| progress => ReferenceEquals(progress.DeliveredToken, earliestSubscriptionToken) | ||
| || ReferenceEquals(progress.ResumeToken, earliestSubscriptionToken))) |


Problem
An Event Hubs pulling agent can retain a checkpoint whose offset is invalid after an Event Hub is recreated or its partition position changes. Repeated reads at that offset leave the partition stalled until checkpoint storage is cleared. The adapter factory also reuses receivers across pulling-agent lifetimes, so each initialization needs a complete shutdown of its resources.
Solution
ArgumentExceptionservice message when the receiver started from a persisted checkpoint, then reset the checkpoint and recreate the receiver and cache. Azure.Messaging.EventHubs 5.12.2 exposes this rejection through its service-message text; recognition depends on that format.EventHubReceiverOptions.StartFromNow: the default starts at the partition tail and skips retained backlog, whilefalsereplays from the earliest retained event.Rationale
Recovery belongs at the receiver/checkpointer boundary because that layer knows the persisted partition position, configured recovery policy, and resources owned by the current lifetime. Coordinating checkpoint storage, cache, and receiver lets the pulling agent resume while preserving delivery progress. Successful shutdown drains the current lifetime before ownership moves; operators choose the recovery position, and idempotent consumers handle possible replay from the earliest retained event.
Microsoft Reviewers: Open in CodeFlow