Skip to content

fix(streaming): recover Event Hubs from stale checkpoints - #7464

Open
oising wants to merge 31 commits into
dotnet:mainfrom
oising:gh-7256-pulling-agent-checkpoint-outofbounds
Open

oising wants to merge 31 commits into
dotnet:mainfrom
oising:gh-7256-pulling-agent-checkpoint-outofbounds

Conversation

@oising

@oising oising commented Jan 5, 2022 •

Copy link
Copy Markdown
Contributor

Problem

An Event Hubs pulling agent can retain a checkpoint whose offset is invalid after an Event Hub is recreated or its partition position changes. Repeated reads at that offset leave the partition stalled until checkpoint storage is cleared. The adapter factory also reuses receivers across pulling-agent lifetimes, so each initialization needs a complete shutdown of its resources.

Solution

  • Add a cancellation-aware reset contract to stream queue checkpointers and implement it for the built-in Azure Table, grain-backed, DynamoDB, and no-op checkpointers.
  • Recognize the known invalid-offset ArgumentException service message when the receiver started from a persisted checkpoint, then reset the checkpoint and recreate the receiver and cache. Azure.Messaging.EventHubs 5.12.2 exposes this rejection through its service-message text; recognition depends on that format.
  • Resume according to EventHubReceiverOptions.StartFromNow: the default starts at the partition tail and skips retained backlog, while false replays from the earliest retained event.
  • Rebind active subscription cursors to the replacement cache and clear batches retained from the previous cache.
  • Serialize reset with checkpoint writes. Initial receiver creation and reads observe caller cancellation. Once recovery detaches a failed receiver, reset remains a completion barrier; caller cancellation ends its wait while recovery continues with independent cleanup and replacement-initialization timeouts.
  • Use receiver lifecycle state for shutdown completion. Shutdown drains admitted initialization and recovery before flushing and cleaning up, and subsequent initialization starts a fresh lifetime. Timed-out or failed shutdown remains retryable.

Rationale

Recovery belongs at the receiver/checkpointer boundary because that layer knows the persisted partition position, configured recovery policy, and resources owned by the current lifetime. Coordinating checkpoint storage, cache, and receiver lets the pulling agent resume while preserving delivery progress. Successful shutdown drains the current lifetime before ownership moves; operators choose the recovery position, and idempotent consumers handle possible replay from the earliest retained event.

Microsoft Reviewers: Open in CodeFlow

@ReubenBond ReubenBond added this to the 3.6.1 milestone Jan 5, 2022
@ReubenBond ReubenBond added the area-streaming Category for Orleans streaming issues label Jan 5, 2022
@ReubenBond ReubenBond modified the milestones: 3.6.1, Backlog Mar 30, 2022
@ReubenBond ReubenBond changed the title Address #7256 pulling agent checkpoint out of bounds fix(streaming): recover Event Hubs from stale checkpoints Aug 21, 2026
@ReubenBond
ReubenBond marked this pull request as ready for review August 21, 2026 22:32
Copilot AI lite review requested due to automatic review settings August 21, 2026 22:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve Orleans Event Hubs streaming resiliency by allowing the provider to recover from stale/invalid persisted checkpoints by resetting/deleting the stored offset and resuming from a fresh position.

Changes:

  • Adds a new Reset() capability to IStreamQueueCheckpointer<TCheckpoint> (defaulting to NotSupportedException).
  • Implements Reset() for the Event Hubs Azure Table checkpointer by deleting the persisted checkpoint row.
  • Attempts to trigger checkpoint reset when Event Hubs reads fail with an ArgumentException.
Show a summary per file
File Description
src/Orleans.Streaming/PersistentStreams/IStreamQueueCheckpointer.cs Adds optional Reset() operation to the streaming checkpointer abstraction.
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/IEventHubReceiver.cs Formatting-only change in receiver proxy constructor.
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubCheckpointer.cs Adds Event Hubs checkpoint delete/reset implementation.
src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs Invokes checkpoint reset on read failures to recover from stale offsets.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 4/4 changed files
  • Comments generated: 2
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 22, 2026 10:08
@ReubenBond
ReubenBond force-pushed the gh-7256-pulling-agent-checkpoint-outofbounds branch from 4580e7a to b3a069f Compare August 22, 2026 10:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 4
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 22, 2026 10:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:244

  • ResetReceiver() nulls cache before nulling flowController. Since TryPurgeFromCache() calls IsUnderPressure() (which reads flowController) and then dereferences this.cache!, there is a race window where another thread can see a non-null flowController (returning > 0) but a null cache, causing a NullReferenceException. Clearing flowController first ensures IsUnderPressure() returns true during reset and avoids dereferencing a null cache.
            var checkpointer = Interlocked.Exchange(ref this.checkpointer, null);
            var receiver = Interlocked.Exchange(ref this.receiver, null);
            var cache = Interlocked.Exchange(ref this.cache, null);
            Interlocked.Exchange(ref this.flowController, null);
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@ReubenBond
ReubenBond force-pushed the gh-7256-pulling-agent-checkpoint-outofbounds branch from b3f61e5 to dcf3d75 Compare August 23, 2026 07:59
Copilot AI review requested due to automatic review settings August 23, 2026 07:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 13/13 changed files
  • Comments generated: 2
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 23, 2026 08:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (1)

src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:556

  • LogWarningRetryingInitializationOfEventHubPartition is using OrleansEventHubErrorCode.FailedPartitionRead for its EventId, but OrleansEventHubErrorCode defines a dedicated RetryReceiverInit value. Using the dedicated ID makes log queries/alerts unambiguous.
    [LoggerMessage(
        Level = LogLevel.Warning,
        EventId = (int)OrleansEventHubErrorCode.FailedPartitionRead,
        Message = "Retrying initialization of EventHub partition {EventHubName}-{Partition}."
    )]
    private partial void LogWarningRetryingInitializationOfEventHubPartition(string eventHubName, string partition);
  • Files reviewed: 15/15 changed files
  • Comments generated: 1
  • Review effort level: Lite

@github-actions

github-actions Bot commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 83.52% (119,780 / 143,409) 83.58% (119,037 / 142,429) -0.0530 pp
Branches 73.27% (35,652 / 48,658) 73.21% (35,432 / 48,398) +0.0609 pp

Report-only conclusion: mixed.

The current-main baseline is commit fbe240c34e and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI review requested due to automatic review settings August 23, 2026 09:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:498

  • Cursor construction assumes owner.cache is non-null (using the null-forgiving operator). If GetCacheCursor is called after Shutdown (or during a reset/reinit window where cache is temporarily cleared), this can throw a NullReferenceException. Consider throwing ObjectDisposedException (or otherwise handling null) to make the failure mode deterministic and easier to diagnose.

This issue also appears on line 522 of the same file.

            this.owner = owner;
            this.streamId = streamId;
            this.cache = owner.cache!;
            this.cursor = this.cache.GetCursor(streamId, token);
        }

src/Azure/Orleans.Streaming.EventHubs/Providers/EventDataGeneratorStreamProvider/NoOpCheckpointer.cs:10

  • The public singleton Instance field is mutable (not readonly), so external code can reassign it, which is surprising for a singleton and makes behavior harder to reason about. Making it readonly prevents accidental reassignment.

This issue also appears on line 23 of the same file.

    public static NoOpCheckpointerFactory Instance = new NoOpCheckpointerFactory();

src/Azure/Orleans.Streaming.EventHubs/Providers/Streams/EventHub/EventHubAdapterReceiver.cs:526

  • Cursor.Refresh() rebinds to owner.cache using the null-forgiving operator. If the receiver has been shut down (owner.cache == null) or the cache is being recreated, this can throw NullReferenceException. It would be safer to explicitly handle a null cache (eg, throw ObjectDisposedException) before rebinding.
        public void Refresh(StreamSequenceToken? token)
        {
            if (!ReferenceEquals(this.cache, this.owner.cache))
            {
                this.cache = this.owner.cache!;

src/Azure/Orleans.Streaming.EventHubs/Providers/EventDataGeneratorStreamProvider/NoOpCheckpointer.cs:23

  • The public singleton Instance field is mutable (not readonly), so external code can reassign it. Making it readonly prevents accidental reassignment and better matches singleton intent.
    public static NoOpCheckpointer Instance = new NoOpCheckpointer();
  • Files reviewed: 15/15 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open issues

Review effort: Lite

Comment on lines +800 to +808
if (this.recoveredCursorProgress is { } recoveredProgress)
{
if (this.recoveryPendingCursors is { Count: > 0 }
|| !recoveredProgress.Values.Any(
progress => ReferenceEquals(progress.DeliveredToken, earliestSubscriptionToken)
|| ReferenceEquals(progress.ResumeToken, earliestSubscriptionToken)))
{
return;
}
Comment on lines +171 to +192
public Task Reset(CancellationToken cancellationToken)
{
var completion = new TaskCompletionSource(
TaskCreationOptions.RunContinuationsAsynchronously);
lock (_lock)
{
_pendingResetCount++;
_throttleSavesUntilUtc = DateTime.MaxValue;
var inProgressSave = _inProgressSave;
_inProgressSave = completion.Task;
RunReset(
completion,
inProgressSave,
_inProgressSave,
_latestCheckpoint,
_updateGeneration,
cancellationToken).Ignore();
completion.Task.Ignore();
}

return completion.Task.WaitAsync(cancellationToken);
}
Comment on lines +706 to +719
lock (this.cacheLock)
{
var cache = this.cache
?? throw new InvalidOperationException("The Event Hub receiver is recovering.");
#pragma warning disable CS0618 // Preserve the exact legacy exception and cursor behavior.
var cursor = cache.GetCursor(streamId, token);
#pragma warning restore CS0618
return new Cursor(
this,
streamId,
cache,
cursor,
ReferenceEquals(cache, this.recoveryCache) ? token : null);
}
Comment on lines +1211 to +1233
var result = cache.TryGetCursor(this.streamId, token);
switch (result.Kind)
{
case QueueCacheCursorResultKind.Success:
this.owner.ClearRecoveredDeliveryToken(this);
this.cache = cache;
this.cursor = result.Cursor;
this.current = null;
this.recoveryResumeToken =
ReferenceEquals(cache, this.owner.recoveryCache) ? token : null;
this.pendingMoveResult = null;
break;
case QueueCacheCursorResultKind.CacheMiss:
this.owner.ClearRecoveredDeliveryToken(this);
this.cache = cache;
this.cursor = null;
this.current = null;
this.recoveryResumeToken = null;
this.pendingMoveResult = QueueCacheCursorMoveResult.FromCacheMiss(result.CacheMiss!.Value);
break;
default:
throw new InvalidOperationException("The cursor result is not initialized.");
}
Comment on lines +78 to +99
public Task Reset(CancellationToken cancellationToken)
{
var completion = new TaskCompletionSource(
TaskCreationOptions.RunContinuationsAsynchronously);
lock (_lock)
{
_pendingResetCount++;
_throttleSavesUntilUtc = DateTime.MaxValue;
var inProgressSave = _inProgressSave;
_inProgressSave = completion.Task;
RunReset(
completion,
inProgressSave,
_inProgressSave,
_latestCheckpoint,
_updateGeneration,
cancellationToken).Ignore();
completion.Task.Ignore();
}

return completion.Task.WaitAsync(cancellationToken);
}

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
lock (this.cacheLock)
{
this.receiverState = ReceiverRunning;
Comment on lines +507 to +508
this.recoveredCursorProgress = new Dictionary<Cursor, RecoveredCursorProgress>(ReferenceEqualityComparer.Instance);
this.recoveryPendingCursors = new HashSet<Cursor>(this.cursors, ReferenceEqualityComparer.Instance);

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +467 to +469
=> exception is ArgumentException
&& exception.Message.StartsWith("The supplied offset", StringComparison.OrdinalIgnoreCase)
&& exception.Message.Contains(" is invalid.", StringComparison.OrdinalIgnoreCase);

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment on lines +22 to +23
}
/// <summary>
Comment on lines +445 to +456
List<StreamPosition> messageStreamPositions;
lock (this.cacheLock)
{
if (!ReferenceEquals(receiver, this.receiver)
|| !ReferenceEquals(cache, this.cache)
|| cache is null)
{
return batches;
}

messageStreamPositions = cache.Add(messages, dequeueTimeUtc);
}
Comment on lines +802 to +805
if (this.recoveryPendingCursors is { Count: > 0 }
|| !recoveredProgress.Values.Any(
progress => ReferenceEquals(progress.DeliveredToken, earliestSubscriptionToken)
|| ReferenceEquals(progress.ResumeToken, earliestSubscriptionToken)))

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-streaming Category for Orleans streaming issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EventHub messages ignoring both checkpoint and StreamSequenceToken

4 participants