Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
cc3b4e6
feat(streaming): report delivery progress periodically
ReubenBond Aug 13, 2026
30a8770
test(streaming): cover delivery progress barriers
ReubenBond Aug 13, 2026
54bca27
feat(streaming): extract reusable checkpointing
ReubenBond Aug 13, 2026
99ecb39
fix(streaming): make Kinesis recovery restart-safe
ReubenBond Aug 13, 2026
f3c5a66
feat(streaming): add recoverable partition pipeline
ReubenBond Aug 13, 2026
603918f
feat(streaming)!: add ADO.NET retained partition log
ReubenBond Aug 13, 2026
ce92965
feat(streaming): migrate Kinesis to recoverable pipeline
ReubenBond Aug 13, 2026
5a64aa6
feat(streaming)!: wire ADO.NET retained log
ReubenBond Aug 13, 2026
98a50bb
fix(streaming): incorporate preserved recovery fixes
ReubenBond Aug 13, 2026
590593d
fix(streaming): address recoverable cache review
ReubenBond Aug 13, 2026
ba03686
fix(streaming): restore retained-log invariants
ReubenBond Aug 19, 2026
6720690
fix(streaming): preserve empty-cache progress
ReubenBond Aug 19, 2026
38d7d17
fix(streaming): clarify cache initialization
ReubenBond Aug 19, 2026
6f01ed2
refactor(streaming): adopt stream partition terminology
ReubenBond Aug 19, 2026
006662b
fix(streaming): reconcile ADO.NET checkpoint conflicts
ReubenBond Aug 20, 2026
005b245
test(streaming): adopt xunit v3 conventions
ReubenBond Aug 21, 2026
91b9d5f
fix(streaming): release recoverable cache buffers
ReubenBond Aug 21, 2026
36fe122
fix(streaming): seed subscription delivery progress
ReubenBond Aug 21, 2026
7dddf93
fix(streaming): preserve recoverable progress invariants
ReubenBond Aug 24, 2026
6966d2b
fix(streaming): serialize receiver creation
ReubenBond Aug 24, 2026
33b8b07
fix(streaming): roll back failed cache packing
ReubenBond Aug 24, 2026
9b01569
fix(streaming): separate scan and delivery progress
ReubenBond Aug 24, 2026
bc50051
fix(streaming): rewind failed cache delivery
ReubenBond Aug 24, 2026
a731f46
fix(streaming): retain from checkpoint eligibility
ReubenBond Aug 24, 2026
effe067
fix(streaming): cancel Kinesis initialization attempts
ReubenBond Aug 24, 2026
8a29fbc
perf(streaming): limit checkpoint eligibility updates
ReubenBond Aug 24, 2026
b3aa49e
fix(streaming): preserve legacy resume boundaries
ReubenBond Aug 24, 2026
aaf3de5
fix(streaming): address Kinesis cache review
ReubenBond Aug 24, 2026
b238866
fix(streaming): retry canceled initialization
ReubenBond Aug 24, 2026
fdfbf39
fix(streaming): close retained recovery gaps
ReubenBond Aug 27, 2026
0246f0d
test(streaming): require persisted ADO.NET messages
ReubenBond Aug 27, 2026
8cbf071
fix(streaming): retry receiver creation
ReubenBond Aug 28, 2026
0ef3c8a
test(streaming): flow test cancellation
ReubenBond Aug 28, 2026
b09a062
test(streaming): align provider tests with main
ReubenBond Aug 28, 2026
490c248
fix(streaming): close recoverable stream review gaps
ReubenBond Aug 28, 2026
7a79b25
fix(streaming): use utf8mb4 partition identifiers
ReubenBond Aug 28, 2026
bfe386a
fix(streaming): dispose Kinesis lifecycle cancellation
ReubenBond Aug 29, 2026
fc3c7d2
fix(streaming): emit unregister after clearing pubsub
ReubenBond Aug 30, 2026
53b7c54
fix(streaming): reconcile recovery with cursor start positions
ReubenBond Sep 3, 2026
d812179
fix(streaming): align recovery with cancellation contracts
ReubenBond Sep 3, 2026
8466897
fix: forward remaining initialization cancellation
ReubenBond Sep 3, 2026
6b32bbb
fix(streaming): validate measurable failure counts
ReubenBond Sep 4, 2026
40c0af0
fix(streaming): serialize adapter initialization
ReubenBond Sep 4, 2026
6a18ccb
fix(streaming): guard Kinesis receiver initialization
ReubenBond Sep 4, 2026
c85ee97
fix(packaging): acknowledge ADO.NET alpha breaks
ReubenBond Sep 4, 2026
ef42dd0
style(streaming): sort cached block usings
ReubenBond Sep 5, 2026
7913ba5
fix(streaming): publish Kinesis adapter state atomically
ReubenBond Sep 5, 2026
f1e5a0f
style(streaming): use file-scoped namespaces
ReubenBond Sep 5, 2026
9dbb1cf
fix(streaming): route unavailable clients and trace cleanup
ReubenBond Sep 5, 2026
90b8ab9
fix(streaming): advance bounded recovery cursors safely
ReubenBond Sep 5, 2026
1cd14cf
test(streaming): use concrete unavailable consumer fixture
ReubenBond Sep 5, 2026
0aa22f0
fix(streaming): preserve ADO.NET retention continuity
ReubenBond Sep 5, 2026
bb3e829
feat(streaming): bound recovery cache encoded bytes
ReubenBond Sep 5, 2026
917b95c
test(streaming): cover production provider recovery paths
ReubenBond Sep 5, 2026
c954014
test(streaming): use concrete bounded recovery consumer
ReubenBond Sep 5, 2026
f49b393
fix(streaming): normalize legacy recovery comparisons
ReubenBond Sep 5, 2026
19f3612
fix(streaming): finalize recovery integration and API contracts
ReubenBond Sep 5, 2026
b39e144
fix(streaming): harden cross-framework and cancellation validation
ReubenBond Sep 5, 2026
1e5b1f4
test(streaming): expose acquisition lock diagnostics
ReubenBond Sep 5, 2026
642a8ce
fix: refresh stale client routes at gateway owners
ReubenBond Sep 5, 2026
7d26081
fix: scope client route refreshes to requested clients
ReubenBond Sep 5, 2026
57872a7
test: finalize authoritative client routing regressions
ReubenBond Sep 5, 2026
59ee41b
test(streaming): allow MariaDB lock snapshots to refresh
ReubenBond Sep 5, 2026
123ff5e
fix(streaming): reconcile recovery with typed cursor results
ReubenBond Sep 6, 2026
28decf0
test(transactions): cover cancellation before and during table initia…
ReubenBond Sep 6, 2026
c1a7ec4
fix(streaming): detach unavailable consumers locally
ReubenBond Sep 8, 2026
263ec5e
fix(streaming): close recovery edge cases
ReubenBond Sep 8, 2026
9f35e47
fix(streaming): drain consumer cleanup on shutdown
ReubenBond Sep 9, 2026
fbc83cc
fix(testing): bound streaming subscription diagnostic reads
ReubenBond Sep 9, 2026
99b0f31
fix(streaming): use numeric Kinesis offsets when slicing batches
ReubenBond Sep 9, 2026
903db7d
fix(runtime): isolate client route invalidations
ReubenBond Sep 9, 2026
4f494d2
fix(streaming): drain in-flight consumer delivery
ReubenBond Sep 10, 2026
122f74a
fix(streaming): preserve registration barriers during shutdown
ReubenBond Sep 10, 2026
b6e947d
docs(streaming): clarify minimum ADO.NET retention intervals
ReubenBond Sep 10, 2026
bf9cd9a
test(streaming): retain delivery progress coverage after token rebase
ReubenBond Sep 10, 2026
b9c152b
fix(streaming): acquire published ADO.NET adapter before cache access
ReubenBond Sep 10, 2026
5128d2e
fix(streaming): deprecate tokenless Event Hubs checkpointer factory
ReubenBond Sep 10, 2026
cfa45e8
fix(streaming): forward cancellation while loading ADO.NET queries
ReubenBond Sep 11, 2026
16dd269
fix(streaming): close retained recovery review gaps
ReubenBond Sep 13, 2026
8c1ee10
test(streaming): align dropped-client cleanup with shutdown draining
ReubenBond Sep 15, 2026
f744bc5
fix(streaming): make producer-initiated retirement callback-safe
ReubenBond Sep 15, 2026
315d80d
refactor(streaming): gate processing and retirement shutdown
ReubenBond Sep 15, 2026
7a298c6
fix(streaming): integrate drained checkpoints with admission shutdown
ReubenBond Sep 15, 2026
452cc10
docs(streaming): align receiver shutdown with retained-log ownership
ReubenBond Sep 16, 2026
5972495
test(streaming): align retirement setup with publisher registration
ReubenBond Sep 16, 2026
790542a
fix(streaming): align detached work and retirement lifetimes
ReubenBond Sep 16, 2026
10c1602
fix(streaming): release excluded Latest cache prefixes
ReubenBond Sep 16, 2026
77464b8
fix(streaming): drain admitted work before ending processing lifetime
ReubenBond Sep 16, 2026
2bb8106
fix(streaming): release Kinesis resources after shutdown cancellation
ReubenBond Sep 16, 2026
5b02a6b
fix(streaming): preserve ADO.NET receiver shutdown ownership
ReubenBond Sep 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/site/src/content/docs/host/monitoring/signals.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,12 @@ If spans appear as separate traces, confirm <xref:Orleans.Hosting.ClientBuilderE

Avoid recording grain keys and state values by default. They can be high-cardinality or sensitive. If an incident requires them, use restricted, time-limited capture and remove it afterward.

## Stream consumer cleanup

Subscribe to <xref:Orleans.Streaming.Diagnostics.StreamingEvents.AllEvents*> to correlate a consumer's delivery and cleanup transitions by provider, stream, subscription, consumer endpoint, and silo. <xref:Orleans.Streaming.Diagnostics.StreamingEvents.MessageDeliveryFailed> includes the attempted position and original exception. <xref:Orleans.Streaming.Diagnostics.StreamingEvents.SubscriptionUnregistration> reports when the pulling agent requests pubsub cleanup and when that request completes or fails. <xref:Orleans.Streaming.Diagnostics.StreamingEvents.SubscriptionUnregistered> marks the successful durable removal.

When a client route becomes stale, the runtime refreshes its candidate routes at the owning gateways while preserving the directory's versioned routing state. An available gateway receives the request; an unavailable client produces <xref:Orleans.Runtime.ClientNotAvailableException>, which initiates subscription cleanup. Owner-refresh failures retain the pending refresh so that a subsequent lookup can retry. For a cleanup timeout, capture the last observed transition and its exception: a delivery failure identifies the routing or consumer failure, a pending unregistration identifies the pubsub request, and a failed unregistration identifies its storage or callback failure. Keep diagnostic capture bounded and retain positions and identifiers rather than event payloads.

## Baseline alerts

Tune alerts against normal traffic and service objectives. A practical initial set is:
Expand All @@ -73,4 +79,4 @@ Page on user impact or imminent data/availability risk. Route isolated warnings

## Dashboard and telemetry backends

The Orleans Dashboard provides a current operational view and method profiling. It isn't a replacement for retained logs, metrics, traces, or alerts. Secure it as an administrative endpoint and use OTLP for durable telemetry. See [Orleans Dashboard](../../dashboard/index.md).
The Orleans Dashboard provides a current operational view and method profiling. Use logs, metrics, traces, and alerts for durable telemetry. Secure the dashboard as an administrative endpoint and export telemetry through OTLP. See [Orleans Dashboard](../../dashboard/index.md).
Original file line number Diff line number Diff line change
Expand Up @@ -75,27 +75,31 @@ Each `PersistentStreamPullingAgent` is a system target with single-threaded Orle

The default maximum adapter batch-container batch size is 1 and the empty-poll period is 100 ms. These defaults are runtime behavior, not a universal throughput recommendation.

### Shutdown and queue handoff
### Shutdown and admitted work

When an agent stops, it closes admission for new background work and stops its polling timer. It waits for receiver initialization, the active queue pump, and accepted producer registrations, subscription handshakes, and deliveries to finish. Accepted work completes its token bookkeeping and releases registration pins and batch protection while the cache and receiver remain available. Outstanding calls retain their existing messaging timeouts and retry limits while accepted work drains.

The agent then reports final delivery progress to the cache, disposes subscription cursors, and shuts down the receiver so provider-specific checkpoint flushing observes the completed progress. Registrations pending when shutdown starts keep the existing checkpoint, since their subscriber positions are still uncertain. Producer unregistration follows receiver cleanup. When the manager reuses an agent for a reassigned queue, initialization waits for that full cleanup and opens admission for the new run.
Each pulling-agent run owns admission for queue reads, producer registrations, subscription handshakes, and consumer delivery. Shutdown closes processing admission, stops polling, and drains admitted work and actual receiver initialization. Accepted calls retain their messaging timeouts and retry limits while completing token bookkeeping and releasing registration pins and batch protection with the cache and receiver still available. Repeated shutdown calls share the same completion. Initialization waits for the prior run's full cleanup before opening fresh admission and cancellation scopes.

Explicit subscription notifications receive an immediate acknowledgement while the agent tracks their asynchronous handshake through completion. This lets the subscribing consumer finish its current call and respond to the handshake.

A completed handshake establishes the subscription's current cursor and replay position. The latest requested handshake owns reconciliation; responses from superseded requests preserve that ownership. Delivery completions and error handling from an older handshake generation release their work while preserving the replacement position, so final checkpoint progress reflects the accepted rewind.

A failed re-handshake leaves the subscription's position uncertain even when it was previously registered. The agent retains the stream entry across idle cleanup and keeps the existing checkpoint until a successful handshake reconciles that position.

Subscription removal revokes in-flight handshake and delivery ownership. A terminal pub-sub action issued under valid ownership completes cleanup for that subscription identity, including when a cursor reconciliation overlaps its persistence.
Subscription removal revokes in-flight handshake and delivery ownership. A terminal pub-sub action issued under valid ownership completes cleanup for that subscription identity, including when a cursor reconciliation overlaps its persistence. Once fault persistence starts, it uses the cleanup lifetime so shutdown drains the durable operation.

An unavailable client is detached locally immediately. Durable subscription retirement has its own admission and cancellation lifetime, allowing a finishing delivery to start cleanup after processing admission closes. Once processing drains, shutdown closes retirement admission and waits for persistence and notification retries to finish. Producer-initiated retirement persists the removal and notifies the other producers; the requesting producer has already detached that subscription. Ordinary consumer unregistration notifies every registered producer.

The final delivery-progress scan preserves the checkpoint barrier for producer registrations which were pending when shutdown began. Accepted handshakes establish their final position during the drain; unresolved handshakes keep the existing checkpoint. The agent disposes subscription cursors before receiver shutdown flushes the safe checkpoint and releases its resources. Producer unregistration follows receiver cleanup. Operation failures remain observable through their lifecycle outcomes and correlated diagnostics.

## Cache and cursor invariants <a name="queue-cache"></a>

<a name="backpressure"></a>

An <xref:Orleans.Streams.IQueueCache> decouples queue reads from consumer delivery. Each subscription has an <xref:Orleans.Streams.IQueueCacheCursor>, so a slow consumer does not directly block a fast consumer at a later cursor.

The cache tracks the earliest delivery progress across active subscriptions. Purging must not remove an item still needed by any cursor. <xref:Orleans.Providers.Streams.Common.SimpleQueueCache> uses pressure buckets to stop or slow reads as lag grows instead of discarding undelivered events. Its default capacity is 4,096 batch containers.
The cache tracks the earliest contiguous partition position which is safe across active subscriptions. A matching record becomes safe after delivery or intentional filtering. A cursor also advances safely across records for other streams when no earlier matching delivery is pending, so a quiet stream does not pin an otherwise busy partition. Purging must not remove an item still needed by any cursor. <xref:Orleans.Providers.Streams.Common.SimpleQueueCache> uses pressure buckets to stop or slow reads as lag grows instead of discarding undelivered events. Its default capacity is 4,096 batch containers.

A `Latest` subscription attached to a populated recoverable cache starts after its current partition tail. That excluded prefix contributes safe scan progress immediately, allowing a byte-bounded cache to reclaim it once the other subscriptions have also advanced. The first included record remains pending until delivery succeeds or filtering accepts it. A `Latest` cursor created while the cache is empty starts with the first subsequently admitted record.

```mermaid
flowchart TB
Expand All @@ -111,11 +115,13 @@ flowchart TB

Cache capacity is not durability. The queue remains the durable boundary, subject to the adapter's acknowledgement contract.

Recoverable partitioned stream providers can compose a stream partition pipeline from <xref:Orleans.Providers.Streams.Common.RecoverableStreamReceiver%601>, a partition source, and a data adapter. The pipeline admits immutable stream records into pooled storage, reconstructs batches lazily, reconciles the earliest safe subscription scan/delivery watermark, and persists a checkpoint which resumes strictly after that position.

## Pub-sub handshake

The agent registers as a producer for each stream and obtains subscription records from stream pub-sub. It holds a pin cursor while subscription handshakes complete so cache cleanup cannot pass the requested start token. New subscription notifications update the agent's local pub-sub cache.

Sequence tokens allow a rewindable adapter to start from a supported historical position. An adapter whose <xref:Orleans.Streams.IQueueAdapter.IsRewindable?displayProperty=nameWithType> property is `false` must reject unsupported tokens rather than pretending to honor them.
Sequence tokens allow a rewindable adapter to start from a supported historical position. A start token is inclusive and remains unsafe until its record is delivered or intentionally filtered. A delivery handshake token confirms that its position was already processed. Exact `EventSequenceToken` and `EventSequenceTokenV2` values interoperate for legacy compatibility. Derived tokens compare only with the same concrete type unless the provider overrides equality, ordering, and hashing together. An adapter whose <xref:Orleans.Streams.IQueueAdapter.IsRewindable?displayProperty=nameWithType> property is `false` must reject unsupported tokens rather than pretending to honor them.

## Delivery and failure semantics

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The factory composes the adapter with queue mapping, caching, and failure handli

Custom pooled caches use <xref:Orleans.Providers.Streams.Common.ICacheDataAdapter.Compare*> to position cursors against cached messages. The default implementation compares `SequenceNumber` and `EventIndex`, preserving the numeric ordering used by existing providers. Override it when the authoritative provider position is encoded in <xref:Orleans.Providers.Streams.Common.CachedMessage.Segment>; return an order consistent with the token produced by <xref:Orleans.Providers.Streams.Common.ICacheDataAdapter.GetSequenceToken*> so cache bounds, block selection, and cache-miss detection use the same position contract.

`AddPersistentStreams` leaves checkpointing to the adapter. The non-rewindable example acknowledges completed messages through its receiver and therefore has no independent checkpoint. For a retained-log transport, implement an <xref:Orleans.Streams.IStreamQueueCheckpointerFactory>, have the receiver or cache load and update the per-partition position, and register it as a named component with `ConfigureComponent`. Persist a checkpoint only after all consumers have advanced beyond the corresponding cached messages. A no-op checkpointer is suitable only when replay position is deliberately disposable.
`AddPersistentStreams` leaves checkpointing to the adapter. The non-rewindable example acknowledges completed messages through its receiver and therefore has no independent checkpoint. For a partitioned stream transport, implement an <xref:Orleans.Streams.IStreamQueueCheckpointerFactory>, have the receiver or cache load and update the stream partition position, and register it as a named component with `ConfigureComponent`. Treat a requested cursor start as inclusive: selecting that position does not confirm its record. Persist only the earliest contiguous partition position which every subscription has delivered, intentionally filtered, or safely scanned as belonging to another stream. A no-op checkpointer is suitable only when replay position is deliberately disposable.

For a durable custom checkpoint backend, implement <xref:Orleans.Streams.IStreamCheckpointStore> and pass it to <xref:Orleans.Streams.StreamQueueCheckpointer> with <xref:Orleans.Streams.StreamQueueCheckpointerOptions>. Load the checkpointer before processing the partition. Each store update receives the expected backend version and returns the resulting <xref:Orleans.Streams.StreamCheckpointStoreState>, so a version conflict supplies the authoritative checkpoint and version for retry or reconciliation.

Expand Down Expand Up @@ -98,7 +98,9 @@ Test the adapter against the real queue service, including:
1. queue ownership moving between silos during membership changes;
1. duplicate delivery and consumer idempotency;
1. stable stream-to-partition mapping across restarts and upgrades;
1. sustained load beyond cache capacity to verify backpressure and queue retention; and
1. sustained load beyond cache capacity to verify backpressure and queue retention;
1. quiet and busy streams sharing a partition, including restart after the quiet cursor scans unrelated records;
1. cancellation while partition ownership acquisition is blocked, followed by reassignment and late command completion; and
1. sequence-token equality, ordering, and hashing in both comparison directions.

Monitor queue depth and oldest-message age by partition, receive and acknowledgement latency, redelivery count, throttling, pulling-agent errors, and consumer delivery failures. Alert before retention or visibility limits can cause data loss or a redelivery storm.
2 changes: 1 addition & 1 deletion docs/site/src/content/docs/streaming/data-adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ For Event Hubs, derive from <xref:Orleans.Streaming.EventHubs.EventHubDataAdapte
- <xref:Orleans.Streaming.EventHubs.EventHubDataAdapter.ToQueueMessage*> to encode events published through Orleans; and
- <xref:Orleans.Streaming.EventHubs.EventHubDataAdapter.GetPartitionKey*> to select the physical Event Hubs partition key.

The adapter also participates in cache conversion and sequence positioning through <xref:Orleans.Streaming.EventHubs.IEventHubDataAdapter>. Preserve the Event Hubs offset and sequence number when constructing batch tokens so checkpoint and rewind behavior remains aligned with the partition log.
The adapter also participates in cache conversion and sequence positioning through <xref:Orleans.Streaming.EventHubs.IEventHubDataAdapter>. Preserve the Event Hubs offset and sequence number when constructing batch tokens so checkpoint and rewind behavior remains aligned with the stream partition.

Register the adapter and Event Hubs connection under the same provider name on silos and publishing clients. The silo registration also configures durable Azure Table checkpoints:

Expand Down
Loading
Loading