Repository navigation
feat(messaging): process inbox messages - #11284
ReubenBond wants to merge 69 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved lifecycle, selection-mutation, scheduling, recovery, retention, validation, and outbox-contract issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (3)
What changed in this PR
Adds a non-packable Durable Messaging inbox runtime integrated with Journaling and Durable Jobs, including recoverable ownership, transactional handler processing, retries, deduplication, and dead-lettering.
Changes:
- Adds durable messaging contracts, routing, diagnostics, and configuration.
- Implements journal-backed acceptance, recovery, pump coordination, and handler processing.
- Adds Journaling integration and comprehensive contract and functional tests.
| File | Reviewed change |
|---|---|
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbeTests.cs |
Snapshot probe tests |
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs |
Snapshot waiting helper |
test/Orleans.DurableMessaging.Tests/Support/ReceiverTestServices.cs |
Test service composition |
test/Orleans.DurableMessaging.Tests/Support/JournaledTestOutbox.cs |
Journal-backed test outbox |
test/Orleans.DurableMessaging.Tests/Support/HandlerProbe.cs |
Handler synchronization helper |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingMetricProbe.cs |
Metrics test probe |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingClusterFixture.cs |
Messaging test cluster |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingBehaviorTestBase.cs |
Shared behavior test base |
test/Orleans.DurableMessaging.Tests/Support/ControlledJournalStorageProvider.cs |
Journal failure injection |
test/Orleans.DurableMessaging.Tests/Support/ControlledDurableJobManager.cs |
Controlled job scheduling |
test/Orleans.DurableMessaging.Tests/Orleans.DurableMessaging.Tests.csproj |
Test project configuration |
test/Orleans.DurableMessaging.Tests/Functional/MessagingRoutingAndDeadLetterTests.cs |
Routing and dead-letter tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxHandlerTransactionTests.cs |
Handler transaction tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxCapacityBehaviorTests.cs |
Capacity behavior tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxCallbackOwnershipTests.cs |
Callback ownership tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxAcceptanceBehaviorTests.cs |
Acceptance behavior tests |
test/Orleans.DurableMessaging.Tests/Functional/DurableMessagingClusterCollection.cs |
Test collection definition |
test/Orleans.DurableMessaging.Tests/Functional/DedupeExpiryBehaviorTests.cs |
Deduplication expiry tests |
test/Orleans.DurableMessaging.Tests/Contracts/HandlerRoutingContractTests.cs |
Handler routing contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DurableMessagingPumpResultsTests.cs |
Pump result contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DurableEnvelopeContractTests.cs |
Envelope contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DeliveryAndOptionsContractTests.cs |
Delivery and options contracts |
test/Orleans.DurableMessaging.Tests/Contracts/ActivationValidationTests.cs |
Activation validation contracts |
src/Orleans.Journaling/IJournaledStateObserver.cs |
Journal lifecycle observer API |
src/Orleans.Journaling/IJournaledStateManager.cs |
Observer registration API |
src/Orleans.Journaling/IJournaledGrainParticipant.cs |
Participant contract |
src/Orleans.Journaling/DurableGrain.cs |
Participant initialization |
src/Orleans.DurableMessaging/RoutePrefixHandler.cs |
Prefix route handler |
src/Orleans.DurableMessaging/RouteKeyHandler.cs |
Exact route handler |
src/Orleans.DurableMessaging/README.md |
Runtime documentation |
src/Orleans.DurableMessaging/Orleans.DurableMessaging.csproj |
Messaging project configuration |
src/Orleans.DurableMessaging/InboxHandlerSelectionContext.cs |
Handler selection context |
src/Orleans.DurableMessaging/InboxHandlerContext.cs |
Handler context implementation |
src/Orleans.DurableMessaging/IInboxHandlerContext.cs |
Handler context API |
src/Orleans.DurableMessaging/IInboxHandler.cs |
Handler contract |
src/Orleans.DurableMessaging/IDurableOutbox.cs |
Outbox contract |
src/Orleans.DurableMessaging/IDurableMessagingDiagnostics.cs |
Diagnostics API |
src/Orleans.DurableMessaging/IDurableInboxExtension.cs |
Inbox extension API |
src/Orleans.DurableMessaging/IDurableInbox.cs |
Inbox API |
src/Orleans.DurableMessaging/DurableMessagingTime.cs |
Time arithmetic helpers |
src/Orleans.DurableMessaging/DurableMessagingStateNames.cs |
Durable state names |
src/Orleans.DurableMessaging/DurableMessagingStateManagerCapabilities.cs |
State manager capability checks |
src/Orleans.DurableMessaging/DurableMessagingPumpResults.cs |
Pump execution results |
src/Orleans.DurableMessaging/DurableMessagingPumpCoordinator.cs |
Pump coordination |
src/Orleans.DurableMessaging/DurableMessagingJobOwnership.cs |
Job ownership metadata |
src/Orleans.DurableMessaging/DurableMessagingInstruments.cs |
Messaging metrics |
src/Orleans.DurableMessaging/DurableMessagingGrainParticipant.cs |
Messaging participant |
src/Orleans.DurableMessaging/DurableMessagingActivationValidator.cs |
Activation validation |
src/Orleans.DurableMessaging/DurableMessageState.cs |
Retry and dead-letter state |
src/Orleans.DurableMessaging/DurableInbox.cs |
Inbox storage and routing |
src/Orleans.DurableMessaging/DurableEnvelopeData.cs |
Deferred payload access |
src/Orleans.DurableMessaging/DurableEnvelope.cs |
Durable envelope model |
src/Orleans.DurableMessaging/DurableDeadLetterRetention.cs |
Dead-letter retention |
src/Orleans.DurableMessaging/DeliveryStatus.cs |
Delivery status values |
src/Orleans.DurableMessaging/DeliveryResult.cs |
Delivery result contract |
src/Orleans.DurableMessaging/CorrelationHandler.cs |
Correlation routing |
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs |
Inbox configuration |
src/api/Orleans.Journaling/Orleans.Journaling.cs |
Generated Journaling API |
Orleans.slnx |
Solution project registration |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage
Report-only conclusion: improved. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 10/11, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Six unresolved moderate findings and one nit remain, covering state rollback, option validation, shutdown and recovery handling, test-outbox idempotency, and an unused injected field.
Review effort: Lite
Findings: 2
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
One critical build issue and five moderate correctness, lifecycle, and test-isolation issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Six unresolved findings remain, including build-blocking and critical correctness issues.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 5
Open (6)
Composed keys with multi-segment child values lose segments · New Import CancellationToken in the handler context contract · New Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary Honor idempotent duplicate sends in the test outbox
4d18916 to
b05c432
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved ownership validation, compilation, and routing/performance issues require changes before approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 6
Open (6)
Reject delivery when recovered ownership is invalid · New Import CancellationToken in the handler context contract Composed keys with multi-segment child values lose segments Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary
Resolved since last review (1)
b05c432 to
c7b6bd9
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved ownership-race, retry/dead-letter accounting, deduplication, and shutdown-fault handling issues block approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (2)
Resolved since last review (7)
Reject delivery when recovered ownership is invalid Import CancellationToken in the handler context contract Composed keys with multi-segment child values lose segments Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary Honor idempotent duplicate sends in the test outbox
c7b6bd9 to
67a4a2e
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Two critical issues remain in Inbox outcome handling: conflicting Complete-after-Fail calls can silently commit a dead letter, and duplicate responses can return before the original acceptance is durable.
2 open issues
4 resolved since last review
Review effort: Lite
Journaled states stage changes into one buffer shared by every caller, so a handler which threw after staging a change committed it with its retry record, and every retry applied the effect again until the message dead-lettered. The inbox runs each attempt's handlers in a staged-change scope. The pump runs handlers outside any grain request, so requests and deliveries interleave at every await; the scope follows only the handler's own flow, so their changes remain theirs. An attempt which ends without an outcome after its own flow staged a change cannot record a retry without committing it, so it fails the activation; recovery from the journal discards the change and the message runs again. Messaging failure now vetoes every later write through state preparation, so neither an interleaved call nor deactivation can commit the abandoned changes or a torn batch. A return without Complete or Fail stages nothing on its own, so it retries like an exception instead of deactivating the grain, which crash-looped the inbox without recording an attempt. The docs give one order everywhere: fallible work first, then send, mutate and complete in one synchronous block.
…drop racy waits Every retry fixture made the grain clock and the Durable Jobs clock the same object, so reading either clock passed. The retry fixture now runs its own job clock days apart from the grain clock, and a test shows that advancing only the grain clock neither retries nor dead-letters a message. Two acceptance tests asserted a wait by sleeping or by checking IsCompleted right after a call, which passed by race. They now assert the ordering contract directly: a duplicate is answered only once the completion it depends on is durable, and a duplicate of a durable pending message is answered at once.
…every write The inbox ensures wake-ups for work which other states hold, and vetoes writes after a messaging failure, so it relies on the manager preparing every registered state before each append or snapshot, including states without pending changes. The contract described only a state's own prerequisites, which would permit an optimization that silently drops those wake-ups. State the guarantee and pin it with a test.
Wake-ups for due work now fire up to RecheckDelay later, because the live loop is expected to perform the work. The loop started only on a delivery or a job run, so recovered work, and work staged by an activation which no messaging call had reached, waited for the wake-up. Start the loop once activation completes, when the activation services calls.
DeliverAsync interleaves with other requests, so a delivery reaches a grain whose running request is suspended, which is what lets a request wait for a message.
…plication window An inbox recognizes a redelivery only while it keeps the message's processed record. Before recording a new message, it compares the sender's redelivery deadline with the time its window covers on the Durable Jobs clock, which the deadline uses, and answers DeadLettered with DeduplicationWindowExceededCode instead of accepting a message whose redelivery it could process again. The inbox stages nothing for the rejection and the answer doesn't depend on its state, so every redelivery receives it. Duplicates are checked first, so a message already recorded stays a duplicate. The rejection logs an error naming the configuration to fix.
A self-send stages its acceptance in the sending turn's flow, so it belongs to a handler attempt only when the handler itself sends it.
Apply the coordinator-selected ordering: known equal or earlier sends retain their answer, and expiration precedes the future bound after rollback. New sends and retention extensions remain bounded.
Messaging fails the activation when a failed handler attempt staged changes, so they can't commit with its retry record, and decides that with a staged-change scope around the attempt. A JournaledPersistentState is mutated in place and serialized whole when a write captures it, so its changes never reach the scope: a handler which mutated it and threw would have its partial state committed by the retry write. States which report every change now implement the internal IReportsStagedChanges: the built-in collections, values, and task completion sources, and the messaging inbox. While any other state is registered, every scope reports staged changes, so a failed attempt recovers from the journal instead.
CompactProcessed removes processed records while enumerating them, which Dictionary supports on .NET Core 3.0 and later, and so on both frameworks Messaging targets. The test places expired records first and between retained ones, so compaction keeps enumerating after each removal.
Options validation accepts any positive DeduplicationWindow, including TimeSpan.MaxValue. The floor subtracts the window from the clock's ticks, which stays above long.MinValue for every representable time, and clamps at the minimum time. The test runs delivery, deduplication, and snapshot compaction with that window.
A handler attempt which fails after staging changes fails the activation, so recovery discards those changes, but it also discards the attempt's failure record. The message then reran with the same attempt count, so backoff and InboxRetryTimeout never applied: a handler which always failed after staging looped indefinitely, and so did any failure on a grain whose state can't report its changes. A graceful stop now journals a clean-stop record when messages are pending, and recovery charges one attempt to each due message unless the journal ends with that record. The charge commits before the pump runs a handler, and a recovery which finds a clean stop journals that it resumed, so a later unclean stop is charged. The happy path writes nothing extra; the cost falls on deactivation with pending work and on recovery. A crash also charges due messages which hadn't been dispatched, so attempts can overcount by one per unclean stop. A charge starts a message's retry window but dead-letters only a message whose window an earlier failure started, so one crash never dead-letters a message by itself.
The future bound, its exemption for recorded and floor-expired identities, and equality admission shipped without tests. Pin each branch, including receiver clock rollback, on a Durable Jobs clock which stands still between reading the bound and delivering.
There was a problem hiding this comment.
🔵 Needs a closer look
Human validation is still warranted because the change spans durable journal protocols, activation recovery, retention timing, wake-up scheduling, and public contracts, with a lifecycle-state bug and unverified rolling/provider behavior.
No open issues 🎉
2 resolved since last review
Review effort: Lite



Durable messaging stack: #11282 (contracts, merged as ec2c1bd) → #11284 (inbox) → #11285 (outbox) → #10693 (integration, documentation, samples, NativeAOT smoke, benchmarks, and playground). This layer is based on
main.Problem
An inbox must durably accept commands, run handlers after acknowledgement, and commit business effects and completion together. Identity history must be reclaimable while delayed delivery, lost replies, and clock rollback preserve recorded decisions.
Solution
DeduplicationWindow. A completed record becomes removable once the durable floor covers its latest observedSentAt. Acknowledged snapshots and recovery reclaim it; ordinary and scalar acknowledgements remain constant-time.SentAtextensions: a redelivery with a laterSentAtextends the identity's retention before the inbox answers it. A processed extension is a self-contained completion upsert, so replay succeeds when a preceding snapshot omitted the older record. A pending extension is a delta and keeps the payload, subject, andDeliverAtconflict checks. A missing pending entry fails replay explicitly, and staging asserts ownership.SentAtfuture-time and expiry checks, so a known identity receives its recorded answer.Rationale
The journal owns atomicity and acknowledgement; the inbox owns admission, recorded outcomes, and latest-observed times. Bounded extensions keep history reclaimable; the durable floor preserves expiration, and known outcomes survive backward clock movement. #11285 supplies the outbox runtime; this layer remains independently buildable.