Skip to content

feat(messaging): process inbox messages - #11284

Open
ReubenBond wants to merge 69 commits into
dotnet:mainfrom
ReubenBond:rb-potential-happiness
Open

ReubenBond wants to merge 69 commits into
dotnet:mainfrom
ReubenBond:rb-potential-happiness

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Durable messaging stack: #11282 (contracts, merged as ec2c1bd) → #11284 (inbox) → #11285 (outbox) → #10693 (integration, documentation, samples, NativeAOT smoke, benchmarks, and playground). This layer is based on main.

Problem

An inbox must durably accept commands, run handlers after acknowledgement, and commit business effects and completion together. Identity history must be reclaimable while delayed delivery, lost replies, and clock rollback preserve recorded decisions.

Solution

  • Durable admission and retention: a non-decreasing journaled admission floor advances to receiver time minus DeduplicationWindow. A completed record becomes removable once the durable floor covers its latest observed SentAt. Acknowledged snapshots and recovery reclaim it; ordinary and scalar acknowledgements remain constant-time.
  • Replayable SentAt extensions: a redelivery with a later SentAt extends the identity's retention before the inbox answers it. A processed extension is a self-contained completion upsert, so replay succeeds when a preceding snapshot omitted the older record. A pending extension is a delta and keeps the payload, subject, and DeliverAt conflict checks. A missing pending entry fails replay explicitly, and staging asserts ownership.
  • Durable processing and application state: acceptance, handlers, completion, business changes, and outgoing intent use their owning journal boundaries. Sinks share acceptance writes; work sources and the activation pump retain wake-ups. Scheduled work occupies capacity until completion. Bounded handler retry, prepare-write prerequisites, and keep-alive holds preserve recovery and progress.
  • Operational diagnosis: expiration and future-time warnings identify the message, receiver grain, send time, and the floor or receiver time that decided the outcome.
  • Delivery order: deduplication runs before the SentAt future-time and expiry checks, so a known identity receives its recorded answer.

Rationale

The journal owns atomicity and acknowledgement; the inbox owns admission, recorded outcomes, and latest-observed times. Bounded extensions keep history reclaimable; the durable floor preserves expiration, and known outcomes survive backward clock movement. #11285 supplies the outbox runtime; this layer remains independently buildable.

Copilot AI lite review requested due to automatic review settings September 16, 2026 22:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved lifecycle, selection-mutation, scheduling, recovery, retention, validation, and outbox-contract issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)
What changed in this PR

Adds a non-packable Durable Messaging inbox runtime integrated with Journaling and Durable Jobs, including recoverable ownership, transactional handler processing, retries, deduplication, and dead-lettering.

Changes:

  • Adds durable messaging contracts, routing, diagnostics, and configuration.
  • Implements journal-backed acceptance, recovery, pump coordination, and handler processing.
  • Adds Journaling integration and comprehensive contract and functional tests.
File Reviewed change
test/​Orleans.DurableMessaging.Tests/​Support/​SnapshotProbeTests.cs Snapshot probe tests
test/​Orleans.DurableMessaging.Tests/​Support/​SnapshotProbe.cs Snapshot waiting helper
test/​Orleans.DurableMessaging.Tests/​Support/​ReceiverTestServices.cs Test service composition
test/​Orleans.DurableMessaging.Tests/​Support/​JournaledTestOutbox.cs Journal-backed test outbox
test/​Orleans.DurableMessaging.Tests/​Support/​HandlerProbe.cs Handler synchronization helper
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingMetricProbe.cs Metrics test probe
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingClusterFixture.cs Messaging test cluster
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingBehaviorTestBase.cs Shared behavior test base
test/​Orleans.DurableMessaging.Tests/​Support/​ControlledJournalStorageProvider.cs Journal failure injection
test/​Orleans.DurableMessaging.Tests/​Support/​ControlledDurableJobManager.cs Controlled job scheduling
test/​Orleans.DurableMessaging.Tests/​Orleans.DurableMessaging.Tests.csproj Test project configuration
test/​Orleans.DurableMessaging.Tests/​Functional/​MessagingRoutingAndDeadLetterTests.cs Routing and dead-letter tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxHandlerTransactionTests.cs Handler transaction tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxCapacityBehaviorTests.cs Capacity behavior tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxCallbackOwnershipTests.cs Callback ownership tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxAcceptanceBehaviorTests.cs Acceptance behavior tests
test/​Orleans.DurableMessaging.Tests/​Functional/​DurableMessagingClusterCollection.cs Test collection definition
test/​Orleans.DurableMessaging.Tests/​Functional/​DedupeExpiryBehaviorTests.cs Deduplication expiry tests
test/​Orleans.DurableMessaging.Tests/​Contracts/​HandlerRoutingContractTests.cs Handler routing contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DurableMessagingPumpResultsTests.cs Pump result contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DurableEnvelopeContractTests.cs Envelope contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DeliveryAndOptionsContractTests.cs Delivery and options contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​ActivationValidationTests.cs Activation validation contracts
src/​Orleans.Journaling/​IJournaledStateObserver.cs Journal lifecycle observer API
src/​Orleans.Journaling/​IJournaledStateManager.cs Observer registration API
src/​Orleans.Journaling/​IJournaledGrainParticipant.cs Participant contract
src/​Orleans.Journaling/​DurableGrain.cs Participant initialization
src/​Orleans.DurableMessaging/​RoutePrefixHandler.cs Prefix route handler
src/​Orleans.DurableMessaging/​RouteKeyHandler.cs Exact route handler
src/​Orleans.DurableMessaging/​README.md Runtime documentation
src/​Orleans.DurableMessaging/​Orleans.DurableMessaging.csproj Messaging project configuration
src/​Orleans.DurableMessaging/​InboxHandlerSelectionContext.cs Handler selection context
src/​Orleans.DurableMessaging/​InboxHandlerContext.cs Handler context implementation
src/​Orleans.DurableMessaging/​IInboxHandlerContext.cs Handler context API
src/​Orleans.DurableMessaging/​IInboxHandler.cs Handler contract
src/​Orleans.DurableMessaging/​IDurableOutbox.cs Outbox contract
src/​Orleans.DurableMessaging/​IDurableMessagingDiagnostics.cs Diagnostics API
src/​Orleans.DurableMessaging/​IDurableInboxExtension.cs Inbox extension API
src/​Orleans.DurableMessaging/​IDurableInbox.cs Inbox API
src/​Orleans.DurableMessaging/​DurableMessagingTime.cs Time arithmetic helpers
src/​Orleans.DurableMessaging/​DurableMessagingStateNames.cs Durable state names
src/​Orleans.DurableMessaging/​DurableMessagingStateManagerCapabilities.cs State manager capability checks
src/​Orleans.DurableMessaging/​DurableMessagingPumpResults.cs Pump execution results
src/​Orleans.DurableMessaging/​DurableMessagingPumpCoordinator.cs Pump coordination
src/​Orleans.DurableMessaging/​DurableMessagingJobOwnership.cs Job ownership metadata
src/​Orleans.DurableMessaging/​DurableMessagingInstruments.cs Messaging metrics
src/​Orleans.DurableMessaging/​DurableMessagingGrainParticipant.cs Messaging participant
src/​Orleans.DurableMessaging/​DurableMessagingActivationValidator.cs Activation validation
src/​Orleans.DurableMessaging/​DurableMessageState.cs Retry and dead-letter state
src/​Orleans.DurableMessaging/​DurableInbox.cs Inbox storage and routing
src/​Orleans.DurableMessaging/​DurableEnvelopeData.cs Deferred payload access
src/​Orleans.DurableMessaging/​DurableEnvelope.cs Durable envelope model
src/​Orleans.DurableMessaging/​DurableDeadLetterRetention.cs Dead-letter retention
src/​Orleans.DurableMessaging/​DeliveryStatus.cs Delivery status values
src/​Orleans.DurableMessaging/​DeliveryResult.cs Delivery result contract
src/​Orleans.DurableMessaging/​CorrelationHandler.cs Correlation routing
src/​Orleans.DurableMessaging/​Configuration/​DurableInboxOptions.cs Inbox configuration
src/​api/​Orleans.Journaling/​Orleans.Journaling.cs Generated Journaling API
Orleans.slnx Solution project registration

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 16, 2026 22:19
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 84.54% (124,312 / 147,049) 84.45% (123,257 / 145,960) +0.0921 pp
Branches 75.06% (37,937 / 50,540) 74.94% (37,518 / 50,067) +0.1277 pp

Report-only conclusion: improved.

The current-main baseline is commit 1917282724 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 10/11, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Six unresolved moderate findings and one nit remain, covering state rollback, option validation, shutdown and recovery handling, test-outbox idempotency, and an unused injected field.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)

Copilot AI review requested due to automatic review settings September 16, 2026 22:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One critical build issue and five moderate correctness, lifecycle, and test-isolation issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 High severity · 1 Medium severity

Open (4)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Six unresolved findings remain, including build-blocking and critical correctness issues.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 High severity · 1 Medium severity

Open (6)

Copilot AI review requested due to automatic review settings September 16, 2026 23:05
@ReubenBond
ReubenBond force-pushed the rb-potential-happiness branch from 4d18916 to b05c432 Compare September 16, 2026 23:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved ownership validation, compilation, and routing/performance issues require changes before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 6 High severity

Open (6)
Resolved since last review (1)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved ownership-race, retry/dead-letter accounting, deduplication, and shutdown-fault handling issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 1 Medium severity

Open (2)
Resolved since last review (7)

Copilot AI review requested due to automatic review settings September 17, 2026 01:26
@ReubenBond
ReubenBond force-pushed the rb-potential-happiness branch from c7b6bd9 to 67a4a2e Compare September 17, 2026 01:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One critical and two moderate findings remain unresolved in durable inbox acceptance and scheduling.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)

Copilot AI review requested due to automatic review settings September 17, 2026 01:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
Resolved since last review (3)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two critical issues remain in Inbox outcome handling: conflicting Complete-after-Fail calls can silently commit a dead letter, and duplicate responses can return before the original acceptance is durable.

2 open issues
4 resolved since last review

Review effort: Lite

Comment thread src/Orleans.Messaging/Inbox.cs Resolved
Comment thread src/Orleans.Messaging/Inbox.cs Resolved
Journaled states stage changes into one buffer shared by every caller, so a
handler which threw after staging a change committed it with its retry record,
and every retry applied the effect again until the message dead-lettered.

The inbox runs each attempt's handlers in a staged-change scope. The pump runs
handlers outside any grain request, so requests and deliveries interleave at
every await; the scope follows only the handler's own flow, so their changes
remain theirs. An attempt which ends without an outcome after its own flow staged
a change cannot record a retry without committing it, so it fails the
activation; recovery from the journal discards the change and the message runs
again. Messaging failure now vetoes every later write through state preparation,
so neither an interleaved call nor deactivation can commit the abandoned changes
or a torn batch.

A return without Complete or Fail stages nothing on its own, so it retries like
an exception instead of deactivating the grain, which crash-looped the inbox
without recording an attempt. The docs give one order everywhere: fallible work
first, then send, mutate and complete in one synchronous block.
…drop racy waits

Every retry fixture made the grain clock and the Durable Jobs clock the same
object, so reading either clock passed. The retry fixture now runs its own job
clock days apart from the grain clock, and a test shows that advancing only the
grain clock neither retries nor dead-letters a message.

Two acceptance tests asserted a wait by sleeping or by checking IsCompleted
right after a call, which passed by race. They now assert the ordering contract
directly: a duplicate is answered only once the completion it depends on is
durable, and a duplicate of a durable pending message is answered at once.
…every write

The inbox ensures wake-ups for work which other states hold, and vetoes writes
after a messaging failure, so it relies on the manager preparing every registered
state before each append or snapshot, including states without pending changes.
The contract described only a state's own prerequisites, which would permit an
optimization that silently drops those wake-ups. State the guarantee and pin it
with a test.
Wake-ups for due work now fire up to RecheckDelay later, because the live loop
is expected to perform the work. The loop started only on a delivery or a job
run, so recovered work, and work staged by an activation which no messaging call
had reached, waited for the wake-up. Start the loop once activation completes,
when the activation services calls.
DeliverAsync interleaves with other requests, so a delivery reaches a grain whose
running request is suspended, which is what lets a request wait for a message.
…plication window

An inbox recognizes a redelivery only while it keeps the message's processed
record. Before recording a new message, it compares the sender's redelivery
deadline with the time its window covers on the Durable Jobs clock, which the
deadline uses, and answers DeadLettered with DeduplicationWindowExceededCode
instead of accepting a message whose redelivery it could process again.

The inbox stages nothing for the rejection and the answer doesn't depend on its
state, so every redelivery receives it. Duplicates are checked first, so a
message already recorded stays a duplicate. The rejection logs an error naming
the configuration to fix.
A self-send stages its acceptance in the sending turn's flow, so it belongs to
a handler attempt only when the handler itself sends it.
Apply the coordinator-selected ordering: known equal or earlier sends retain their answer, and expiration precedes the future bound after rollback. New sends and retention extensions remain bounded.
Messaging fails the activation when a failed handler attempt staged changes, so
they can't commit with its retry record, and decides that with a staged-change
scope around the attempt. A JournaledPersistentState is mutated in place and
serialized whole when a write captures it, so its changes never reach the
scope: a handler which mutated it and threw would have its partial state
committed by the retry write.

States which report every change now implement the internal
IReportsStagedChanges: the built-in collections, values, and task completion
sources, and the messaging inbox. While any other state is registered, every
scope reports staged changes, so a failed attempt recovers from the journal
instead.
CompactProcessed removes processed records while enumerating them, which Dictionary supports on .NET Core 3.0 and later, and so on both frameworks Messaging targets. The test places expired records first and between retained ones, so compaction keeps enumerating after each removal.
Options validation accepts any positive DeduplicationWindow, including TimeSpan.MaxValue. The floor subtracts the window from the clock's ticks, which stays above long.MinValue for every representable time, and clamps at the minimum time. The test runs delivery, deduplication, and snapshot compaction with that window.
A handler attempt which fails after staging changes fails the activation, so
recovery discards those changes, but it also discards the attempt's failure
record. The message then reran with the same attempt count, so backoff and
InboxRetryTimeout never applied: a handler which always failed after staging
looped indefinitely, and so did any failure on a grain whose state can't report
its changes.

A graceful stop now journals a clean-stop record when messages are pending, and
recovery charges one attempt to each due message unless the journal ends with
that record. The charge commits before the pump runs a handler, and a recovery
which finds a clean stop journals that it resumed, so a later unclean stop is
charged. The happy path writes nothing extra; the cost falls on deactivation
with pending work and on recovery.

A crash also charges due messages which hadn't been dispatched, so attempts can
overcount by one per unclean stop. A charge starts a message's retry window but
dead-letters only a message whose window an earlier failure started, so one
crash never dead-letters a message by itself.
The future bound, its exemption for recorded and floor-expired identities, and equality admission shipped without tests. Pin each branch, including receiver clock rollback, on a Durable Jobs clock which stands still between reading the bound and delivering.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human validation is still warranted because the change spans durable journal protocols, activation recovery, retention timing, wake-up scheduling, and public contracts, with a lifecycle-state bug and unverified rolling/provider behavior.

No open issues 🎉

2 resolved since last review

Review effort: Lite

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants