Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
e09b6c5
feat(journaling): add journaled state hooks and write preparation
ReubenBond Oct 12, 2026
75f9a44
feat(messaging): process inbox messages
ReubenBond Oct 12, 2026
2ce72bf
refactor(journaling): remove operation hooks
ReubenBond Oct 12, 2026
fbcc68c
fix(messaging): start the pump loop asynchronously
ReubenBond Oct 12, 2026
f1d04ea
test(messaging): remove handler races and simplify timeout reporting
ReubenBond Oct 13, 2026
a8d9b4f
test(messaging): capture pump shutdown token before deactivation
ReubenBond Oct 13, 2026
082033c
test(messaging): hold the active handler through deactivation
ReubenBond Oct 13, 2026
2b7f8ae
fix(messaging): address inbox review feedback
ReubenBond Oct 13, 2026
e6653ec
fix(messaging): discard wake-ups overtaken by a completed job run
ReubenBond Oct 13, 2026
4934783
feat(messaging): process inbox messages at DeliverAt
ReubenBond Oct 13, 2026
03c67c9
test(messaging): fail once per message across reactivations
ReubenBond Oct 13, 2026
b3d305b
test(messaging): compare staged test outbox messages by command equiv…
ReubenBond Oct 13, 2026
81bfc2d
fix(messaging): bind IInbox to the delivered inbox
ReubenBond Oct 13, 2026
764674f
feat(messaging): accept messages into application state through inbox…
ReubenBond Oct 13, 2026
f79ec9e
test(messaging): implement the source-provided outbox send in the tes…
ReubenBond Oct 13, 2026
e5cb39d
feat(messaging): identify messages with MessageId wrapping Hierarchic…
ReubenBond Oct 13, 2026
6ffbaf3
feat(messaging): offer scheduled messages to inbox sinks and let sink…
ReubenBond Oct 13, 2026
1268886
refactor(messaging): rename IInboxExtension to IMessagingExtension
ReubenBond Oct 13, 2026
20648e0
feat(messaging)!: demote failed deliveries to later batches and retry…
ReubenBond Oct 14, 2026
f991c53
refactor(messaging)!: track one outbox retry period and drop MaxHandl…
ReubenBond Oct 14, 2026
bceb8bb
refactor(messaging)!: align outbox retry terminology with Temporal
ReubenBond Oct 14, 2026
3edd778
feat(messaging): keep activations alive while the pump has inbox work
ReubenBond Oct 14, 2026
2bda7df
feat(messaging)!: bound inbox handler retries by time instead of atte…
ReubenBond Oct 15, 2026
5949e5e
feat(messaging): wake grains for work held by registered inbox work s…
ReubenBond Oct 15, 2026
b487eec
fix(messaging): acknowledge duplicates only after their completion is…
ReubenBond Oct 18, 2026
2223b8f
fix(messaging): count sink dead-letters as received dead_lettered
ReubenBond Oct 18, 2026
6702579
fix(messaging): resume pending messages when a handler registers inst…
ReubenBond Oct 18, 2026
340640c
fix(messaging): schedule inbox retries on the Durable Jobs clock
ReubenBond Oct 18, 2026
d2faef6
fix(messaging): schedule an earlier wake-up for work due before the c…
ReubenBond Oct 18, 2026
4873a1c
feat(messaging)!: make inbox handler selection read-only
ReubenBond Oct 18, 2026
c163082
fix(messaging): report null delivery pipeline through options validation
ReubenBond Oct 18, 2026
7b11ccb
chore(messaging): remove unused ObjectPool package reference
ReubenBond Oct 18, 2026
bef8dbc
test(messaging): fix test helper edge cases
ReubenBond Oct 18, 2026
2d17a0c
test(messaging): build the messaging test cluster during fixture init…
ReubenBond Oct 18, 2026
4f2c37c
fix(messaging): normalize non-positive work-source delays to a prompt…
ReubenBond Oct 18, 2026
69782e3
fix(messaging): answer duplicates of sink-rejected messages with thei…
ReubenBond Oct 18, 2026
0e16e44
fix(messaging): preserve inbox wakeups across capture and failed passes
ReubenBond Oct 19, 2026
d503394
fix(messaging)!: implement the sink failure and rejection-code contra…
ReubenBond Oct 19, 2026
7820fbf
fix(messaging): count scheduled messages toward inbox capacity
ReubenBond Oct 19, 2026
e3ba336
fix(messaging): keep a wake-up for committed work through any outage
ReubenBond Oct 19, 2026
fdbe1dc
feat(journaling): record staged changes per asynchronous flow for mes…
ReubenBond Oct 19, 2026
b898334
fix(messaging): discard a failed handler attempt's staged changes
ReubenBond Oct 19, 2026
65c6d5b
test(messaging): check inbox retries on an independent job clock and …
ReubenBond Oct 19, 2026
f34efca
docs(journaling): state that write preparation covers every state on …
ReubenBond Oct 19, 2026
ee846f0
fix(messaging): start the pump when activation completes
ReubenBond Oct 19, 2026
cd6eaaa
test(messaging): admit a delivery while a non-reentrant request runs
ReubenBond Oct 19, 2026
5252cc2
fix(messaging): reject a delivery whose redeliveries outlast the dedu…
ReubenBond Oct 20, 2026
0eaf87a
fix(messaging): fence forgotten inbox sends durably
ReubenBond Oct 20, 2026
f161c9c
docs(messaging): describe fenced inbox retention
ReubenBond Oct 20, 2026
c8da7a0
test(messaging): prove immutable inbox admission and retained identity
ReubenBond Oct 20, 2026
31c936e
perf(messaging): compact acknowledged snapshots without completion ti…
ReubenBond Oct 20, 2026
81813e6
test(messaging): preserve admission identity without processing times…
ReubenBond Oct 20, 2026
db83395
feat(messaging): retain identities by their latest send time
ReubenBond Oct 20, 2026
7eb7129
docs(messaging): describe finite send-time retention
ReubenBond Oct 20, 2026
d2dcf6a
test(messaging): cover required send times and finite inbox retention
ReubenBond Oct 20, 2026
e07d39b
style(messaging): format send-time identity fixtures
ReubenBond Oct 20, 2026
87a529d
test(messaging): adapt inbox-layer sink and scheduling send times
ReubenBond Oct 20, 2026
14ad3d4
fix(messaging): make processed send-time extensions replayable
ReubenBond Oct 20, 2026
e6f90f2
test(messaging): record append boundaries and self-contained retentio…
ReubenBond Oct 20, 2026
db6e52e
fix(messaging): reject send times beyond the receiver window
ReubenBond Oct 20, 2026
ff94171
fix(messaging): report receiver and pending extension invariants
ReubenBond Oct 21, 2026
1bc8458
fix(messaging): bound only send times which admission would store
ReubenBond Oct 21, 2026
b55240b
fix(messaging): assert the self-send admission bound
ReubenBond Oct 21, 2026
7a30a1d
fix(journaling): report staged changes while a state can't report them
ReubenBond Nov 18, 2026
55059e0
test(messaging): compact expired records interleaved with retained ones
ReubenBond Oct 21, 2026
3ca91db
test(messaging): pin the admission floor clamp under the maximum window
ReubenBond Oct 21, 2026
b45cc1b
fix(messaging): charge attempts which an unclean stop interrupted
ReubenBond Oct 21, 2026
5282259
test(messaging): pin the inbox send-time admission order
ReubenBond Oct 31, 2026
27a0757
style(messaging): remove trailing blank line from IInboxHandler
ReubenBond Oct 31, 2026
f4f9da9
feat(messaging): deliver outbox messages
ReubenBond Oct 19, 2026
a522509
test(messaging): check delivery under generated fault schedules
ReubenBond Oct 19, 2026
b8f7670
test(messaging): stop reading a deactivating sender's live outbox
ReubenBond Oct 20, 2026
54f8f67
feat(messaging): stamp each delivery attempt with its redelivery dead…
ReubenBond Oct 20, 2026
f942212
fix(messaging): preserve one admission cutoff for each send
ReubenBond Oct 20, 2026
020b200
fix(messaging): validate recovered outbox admission evidence
ReubenBond Oct 20, 2026
cb472f9
test(messaging): prove durable terminal work across delivery faults
ReubenBond Oct 20, 2026
570051a
test(messaging): reclaim fenced history only on snapshot acknowledgement
ReubenBond Oct 20, 2026
5615c7d
test(messaging): hold redelivery before advancing the receiver clock
ReubenBond Oct 20, 2026
595abc1
refactor(messaging): journal send time and derive retry expiration
ReubenBond Oct 20, 2026
795e55f
test(messaging): prove SentAt redelivery and durable retention extension
ReubenBond Oct 20, 2026
9a61081
style(messaging): format send-time recovery fixtures
ReubenBond Oct 20, 2026
ecdc08d
style(messaging): remove duplicated sink fixture import
ReubenBond Oct 20, 2026
6b600cb
refactor(messaging): use expiration and admission-floor terminology
ReubenBond Oct 20, 2026
e86310e
test(messaging): recover retention extensions after omitted snapshots
ReubenBond Oct 20, 2026
e3cd6f5
test(messaging): pin future-time admission and independent clock retries
ReubenBond Oct 20, 2026
0bec7d9
test(messaging): pin rollback admission bounds and replay invariants
ReubenBond Oct 21, 2026
29791af
test(messaging): preserve known outcomes across receiver clock rollback
ReubenBond Oct 21, 2026
02f6aa3
fix(messaging): report the outbox's staged sends
ReubenBond Oct 21, 2026
ca77b06
test(messaging): show a failed attempt's retry stays covered by the s…
ReubenBond Oct 21, 2026
4f5f71d
test(messaging): dispose the bootstrap probe's event subscription
ReubenBond Oct 21, 2026
30e09ad
test(messaging): charge one attempt per due message after an unclean …
ReubenBond Oct 21, 2026
759278b
docs(journaling): document state preparation before journal writes
ReubenBond Oct 19, 2026
240396e
test(messaging): add sequential messaging benchmark
ReubenBond Oct 19, 2026
809161b
test(nativeaot): exercise durable messaging in the NativeAOT smoke test
ReubenBond Oct 19, 2026
e395d28
docs(samples): add messaging sample and order-processing playground
ReubenBond Oct 19, 2026
d80cb7f
docs(messaging): add messaging guides, recipes, and compiled snippets
ReubenBond Oct 19, 2026
cd6176d
docs(samples): send the flagship sample's commands only through the o…
ReubenBond Oct 19, 2026
1869c78
fix(playground): schedule the grace period on the Durable Jobs clock
ReubenBond Oct 19, 2026
01c7192
docs(messaging): explain admission fencing and durable terminal work
ReubenBond Oct 20, 2026
0edca99
docs(messaging): describe retained cutoff evidence
ReubenBond Oct 20, 2026
89f9104
docs(messaging): explain bounded SentAt retention
ReubenBond Oct 20, 2026
bf87820
fix(nativeaot): provide messaging send times
ReubenBond Oct 20, 2026
920f9a4
docs(messaging): diagnose expired sends and unknown outcomes
ReubenBond Oct 20, 2026
9efe17b
test(docs): provide send times in messaging adapters
ReubenBond Oct 20, 2026
44e16d6
docs(messaging): distinguish retries from newly staged resubmissions
ReubenBond Oct 20, 2026
44b270b
docs(messaging): describe expiration at the admission floor
ReubenBond Oct 20, 2026
e37c874
docs(messaging): explain future send-time rejection and retry outcomes
ReubenBond Oct 20, 2026
47d6af4
docs(messaging): describe bounded admission and preserved recorded ou…
ReubenBond Oct 21, 2026
14404c4
docs(messaging): distinguish delivered and self-send time bounds
ReubenBond Oct 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Orleans.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
<Project Path="playground/DashboardCohosted/DashboardCohosted.csproj" />
<Project Path="playground/DashboardSeparateHost/DashboardSeparateHost.csproj" />
</Folder>
<Folder Name="/playground/OrderProcessing/">
<Project Path="playground/OrderProcessing/OrderProcessing.csproj" />
</Folder>
<Folder Name="/Solution Items/">
<File Path=".editorconfig" />
<File Path="Directory.Build.props" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ Exercise restore procedures regularly:
| Activation reports malformed or truncated journal data | Recovery stops before activation completes | Quarantine writes, restore a consistent backup, and preserve the failed data for diagnosis |
| <xref:Orleans.Storage.InconsistentStateException> during a write | The manager fences operations and requests deactivation; a fresh activation recovers durable state | Find stale or competing writers, confirm grain identity and cluster configuration, then retry the application command according to its idempotency contract |
| Storage commits a write but its acknowledgement is lost | The manager fences operations and requests deactivation; the new activation observes the committed mutation | Reconcile using the operation identifier and retry the command only through its idempotency protocol |
| <xref:Orleans.Journaling.JournaledStatePreCommitException> reports a state preparation failure | The storage operation is prevented and pending changes remain available | Restore the prerequisite and explicitly retry the journal write, retaining the staged business outcome |
| Write latency rises with storage queue duration | Writes for the journal are waiting behind earlier operations | Inspect provider latency and throttling, hot-grain traffic, write frequency, and snapshot size |
| Recovery time and journal bytes grow | The current append history is large | Confirm compaction thresholds are enabled and that snapshot replacements succeed |
| Format migration reports an unregistered state | A retired stream still requires its previous codec | Re-register the state for migration or retain the previous write format until retirement and compaction remove it |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,10 @@ Durable collections encode their operation before applying it to the in-memory c
- **Append** adds the encoded operation batch atomically.
- **Snapshot replacement** writes the current state of every registered stream and atomically publishes it as the new journal generation.

Concurrent calls made while the same kind of write is queued can share that queued operation. Each caller observes its completion or failure. Calls made after a storage operation starts are processed by a later operation.
Concurrent calls made while a write is queued share that queued operation. A storage compaction
request or a pending migration selects a snapshot; the requirement remains until a snapshot succeeds.
Each caller observes its completion or failure. Calls made after a storage operation starts are
processed by a later operation.

> [!IMPORTANT]
> In-memory mutation is visible before storage acknowledgement. Return success to a caller only after the required `WriteStateAsync` completes. Recovery reconstructs durable state in a new activation.
Expand All @@ -68,6 +71,20 @@ Orleans executes that synchronous block on a single activation thread. Another g
the operation awaits, so keep shared state safe to commit at each await. Any caller's write can include
staged mutations from other calls.

State implementations establish asynchronous persistence prerequisites through
<xref:Orleans.Journaling.IStateMachine.PrepareWriteAsync*>. The manager invokes this
callback before append or snapshot capture. Preparation is idempotent: prepared
prerequisites complete synchronously and remain valid through capture. After an
asynchronous completion, the manager rechecks all states to include mutations staged
on interleaved turns. A complete synchronous ready pass flows directly into capture.
The <xref:Orleans.Journaling.IStateMachine.OnWriteCompleted> callback then publishes
effects which depend on storage acknowledgement.

A failed preparation reports <xref:Orleans.Journaling.JournaledStatePreCommitException>
with the original failure as its inner exception. Pending state is retained, the manager
stays usable, and a later write retries preparation. Shutdown closes work admission and
cancels queued operations while the current operation drains to its actual storage outcome.

## Consistency and competing writers

Orleans grain placement normally supplies a single active writer for a grain identity. Journal storage providers also use optimistic concurrency to protect the journal when a stale or competing writer reaches storage.
Expand Down
247 changes: 247 additions & 0 deletions docs/site/src/content/docs/grains/messaging-idempotency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
---
title: Messaging idempotency
description: Design application command identities, hierarchical keys, retained completion facts, and external idempotency.
ms.date: 10/09/2026
ms.topic: conceptual
---

# Messaging idempotency

An application assigns one stable hierarchical ID to each logical command. Durable
Messaging uses that ID throughout admission, retries, completion, diagnostics, and
recovery. The receiving inbox's retained completion record recognizes repeated
submissions of that command, from any grain.

For the commit protocol, see [Messaging](messaging.md).
For complete grain examples, see [Practical recipes](messaging-recipes.md).

## Choose an identity at each boundary

| Identity | Owner and scope | Purpose |
| --- | --- | --- |
| Envelope `MessageId` | Exact key in the receiving grain's inbox | Coalesces pending repeats and recognizes completed commands during retention. |
| Envelope `MessageId` | Exact key in the sending grain's outbox | Binds one pending outgoing message to its destination, subject, and body. |
| Workflow root | The application's protocol | Groups related command IDs and deterministic reply IDs. |
| External idempotency key | The external system's account or API namespace | Recognizes a retried external side effect using the canonical command ID. |

Derive a command ID from stable tenant, entity, and action identifiers before
sending. Every retry and reconstructed submission uses the original ID. Namespace
independent producers explicitly. A forwarding grain preserves the original
command ID. A new workflow command
uses a distinct child or sibling ID; fan-out derives one stable recipient child
per destination.

One key identifies one immutable command: destination, subject, and body retain
their original meaning. A genuinely new reservation, authorized compensation, or
revision uses a new key. Protocol upgrades preserve the identity of an existing
command even when producers and consumers are deployed at different times.

## Transport deduplication

For deliveries, the inbox bounds a newly admitted or extended `SentAt` by the receiver's Durable
Jobs time plus
<xref:Orleans.Messaging.Configuration.DurableMessagingOptions.DeduplicationWindow>;
equality is admitted. A tracked identity with an equal or earlier send time gets
its recorded answer, including after a receiver clock rollback puts that time
beyond the bound. A later send time beyond the bound throws
<xref:System.ArgumentOutOfRangeException> with no staged change or sink offer.
For an untracked identity, expiration at the admission floor takes precedence;
otherwise, a time beyond the bound throws the same exception. The sender retries
within its original budget, so diagnose clock skew or an incorrectly stamped
direct caller.

A grain's messages to itself use its local staging time, or the first time above
the admission floor after a backward clock jump. The latter remains just beyond
the floor for compaction, even when it exceeds the delivery-time bound.

Each receiver checks the exact `MessageId`, which identifies one command at that
receiver regardless of sender or subject. A parent and its child are independent
commands. A pending duplicate refers to the already-accepted work and carries the
same subject, body bytes, and `DeliverAt` time; a pending repeat with a different
subject, body, or `DeliverAt` fails.
A completed duplicate returns `Duplicate` while its completion record is retained.
Completed records retain the ID and latest observed
<xref:Orleans.Messaging.InboxMessage.SentAt>. A repeat with a later send time
extends the pending or completed record's retention; that update is durable before
the receiver answers `Duplicate` or repeats its recorded rejection.
Applications uphold the immutable command contract; the completion fact
acknowledges the original execution.

A handler stages its business changes and outgoing messages, then calls
<xref:Orleans.Messaging.IInboxHandlerContext.Complete*>, which stages the inbox
removal and completion record. One journal write persists them together, and replay
restores the business effects and completion record together. Effects therefore
apply once per logical send. A handler can execute
more than once for the same ID, for example after a failed attempt or activation
loss, so its preparation must tolerate repetition.

The sending journal write records `SentAt` from the Durable Jobs clock. Every
attempt and recovery uses that same time, including after backpressure, and the
retry budget ends at `SentAt + OutboxRetryTimeout`. Configure `OutboxRetryTimeout`
shorter than
<xref:Orleans.Messaging.Configuration.DurableMessagingOptions.DeduplicationWindow>.
Each inbox journals a non-decreasing admission floor. The retention period
<xref:Orleans.Messaging.Configuration.DurableMessagingOptions.DeduplicationWindow>
controls how far behind the observed Durable Jobs time the floor advances.
Completed records become eligible for removal once their latest `SentAt` is at
or below that durable floor. Snapshot acknowledgement and recovery reclaim them.
A delayed delivery with that original send time then receives `Expired` before
any sink or handler runs. Forward clock jumps can expire sends early; backward
jumps can retain history longer. Budget history by processing rate and retention.
Application result-query retention serves business query and audit needs independently.

`Expired` means that earlier acceptance is unknown: an acknowledgement may have
been lost before the receiver compacted its completion record. Reconcile business
state using the original ID before authorizing another command. Automatic outbox
retries and recovery preserve the original `SentAt`; direct delivery callers
likewise reuse it. A new outbox submission after retirement receives a new staging
time. A fresh submission after an
identity's retention ends can be processed again when its new send time is
admitted; reconcile the original outcome before authorizing it.

The outbox coalesces equivalent messages that share a pending message ID, whether
staged or durable. Equivalent messages have the same destination, subject,
payload bytes, and `DeliverAt` time; the original message and its headers are kept. Reusing a pending ID
for different content throws and leaves the original message in place.

A message a grain sends to itself is keyed only in that grain's inbox. `Send` applies
the receiver rules synchronously: a pending repeat with the same subject, body bytes,
and `DeliverAt` time coalesces, a retained completed ID is a duplicate, and a pending
repeat with different content throws. A local send receives a staging time which
the inbox admits, including after a clock rollback. A later local repeat extends
the tracked identity's retention in the caller's journal write.

## Hierarchical business-operation keys

<xref:Orleans.Messaging.MessageId> identifies a message by wrapping a
<xref:Orleans.Messaging.HierarchicalKey>. Construct it with `new MessageId(key)` or
<xref:Orleans.Messaging.MessageId.Create*>, and read the key through
<xref:Orleans.Messaging.MessageId.Key>. Message IDs compare, hash, parse, and format
through that key.

<xref:Orleans.Messaging.HierarchicalKey> is an immutable value type wrapping its
canonical escaped path. Equality, `==`, `!=`, and hashing compare that path
ordinally and case-sensitively. The key provides segment-aware navigation through
<xref:Orleans.Messaging.HierarchicalKey.GetParent*>,
<xref:Orleans.Messaging.HierarchicalKey.IsParentOf*>,
<xref:Orleans.Messaging.HierarchicalKey.IsChildOf*>,
<xref:Orleans.Messaging.HierarchicalKey.IsAncestorOf*>, and a segment enumerator.
`default(HierarchicalKey)` is the unset value, exposed by `IsDefault`; envelope
validation requires a set ID.

A useful hierarchy is:

```text
tenants/{tenant}/orders/{order}
inventory/{sku}/reserve
inventory/{sku}/release
payment/charge
payment/refund/{refund}
shipment/create
```

The root groups a workflow; each leaf identifies one effect. Retried requests keep
the same leaf. Assign a stable attempt/revision segment once when authorizing a
new effect. Format numeric components invariantly and normalize business identifiers
before construction. Key comparison preserves their ordinal spelling.

<xref:Orleans.Messaging.HierarchicalKey.Create*> accepts literal segments,
and <xref:Orleans.Messaging.HierarchicalKey.CreateChildKey*> appends one literal
segment. Both escape slash and backslash characters exactly once. Construct
`payment/charge` using separate `"payment"` and `"charge"` arguments.
<xref:Orleans.Messaging.HierarchicalKey.Parse*> reads an already escaped path,
and formatting emits that canonical representation.

The following helper constructs command IDs using literal tenant and SKU identifiers:

:::code source="../snippets/compiled/Grains/MessagingRecipes.cs" id="messaging_operation_keys" language="csharp":::

The SKU `widget/blue` occupies one literal segment:

:::code source="../snippets/compiled/Grains/MessagingRecipes.cs" id="messaging_hierarchy" language="csharp":::

Use the same canonical identity across producers, inboxes, logs, result queries,
and external provider keys.
<xref:Orleans.Messaging.HierarchicalKey.IsAncestorOf*> matches at segment
boundaries: `orders/42` groups `orders/42/payment`, while `orders/420/payment`
belongs to another order. A key is its own ancestor; `IsChildOf` tests exactly one
additional segment. Deduplication always compares the full key.

Trusted ingress derives the authorized tenant/entity root and checks the claimed
command key, subject, and destination against it before accepting work. A key
expresses identity; authorization comes from that trusted application boundary.
Sending and delivery limit a message ID's canonical path to 1,024 UTF-8 bytes and a
subject to 256 UTF-8 bytes.

<a id="persist-a-result-not-just-a-flag"></a>

## Persist queryable results

A handler can stage a durable reply whose ID is the command's stable `result` child.
The [inventory recipe](messaging-recipes.md#reserve-inventory-once-per-order-line)
computes its reservation decision and next stock locally. Typed `SendReply`
encodes and stages the reply, then the method updates stock and completes synchronously. A shortage is a completed rejection with unchanged
stock. The originally committed outbox intent supplies result delivery within its
configured policy. A resubmission acknowledges the retained completion fact and
preserves that original execution.

An inbox `Duplicate` acknowledges pending or completed command identity. Obtain
the business result from the durable reply, or query business state keyed by the
same command ID. A distinct query command can request a result after the original
reply's delivery policy ends.

Store reservation records when lookup, release, or expiry needs them, payment
results for status and reconciliation, and workflow phases for progress queries.
These are business state with explicit query/audit retention. Commit them together
with local effects, outgoing intent, and inbox completion. The
[payment recipe](messaging-recipes.md#charge-an-external-provider-with-a-stable-key)
exposes a result query by the original command ID.

## External side effects

An external payment, email, or HTTP operation has its own commit boundary. An
activation can fail after that operation succeeds and before its result is journaled.
The payment recipe calls its provider during local preparation with
`context.Message.Envelope.MessageId.ToString()`. The provider enforces request consistency
and returns its original result on retries with that key.

Specify the provider account/tenant namespace, key format, retention, conflict
behavior, and outcome-query API as part of that integration. Retain provider
idempotency facts for every retry or replay which can reach it. For a provider's
key-length restriction, an adapter can deterministically hash the canonical ID and
retain the original command ID locally for correlation. Reconcile an ambiguous
response or retry with the original key. An API with an uncertain outcome requires
its reconciliation workflow before another effect is authorized.

Once the result is known, observe cancellation, send the typed reply, and finish the
journaled result and `Complete()` without awaiting. Every asynchronous
provider call precedes the first shared mutation.

## Fan-out and multi-grain workflows

Give each participant or step its own command ID under the workflow root. One live
outbox key maps to one destination, so notifications use stable recipient-specific
children. A coordinator records submitted steps and aggregates correlated outcomes
in its own journal. Participants commit their effects and deterministic replies
atomically; the coordinator commits each received outcome and next-step intent together.

Use a durable step status such as `Requested`, `Succeeded`, `Rejected`, or
`Compensated` with the participant outcome. A rejection can emit separately keyed
release or refund commands for completed steps. Each participant completes that
compensation under its own exact key. This gives the workflow an explicit forward
and recovery path across independently committed grain journals.

## Ordering and replay

| Operation | Application policy |
| --- | --- |
| Reserve stock or charge a payment | Preserve the command ID through retries and retain its inbox completion fact for the supported horizon. |
| Replace a projection with a complete snapshot | Apply only a newer domain version; complete older snapshots as superseded. |
| Apply every incremental event | Persist the expected sequence and buffer gaps before applying contiguous events. |
| Intentionally reprocess with corrected logic | Authorize a new revision key and record its relationship to the original outcome. |
| Replay an inbox dead letter | Reconcile effects keyed by the original ID, then authorize a new revision key, such as a child of the original; a retained completion answers `Duplicate`, and an untracked old send time answers `Expired`. |

Message identity recognizes completed commands; a domain version or sequence
establishes order. The
[projection recipe](messaging-recipes.md#converge-an-out-of-order-stock-projection)
demonstrates a complete-snapshot policy.
Loading
Loading