Skip to content

feat(dynamodb): opt in to ClusterOptions.ServiceId and migrate empty-ServiceId keys - #11352

Closed
Laurianti wants to merge 11 commits into
dotnet:mainfrom
Laurianti:dynamodb-service-id
Closed

Laurianti wants to merge 11 commits into
dotnet:mainfrom
Laurianti:dynamodb-service-id

Conversation

@Laurianti

@Laurianti Laurianti commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

DynamoDBStorageOptions.ServiceId defaults to an empty string, and DynamoDB grain storage builds its keys as {ServiceId}_{grainId}, so without it every key starts with an underscore. The other grain storage providers, and DynamoDB reminders, use ClusterOptions.ServiceId, which defaults to default. Changing the default outright would hide the state already stored under the underscore keys.

Solution

Two options, both unset by default, so nothing changes unless they are set:

Option Unset (default) true false
UseClusterServiceId follows the key format recorded in the table (below); with no record, an empty ServiceId stays empty an empty ServiceId becomes ClusterOptions.ServiceId an empty ServiceId stays empty; refused once the table records ClusterOptions.ServiceId
MigrateLegacyKeys a migration recorded in the table goes on; otherwise no migration state under the underscore keys is moved (below); refused once the table records ClusterOptions.ServiceId without a migration no migration

At startup the provider logs the ServiceId and key format it chose, and a warning when the keys end up built from an empty ServiceId. MigrateLegacyKeys has an effect only when the keys are built from ClusterOptions.ServiceId, and either option set beside an explicit ServiceId, or MigrateLegacyKeys beside an empty one, is logged as having no effect. Both options can also be set in the provider's configuration section, and ClusterOptions.ServiceId comes from the provider's own ClusterOptions when one is registered for it, as GetProviderClusterOptions does, and otherwise from the silo's.

During a migration, a grain with no state under the new key reads it from the underscore key, with a warning, and its next write moves it: the new item is written and the old one deleted in one TransactWriteItems, conditioned on the new item not existing and on the old item's ETag, so a grain never ends up with both items or neither. Clearing such a grain moves the cleared state the same way, or, with DeleteStateOnClear, deletes the old item. A grain that has state under the new key keeps it, and its next write or clear also deletes an old item left beside it, in the same transaction, so every write or clear is a transaction over two items while the migration is on. A moved item keeps the old item's expiry when TimeToLive is not set. A transaction that fails on these conditions, or an ETag that is not a number, surfaces as an InconsistentStateException.

The provider records its choice in the table, in an item keyed __OrleansKeyFormat: EmptyServiceId, ClusterServiceId, or MigratingToClusterServiceId. With the options unset, it follows that record, so removing the options after switching doesn't switch back, and a migration in progress goes on. The record is written only if it still holds what the silo read, and a silo that loses that race reads it again and decides from it. The record is per table, so every provider that uses the table with an empty ServiceId needs the same options. The provider also refuses to start on a record it can't follow safely:

  • a record this version doesn't know;
  • UseClusterServiceId set to false once the table records ClusterOptions.ServiceId, which would hide the state written since, for this silo and every one that follows the record;
  • MigrateLegacyKeys set to true once the table records ClusterOptions.ServiceId without a migration: a grain whose state there was deleted or expired in the meantime would read back the older state left under the underscore key.

The last two messages say how to go ahead on purpose: delete the __OrleansKeyFormat item first.

An explicit ServiceId works as before: the options have no effect and nothing is recorded.

The docs describe the options and their constraints: set MigrateLegacyKeys together with UseClusterServiceId; enable the migration once every silo runs this version, with the same options on every silo, since a silo still writing the underscore keys is not protected from a migrating one (an item recreated after DeleteStateOnClear starts again at ETag 0); set MigrateLegacyKeys to false once every grain has been written, to spare the transactions and the second read for grains that have no state. The new DynamoDBGrainStorageKeyFormatTests cover these cases against DynamoDB Local.

Refs #9971.

Microsoft Reviewers: Open in CodeFlow

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved moderate issues affect configuration setup, migration reliability, memory usage, and concurrency error handling.

Review effort: Lite
Findings: 2 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Adds opt-in DynamoDB grain-key migration from empty ServiceId keys to ClusterOptions.ServiceId, preserving legacy behavior by default.

Changes:

  • Adds service-ID and legacy-key migration options.
  • Records key format and migrates state transactionally.
  • Updates tests, API metadata, configuration, and documentation.

Final findings include four moderate issues involving options setup, migration retry/state retention, unbounded bookkeeping, and transaction error translation, plus one test-coverage nit.

File Description
test/​Extensions/​Orleans.AWS.Tests/​StorageTests/​DynamoDBGrainStorageKeyFormatTests.cs Tests key formats and migration behavior.
src/​AWS/​Orleans.Persistence.DynamoDB/​Provider/​DynamoDBGrainStorage.cs Resolves keys and migrates legacy state.
src/​AWS/​Orleans.Persistence.DynamoDB/​Options/​DynamoDBStorageOptions.cs Adds migration configuration options.
src/​AWS/​Orleans.Persistence.DynamoDB/​Hosting/​DynamoDBGrainStorageServiceCollectionExtensions.cs Wires cluster service-ID configuration.
src/​AWS/​Orleans.Persistence.DynamoDB/​Hosting/​DynamoDBGrainStorageProviderBuilder.cs Supports configuration binding.
src/​api/​AWS/​Orleans.Persistence.DynamoDB/​Orleans.Persistence.DynamoDB.cs Updates the public API surface.
docs/​site/​src/​content/​docs/​grains/​grain-persistence/​dynamodb-storage.md Documents migration behavior and constraints.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@Laurianti

Copy link
Copy Markdown
Contributor Author

@dotnet-policy-service agree

Copilot AI review requested due to automatic review settings September 26, 2026 10:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved concurrency, marker-collision, and provider-scoping issues can cause incorrect migration behavior or state loss.

Review effort: Lite
Findings: 3 High severity

Open (3)
Resolved since last review (3)

Copilot AI review requested due to automatic review settings September 26, 2026 11:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The format marker update needs conditional compare-and-set handling to prevent concurrent initializers from selecting conflicting key formats.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (3)

Copilot AI review requested due to automatic review settings September 26, 2026 12:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A critical delete/recreate ETag ABA flaw can migrate stale state and delete newly created data.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 26, 2026 14:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The transactional storage migration and compatibility behavior warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 26, 2026 17:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unrecognized key-format markers must fail initialization instead of silently downgrading and risking split or hidden state.

Review effort: Lite
Findings: 1 High severity

Open (1)

Copilot AI review requested due to automatic review settings September 26, 2026 18:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Malformed key-format markers must fail initialization rather than cause an indefinite retry loop.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 26, 2026 19:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved migration cleanup and transaction-conflict handling issues can cause stale state resurrection or incorrect exception propagation.

Review effort: Lite
Findings: 1 High severity

Open (1)
Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Detect conditional failures from transaction exception messages

src/​AWS/​Orleans.Persistence.DynamoDB/​Provider/​DynamoDBGrainStorage.cs:574

A transactional conditional failure is only recognized when CancellationReasons is populated. DynamoDB-compatible endpoints and some AWS error paths can omit that collection while including ConditionalCheckFailed in the exception message; those stale migration/clear conflicts then escape as TransactionCanceledException instead of the InconsistentStateException expected by grain storage. Match the existing transaction handler in DynamoDBTransactionalStateStorage by checking the message as a fallback.

…ate read from the current key while migrating
Copilot AI review requested due to automatic review settings September 26, 2026 22:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Three moderate issues remain in legacy migration handling, warning volume, and transaction error classification.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI lite review requested due to automatic review settings October 3, 2026 08:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Invalid ETags can bypass concurrency checks during migration when the current ETag is zero.

Review effort: Lite
Findings: 1 High severity

Open (1)

Copilot AI lite review requested due to automatic review settings October 3, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The migration is complex, and clear-path ETag validation still requires correction.

Review effort: Lite
Findings: None

Resolved since last review (1)

@Laurianti

Copy link
Copy Markdown
Contributor Author

Closing in favor of #11551, which adds the validation suggested in #9971. The branch stays as it is, so this can be reopened if the migration is wanted later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants