You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(dynamodb): opt in to ClusterOptions.ServiceId and migrate empty-ServiceId keys - #11352
DynamoDBStorageOptions.ServiceId defaults to an empty string, and DynamoDB grain storage builds its keys as {ServiceId}_{grainId}, so without it every key starts with an underscore. The other grain storage providers, and DynamoDB reminders, use ClusterOptions.ServiceId, which defaults to default. Changing the default outright would hide the state already stored under the underscore keys.
Solution
Two options, both unset by default, so nothing changes unless they are set:
Option
Unset (default)
true
false
UseClusterServiceId
follows the key format recorded in the table (below); with no record, an empty ServiceId stays empty
an empty ServiceId becomes ClusterOptions.ServiceId
an empty ServiceId stays empty; refused once the table records ClusterOptions.ServiceId
MigrateLegacyKeys
a migration recorded in the table goes on; otherwise no migration
state under the underscore keys is moved (below); refused once the table records ClusterOptions.ServiceId without a migration
no migration
At startup the provider logs the ServiceId and key format it chose, and a warning when the keys end up built from an empty ServiceId. MigrateLegacyKeys has an effect only when the keys are built from ClusterOptions.ServiceId, and either option set beside an explicit ServiceId, or MigrateLegacyKeys beside an empty one, is logged as having no effect. Both options can also be set in the provider's configuration section, and ClusterOptions.ServiceId comes from the provider's own ClusterOptions when one is registered for it, as GetProviderClusterOptions does, and otherwise from the silo's.
During a migration, a grain with no state under the new key reads it from the underscore key, with a warning, and its next write moves it: the new item is written and the old one deleted in one TransactWriteItems, conditioned on the new item not existing and on the old item's ETag, so a grain never ends up with both items or neither. Clearing such a grain moves the cleared state the same way, or, with DeleteStateOnClear, deletes the old item. A grain that has state under the new key keeps it, and its next write or clear also deletes an old item left beside it, in the same transaction, so every write or clear is a transaction over two items while the migration is on. A moved item keeps the old item's expiry when TimeToLive is not set. A transaction that fails on these conditions, or an ETag that is not a number, surfaces as an InconsistentStateException.
The provider records its choice in the table, in an item keyed __OrleansKeyFormat: EmptyServiceId, ClusterServiceId, or MigratingToClusterServiceId. With the options unset, it follows that record, so removing the options after switching doesn't switch back, and a migration in progress goes on. The record is written only if it still holds what the silo read, and a silo that loses that race reads it again and decides from it. The record is per table, so every provider that uses the table with an empty ServiceId needs the same options. The provider also refuses to start on a record it can't follow safely:
a record this version doesn't know;
UseClusterServiceId set to false once the table records ClusterOptions.ServiceId, which would hide the state written since, for this silo and every one that follows the record;
MigrateLegacyKeys set to true once the table records ClusterOptions.ServiceId without a migration: a grain whose state there was deleted or expired in the meantime would read back the older state left under the underscore key.
The last two messages say how to go ahead on purpose: delete the __OrleansKeyFormat item first.
An explicit ServiceId works as before: the options have no effect and nothing is recorded.
The docs describe the options and their constraints: set MigrateLegacyKeys together with UseClusterServiceId; enable the migration once every silo runs this version, with the same options on every silo, since a silo still writing the underscore keys is not protected from a migrating one (an item recreated after DeleteStateOnClear starts again at ETag 0); set MigrateLegacyKeys to false once every grain has been written, to spare the transactions and the second read for grains that have no state. The new DynamoDBGrainStorageKeyFormatTests cover these cases against DynamoDB Local.
Adds opt-in DynamoDB grain-key migration from empty ServiceId keys to ClusterOptions.ServiceId, preserving legacy behavior by default.
Changes:
Adds service-ID and legacy-key migration options.
Records key format and migrates state transactionally.
Updates tests, API metadata, configuration, and documentation.
Final findings include four moderate issues involving options setup, migration retry/state retention, unbounded bookkeeping, and transaction error translation, plus one test-coverage nit.
A transactional conditional failure is only recognized when CancellationReasons is populated. DynamoDB-compatible endpoints and some AWS error paths can omit that collection while including ConditionalCheckFailed in the exception message; those stale migration/clear conflicts then escape as TransactionCanceledException instead of the InconsistentStateException expected by grain storage. Match the existing transaction handler in DynamoDBTransactionalStateStorage by checking the message as a fallback.
Closing in favor of #11551, which adds the validation suggested in #9971. The branch stays as it is, so this can be reopened if the migration is wanted later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DynamoDBStorageOptions.ServiceIddefaults to an empty string, and DynamoDB grain storage builds its keys as{ServiceId}_{grainId}, so without it every key starts with an underscore. The other grain storage providers, and DynamoDB reminders, useClusterOptions.ServiceId, which defaults todefault. Changing the default outright would hide the state already stored under the underscore keys.Solution
Two options, both unset by default, so nothing changes unless they are set:
truefalseUseClusterServiceIdServiceIdstays emptyServiceIdbecomesClusterOptions.ServiceIdServiceIdstays empty; refused once the table recordsClusterOptions.ServiceIdMigrateLegacyKeysClusterOptions.ServiceIdwithout a migrationAt startup the provider logs the
ServiceIdand key format it chose, and a warning when the keys end up built from an emptyServiceId.MigrateLegacyKeyshas an effect only when the keys are built fromClusterOptions.ServiceId, and either option set beside an explicitServiceId, orMigrateLegacyKeysbeside an empty one, is logged as having no effect. Both options can also be set in the provider's configuration section, andClusterOptions.ServiceIdcomes from the provider's ownClusterOptionswhen one is registered for it, asGetProviderClusterOptionsdoes, and otherwise from the silo's.During a migration, a grain with no state under the new key reads it from the underscore key, with a warning, and its next write moves it: the new item is written and the old one deleted in one
TransactWriteItems, conditioned on the new item not existing and on the old item's ETag, so a grain never ends up with both items or neither. Clearing such a grain moves the cleared state the same way, or, withDeleteStateOnClear, deletes the old item. A grain that has state under the new key keeps it, and its next write or clear also deletes an old item left beside it, in the same transaction, so every write or clear is a transaction over two items while the migration is on. A moved item keeps the old item's expiry whenTimeToLiveis not set. A transaction that fails on these conditions, or an ETag that is not a number, surfaces as anInconsistentStateException.The provider records its choice in the table, in an item keyed
__OrleansKeyFormat:EmptyServiceId,ClusterServiceId, orMigratingToClusterServiceId. With the options unset, it follows that record, so removing the options after switching doesn't switch back, and a migration in progress goes on. The record is written only if it still holds what the silo read, and a silo that loses that race reads it again and decides from it. The record is per table, so every provider that uses the table with an emptyServiceIdneeds the same options. The provider also refuses to start on a record it can't follow safely:UseClusterServiceIdset tofalseonce the table recordsClusterOptions.ServiceId, which would hide the state written since, for this silo and every one that follows the record;MigrateLegacyKeysset totrueonce the table recordsClusterOptions.ServiceIdwithout a migration: a grain whose state there was deleted or expired in the meantime would read back the older state left under the underscore key.The last two messages say how to go ahead on purpose: delete the
__OrleansKeyFormatitem first.An explicit
ServiceIdworks as before: the options have no effect and nothing is recorded.The docs describe the options and their constraints: set
MigrateLegacyKeystogether withUseClusterServiceId; enable the migration once every silo runs this version, with the same options on every silo, since a silo still writing the underscore keys is not protected from a migrating one (an item recreated afterDeleteStateOnClearstarts again at ETag 0); setMigrateLegacyKeystofalseonce every grain has been written, to spare the transactions and the second read for grains that have no state. The newDynamoDBGrainStorageKeyFormatTestscover these cases against DynamoDB Local.Refs #9971.
Microsoft Reviewers: Open in CodeFlow