Skip to content

fix(runtime): transition silo status before grain deactivation - #10294

Merged
ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-shutdown-grain-migration
Jul 21, 2026
Merged

ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:reubenbond-fix-shutdown-grain-migration

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Jul 21, 2026 •

Copy link
Copy Markdown
Member

Grains using PreferLocalPlacement could select the shutting-down silo when MigrateOnIdle() was called from OnDeactivateAsync, because membership status publication and grain deactivation ran concurrently.

This changes the shutdown lifecycle ordering so the silo publishes ShuttingDown before grain deactivation begins, then keeps the distributed grain directory available until shutdown-triggered migrations complete. It also adds a two-silo regression test which covers migrations both before and during asynchronous directory ownership handoff.

Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings July 21, 2026 17:38
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the silo shutdown lifecycle so membership state is published before grain deactivation begins, and keeps the distributed grain directory available long enough for shutdown-triggered grain migrations to complete. Adds a regression test covering PreferLocalPlacement grains migrating during shutdown, including a delayed membership-status publication and directory-ownership handoff timing.

Changes:

  • Introduces new lifecycle stages (GrainDeactivation, GrainDirectoryShutdown) to refine shutdown ordering.
  • Moves silo grain deactivation to the new stage and delays distributed grain directory shutdown until migrations are done.
  • Adds/updates tests to validate lifecycle ordering and distributed directory migration behavior; updates Orleans.Core public API surface accordingly.
Show a summary per file
File Description
test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryShutdownMigrationTests.cs New regression test for prefer-local grain migration during shutdown and directory handoff timing.
test/Orleans.Core.Tests/Membership/MembershipAgentTests.cs Updates lifecycle-stage assertions to account for the new shutdown stage.
src/Orleans.Runtime/Silo/Silo.cs Moves grain deactivation work to the new GrainDeactivation stage.
src/Orleans.Runtime/GrainDirectory/DistributedGrainDirectory.cs Delays directory shutdown to GrainDirectoryShutdown so migrations can complete.
src/Orleans.Core/Lifecycle/ServiceLifecycleStage.cs Adds new lifecycle stage constants used to control shutdown ordering.
src/api/Orleans.Core/Orleans.Core.cs Updates generated public API surface with the new lifecycle constants.

Copilot's findings

  • Files reviewed: 6/6 changed files
  • Comments generated: 2

@ReubenBond ReubenBond changed the title Fix grain migration during silo shutdown fix(runtime): migrate grains during silo shutdown Jul 21, 2026
@ReubenBond ReubenBond changed the title fix(runtime): migrate grains during silo shutdown fix(runtime): transition silo status before grain deactivation Jul 21, 2026
Copilot AI review requested due to automatic review settings July 21, 2026 17:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Comments suppressed due to low confidence (1)

test/Orleans.GrainDirectory.Tests/GrainDirectory/GrainDirectoryShutdownMigrationTests.cs:108

  • ViewUpdates.WithCancellation(cancellationToken) will throw OperationCanceledException when the timeout token is canceled, so this method will typically fail with an OCE and the final TimeoutException will never be thrown. Catch the cancellation and rethrow a TimeoutException (as done in other directory tests) so failures are deterministic and have a useful message.
        await foreach (var view in membershipService.ViewUpdates.WithCancellation(cancellationToken))
        {
            if (expectedMembers.SetEquals(view.Members))
            {
                return;
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 21, 2026 17:50
ReubenBond and others added 3 commits July 21, 2026 10:52
Publish the ShuttingDown membership state before grain deactivation and keep the distributed grain directory available until shutdown migrations complete.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dd8b146c-c118-4d86-b2d1-c9f3b97bd8ea
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Translate membership wait cancellation into an actionable timeout and end the artificial membership delay promptly without skipping status publication.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: dd8b146c-c118-4d86-b2d1-c9f3b97bd8ea
@ReubenBond
ReubenBond force-pushed the reubenbond-fix-shutdown-grain-migration branch from c7913c0 to 2dd0198 Compare July 21, 2026 17:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings July 21, 2026 17:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new

This was referenced Jul 29, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants