Skip to content

Start ActivationRebalancerMonitor without eagerly fetching a report - #9228

Merged
ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:fix/rebalancer-startup
Feb 15, 2026
Merged

ReubenBond merged 3 commits into
dotnet:mainfrom
ReubenBond:fix/rebalancer-startup

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Nov 14, 2024 •

Copy link
Copy Markdown
Member

Eager fetching of a report when starting the host adds an extra point of failure.

Microsoft Reviewers: Open in CodeFlow

@ReubenBond
ReubenBond force-pushed the fix/rebalancer-startup branch from 5287976 to d9f782b Compare February 5, 2025 17:56
@ReubenBond

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@ReubenBond

Copy link
Copy Markdown
Member Author

This breaks some tests which expect this to run during startup.

@ReubenBond
ReubenBond marked this pull request as draft March 20, 2025 22:36
@ReubenBond
ReubenBond force-pushed the fix/rebalancer-startup branch from d9f782b to caa7887 Compare November 17, 2025 23:54
@ReubenBond
ReubenBond force-pushed the fix/rebalancer-startup branch from caa7887 to 8ea8425 Compare February 15, 2026 21:50
@ReubenBond
ReubenBond marked this pull request as ready for review February 15, 2026 21:50
Copilot AI review requested due to automatic review settings February 15, 2026 21:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Orleans runtime’s ActivationRebalancerMonitor startup behavior to avoid an eager GetReport() call during silo host start, reducing the number of potential failure points on startup.

Changes:

  • Stop fetching the activation rebalancer report eagerly during OnStart.
  • Start the monitor timer without performing an initial GetReport() call.
Comments suppressed due to low confidence (1)

src/Orleans.Runtime/Placement/Rebalancing/ActivationRebalancerMonitor.cs:76

  • Removing the eager GetReport() call means _latestReport can remain at its default value (Host = SiloAddress.Zero) until the worker reports. OnStop uses _latestReport.Host to decide whether to migrate the rebalancer, so a silo which is actually hosting the worker could skip migration if it shuts down before receiving a report. Consider basing the migration decision solely on _activationDirectory.FindTarget(_rebalancerGrain.GetGrainId()) (since this directory is local to the silo) instead of relying on _latestReport.Host being up-to-date.
        await this.RunOrQueueTask(() =>
        {
            _monitorTimer = RegisterGrainTimer(async ct =>
            {
                var elapsedSinceHeartbeat = _timeProvider.GetElapsedTime(_lastHeartbeatTimestamp);

ReubenBond and others added 2 commits February 15, 2026 14:10
Start the monitor timer immediately and fetch an initial report only when the cached host is unknown, preserving lazy startup while avoiding stale default state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Log whenever ActivationRebalancerMonitor fetches a report, including the initial fetch when host is unknown.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ReubenBond
ReubenBond enabled auto-merge February 15, 2026 22:21
@ReubenBond
ReubenBond added this pull request to the merge queue Feb 15, 2026
Merged via the queue into dotnet:main with commit 9f17d8f Feb 15, 2026
59 checks passed
@ReubenBond
ReubenBond deleted the fix/rebalancer-startup branch February 15, 2026 23:33
rkargMsft pushed a commit to rkargMsft/orleans that referenced this pull request Feb 27, 2026
…otnet#9228)

* Start ActivationRebalancerMonitor without eagerly fetching a report

* Fix monitor startup report initialization

Start the monitor timer immediately and fetch an initial report only when the cached host is unknown, preserving lazy startup while avoiding stale default state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Always log when monitor fetches report

Log whenever ActivationRebalancerMonitor fetches a report, including the initial fetch when host is unknown.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 18, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants