Skip to content

feat(runtime): add RunRequestAsync and IGrainContext.ReceiveLocalRequest - #11505

Merged
ReubenBond merged 31 commits into
dotnet:mainfrom
ReubenBond:rb-runrequestasync-runtime
Oct 16, 2026
Merged

ReubenBond merged 31 commits into
dotnet:mainfrom
ReubenBond:rb-runrequestasync-runtime

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Problem

Grains often do background work, such as stream handlers, messaging pumps or timer-adjacent callbacks, and then need to apply the result to grain state. To do that safely, a section of code has to run under the grain's scheduling rules: reentrancy, call filters and activation lifetime. Today there is no direct way to queue that work as a request on the activation.

Solution

IGrainContext.ReceiveLocalRequest(ILocalRequest request, LocalRequestOptions options) is the general primitive. It queues a request as a local-only, one-way request on the activation.

  • ILocalRequest is an IInvokable which receives its own outcome through void Complete(Response response): the response after the request and its call filters complete, or OrleansMessageRejectionException if the activation deactivates first. The request owns its outcome, including reporting or logging failures and keeping the first outcome it receives.
  • The request follows the grain's reentrancy rules, passes through grain call filters and [MayInterleave] predicates (which observe the request itself) and starts a new call chain, as timer ticks do. Like timer ticks, local requests are admitted regardless of MaxEnqueuedRequestsHardLimit.
  • LocalRequestOptions is a small readonly struct:
    • KeepAlive (default true) restarts the idle collection period.
    • Interleave lets the request interleave like an [AlwaysInterleave] call.
  • ActivationData implements the primitive and SystemTarget implements it explicitly. Other contexts keep the default implementation, which throws NotSupportedException. Rejected requests are completed after the activation lock is released, so Complete can safely queue further requests.

GrainBaseExtensions.RunRequestAsync overloads (synchronous or asynchronous, with and without state, with and without a result, with and without LocalRequestOptions) are the ergonomic API built on the primitive.

  • Filters observe the request as IGrainRequestInvoker.RunRequestAsync with no arguments. A custom ILocalRequest supplies its own interface name, method name and arguments.
  • The cancellation token cancels the request while it's queued. A filter that fails or sets the result without invoking the callback completes the returned task with that outcome.

Grain timers deliver ticks through ReceiveLocalRequest with their reusable per-timer invoker. As a result, TimerRegistry no longer depends on MessageFactory.

Rationale

A single primitive on IGrainContext means the grain context is the runner. It works from any thread, with no DI lookups or per-activation fields. Making the request its own completion keeps the message body the request itself, so invocation, call filters and scheduling predicates need no special handling, and custom requests cover anything beyond the default description.

The allocation cost is kept low:

  • RunRequestAsync allocates one GrainRequest, which is its own invokable, TaskCompletionSource and completion, plus one Message. Its description is constant.
  • Timer ticks allocate only their message.
  • ActivationData gets no new fields.

The public API surface in src/api gains RunRequestAsync, ILocalRequest, LocalRequestOptions and IGrainContext.ReceiveLocalRequest. The request scheduling docs gain compiled examples for running callbacks from background threads, configuring keep-alive and interleaving, and delivering custom requests. The timers and external tasks pages link to it.

Activation keep-alive holds were split out into #11506. This PR is the base of the durable Messaging stack (#11282 → #11284 → #11285 → #10693).

Microsoft Reviewers: Open in CodeFlow

Copilot AI balanced review requested due to automatic review settings October 14, 2026 21:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Filter outcomes can leave request tasks unresolved, and timer filter failures lose warning-level reporting.

2 open findings
What changed in this PR

Adds activation-scheduled background callbacks and counted keep-alive holds, providing runtime foundations for durable messaging.

Changes:

  • Adds four RunRequestAsync overloads with scheduling, cancellation, and rejection handling.
  • Adds keep-alive holds that prevent idle collection.
  • Adds integration tests, generated API entries, and usage documentation.

Automated PR coverage results were unavailable.

File Description
test/​Orleans.Runtime.Internal.Tests/​ActivationsLifeCycleTests/​ActivationCollectorTests.cs Tests counted keep-alive holds.
test/​Orleans.DefaultCluster.Tests/​RunRequestTests.cs Tests request scheduling and outcomes.
test/​Grains/​TestInternalGrains/​ActivationGCTestGrains.cs Implements hold test operations.
test/​Grains/​TestInternalGrainInterfaces/​ActivationGCTestGrainInterfaces.cs Declares hold test operations.
test/​Grains/​TestGrains/​RunRequestGrain.cs Implements request test grains.
test/​Grains/​TestGrainInterfaces/​IRunRequestGrain.cs Declares request test contracts.
src/​Orleans.Runtime/​Timers/​GrainRequest.cs Implements local callback requests.
src/​Orleans.Runtime/​OrleansContracts.txt Records the request invoker contract.
src/​Orleans.Runtime/​Core/​InternalGrainRuntime.cs Exposes the message factory.
src/​Orleans.Runtime/​Core/​InsideRuntimeClient.cs Adjusts invocation failure logging.
src/​Orleans.Runtime/​Catalog/​ActivationData.cs Integrates requests and keep-alive holds.
src/​Orleans.Core/​Runtime/​IActivationKeepAlive.cs Defines the internal hold contract.
src/​Orleans.Core.Abstractions/​Core/​IGrainRequestRunner.cs Defines internal request dispatch.
src/​Orleans.Core.Abstractions/​Core/​IGrainBase.cs Adds public callback scheduling extensions.
src/​api/​Orleans.Core.Abstractions/​Orleans.Core.Abstractions.cs Records the public overloads.
docs/​site/​src/​content/​docs/​grains/​timers.md Links to one-shot callback scheduling.
docs/​site/​src/​content/​docs/​grains/​request-scheduling.md Documents scheduling and deadlock constraints.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 21:47
@ReubenBond
ReubenBond force-pushed the rb-runrequestasync-runtime branch from 0898468 to e5d3546 Compare October 14, 2026 21:47
@ReubenBond ReubenBond changed the title feat(runtime): add RunRequestAsync and activation keep-alive holds feat(runtime): add RunRequestAsync Oct 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Filter short-circuits can leave tasks pending indefinitely, and local requests bypass configured activation queue limits.

4 open findings

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 14, 2026 22:02
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 83.53% (118,923 / 142,371) 83.50% (118,709 / 142,159) +0.0260 pp
Branches 73.16% (35,401 / 48,386) 73.10% (35,330 / 48,334) +0.0682 pp

Report-only conclusion: improved.

The current-main baseline is commit 6341f1a3ee and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 22:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Concurrent submissions can exceed the hard queue limit because admission checking and enqueueing are not atomic.

1 open finding
4 resolved since last review

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 15, 2026 01:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A cancellation race can suppress normal logging of a filter failure that the caller cannot observe.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Preserve warning when cancellation masks filter failure

src/​Orleans.Runtime/​Timers/​GrainRequest.cs:109

When a filter fails without invoking the callback, cancellation can complete the task after the IsCompleted check, including while _registration.Dispose() waits for the cancellation callback. TrySetException then returns false, but OnInvoked still returns true. InsideRuntimeClient consequently logs the filter failure at Debug even though the caller only observes cancellation. Return the result of TrySetException so this unreported failure remains at Warning.

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 15, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Interactions between activation scheduling, cancellation races, filter completion, and shutdown warrant final maintainer review.

0 open findings

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 15, 2026 15:15
@ReubenBond
ReubenBond force-pushed the rb-runrequestasync-runtime branch from 95b98a1 to 2683ab0 Compare October 15, 2026 15:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Scheduling, cancellation, and activation-lifecycle interactions warrant final human review despite focused regression coverage and no confirmed blocking findings.

0 open findings

🧠 Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting scheduling and lifecycle changes need human validation, and local-response copying still requires correction.

1 open finding

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 15:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting scheduling, cancellation, completion ownership, and timer-lifecycle changes warrant final human review despite no confirmed blocking findings.

1 open finding

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 15:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting scheduling, cancellation, and activation-lifecycle changes require final human validation.

1 open finding

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 15:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Public scheduling contracts and interacting cancellation, completion, and activation-lifecycle changes warrant final human review despite substantial regression coverage.

1 open finding

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 15:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Response-isolation gaps remain, and the cross-cutting scheduling and lifecycle changes require final human review.

0 open findings

1 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Callback results bypass response copying and expose mutable grain state

src/​Orleans.Core.Abstractions/​Runtime/​GrainRequest.cs:64

Both callback-success paths publish the original TResult to the returned task and return Response.Completed, so the runtime's response copier never sees the result. A background caller receiving a grain-owned mutable collection can therefore modify grain state off-scheduler, bypassing the response isolation intended for local requests. Copy the result on the activation before publishing it, preserving the callback-first outcome semantics. Cover both synchronously and asynchronously completed callbacks with a mutable-result regression test.

Medium severity Filter results bypass response copying and expose mutable grain state

src/​Orleans.Runtime/​Core/​InsideRuntimeClient.cs:383

A filter that short-circuits with context.Result, or replaces a custom request's result after invocation, bypasses the response copy in GrainMethodInvoker.Invoke. Local requests also skip SafeSendResponse, which copies the final response for ordinary calls. Consequently, Complete receives the filter's original mutable object, potentially exposing grain state to an off-activation caller. Copy the final local response before completion and add a regression test with a mutable filter-produced result.

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 16:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Callback and final filter results can bypass response copying, exposing mutable grain state to external callers.

0 open findings

Previously missed (2)

In code that hasn't changed since last review

Medium severity Copy callback results before publishing to prevent mutable state leaks

src/​Orleans.Core.Abstractions/​Runtime/​GrainRequest.cs:64

Both success paths (Invoke and AwaitCallbackAsync) publish the original callback result to the returned task, then return Response.Completed. The runtime's response copiers therefore never see the actual result. If a background caller receives a mutable grain-owned list, it can modify that same list outside the grain scheduler, bypassing the intended response-copy isolation. Copy the callback result before publishing it on both paths, while preserving the documented callback-first outcome semantics. Add regression tests with a mutable result for synchronous and asynchronous completion.

Medium severity Copy filtered local responses before completion

src/​Orleans.Runtime/​Core/​InsideRuntimeClient.cs:383

A filter which short-circuits by setting context.Result, or replaces the result after invocation, bypasses the copier inside GrainMethodInvoker. Ordinary requests copy the final response in SafeSendResponse, but local requests pass it directly to Complete. A filter returning mutable grain state therefore shares it with an external observer, including RunRequestAsync callers when the callback was skipped. Copy the final local response before completion and add a mutable filter-result regression test.

🧠 Review effort: Balanced

@ReubenBond

ReubenBond commented Oct 16, 2026 •

Copy link
Copy Markdown
Member Author

On the two "previously missed" findings ("Copy callback results before publishing to prevent mutable state leaks" and "Copy filtered local responses before completion"): we're leaving both as they are, by design.

The runtime passes a local request by reference, without copying it, and passes its response the same way. The response copier no longer runs for local requests, with or without call filters. RunRequestAsync hands its callback's state and result to the caller by reference, and a custom ILocalRequest decides what it hands to its own observers. A developer who needs isolation can copy the inputs and outputs.

Copilot AI balanced review requested due to automatic review settings October 16, 2026 17:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting scheduling, cancellation, and activation-lifecycle changes require human validation beyond the available static evidence.

0 open findings

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 16, 2026 20:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting scheduling and lifecycle changes warrant human review, and missing CI-selection traits leave new regression tests unselected.

0 open findings

Previously missed (1)

In code that hasn't changed since last review

Medium severity Add required BVT and None provider attributes to regression tests

test/​Orleans.Core.Tests/​Runtime/​GrainRequestTests.cs:12

Add [TestSuite("BVT")] and [TestProvider("None")] so these regression tests are selected by CI. The filter in .github/workflows/ci.yml:808 requires both Provider=None and a matching Suite, but TestCategory emits only Category (test/TestInfrastructure/TestExtensions/TestCategory.cs:27–28). Without these attributes, the new cancellation-race and submission-cleanup tests are skipped. The neighboring GrainTimerTests.cs:15–16 uses both attributes.

🧠 Review effort: Balanced

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants