Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 27 additions & 11 deletions docs/site/src/content/docs/host/monitoring/metrics-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,20 +72,36 @@ Request latency covers the interval until the caller's callback completes, inclu
| Instrument | Type | Unit | Attributes | Description |
|---|---|---|---|---|
| `orleans-catalog-activation-collections` | C | Scans, implicit | - | Activation-collection scans performed by the silo. |
| `orleans-catalog-activation-concurrent-registration-attempts` | C | Attempts, implicit | - | Activation registrations which encountered an activation already registered elsewhere. |
| `orleans-catalog-activation-created` | C | Activations, implicit | - | Activation objects created on the silo. |
| `orleans-catalog-activation-destroyed` | C | Activations, implicit | - | Activations removed from the silo catalog. |
| `orleans-catalog-activation-failed-to-activate` | C | Activations, implicit | - | Activations whose lifecycle initialization failed or was canceled. |
| `orleans-catalog-activation-latency` | H | `ms` | `status`, `directory` | Activation duration, split by outcome and whether the grain uses the directory. Status values include `success`, `canceled`, `directory_error`, `duplicate`, and `error`. |
| `orleans-catalog-activation-non-existent` | C | Messages, implicit | - | Messages which targeted an activation that wasn't present in the local catalog. |
| `orleans-catalog-activation-shutdown` | C | Activations, implicit | `via` | Activation shutdowns split by `collection`, `deactivateOnIdle`, `deactivateStuckActivation`, or `migration`. |
| `orleans-catalog-activation-working-set` | OG | Activations, implicit | - | Activations currently in the silo's active working set. |
| `orleans-catalog-activations` | OG | Activations, implicit | - | Activations currently registered in the silo catalog. |
| `orleans-catalog-deactivation-latency` | H | `ms` | `via` | Deactivation duration, split by the shutdown path. |
| `orleans-grains` | UDC | Grains, implicit | `type` | Current grain instances by grain type. |
| `orleans-catalog-activation-concurrent-registration-attempts` | C | Attempts, implicit | `grain_type` | Activation registrations which encountered an activation already registered elsewhere. |
| `orleans-catalog-activation-created` | C | Activations, implicit | `grain_type` | Grain contexts created by the silo catalog, including attempts which subsequently fail activation. |
| `orleans-catalog-activation-destroyed` | C | Activations, implicit | `grain_type` | Grain contexts successfully unregistered from the silo catalog. |
| `orleans-catalog-activation-failed-to-activate` | C | Activations, implicit | `grain_type` | Activations whose lifecycle initialization failed or was canceled. |
| `orleans-catalog-activation-latency` | H | `ms` | `grain_type`, `status`, `directory` | Activation duration, split by outcome and whether the grain uses the directory (`enabled` or `disabled`). Status values are `success`, `canceled`, `directory_error`, `duplicate`, and `error`. |
| `orleans-catalog-activation-non-existent` | C | Messages, implicit | `grain_type` | Requests for a new activation which the catalog could not create while the silo was inactive, tagged with the target's canonical grain type. |
| `orleans-catalog-activation-shutdown` | C | Shutdown events, implicit | `via`, `grain_type` for individual activations | Activation shutdowns split by `collection`, `deactivateOnIdle`, `deactivateStuckActivation`, or `migration`. Includes the historical additional event for each nonempty collection batch, tagged only with `via=collection`. |
| `orleans-catalog-activation-working-set` | OG | Activations, implicit | `grain_type` | Activations recently active on this silo, counted until working-set eviction or deactivation. |
| `orleans-catalog-activations` | OG | Catalog entries, implicit | `grain_type` | Targets currently registered in the local activation directory, including system targets and stateless-worker group contexts. |
| `orleans-catalog-deactivation-latency` | H | `ms` | `grain_type`, `via` | Deactivation duration, split by the shutdown path. `via=unknown` identifies completion before a shutdown path was selected. |
| `orleans-grains` | UDC | Grains, implicit | `grain_type` | Constructed grain instances, incremented on instance assignment and decremented on disposal. Each stateless-worker instance contributes individually. |
| `orleans-scheduler-long-running-turns` | C | Turns, implicit | - | Grain micro-turns whose synchronous execution exceeded <xref:Orleans.Configuration.SchedulingOptions.TurnWarningLengthThreshold>. |
| `orleans-system-targets` | UDC | System targets, implicit | `type` | Current Orleans system-target instances by type. |

### Grain-type identity and aggregation

`grain_type` is the canonical string representation of the target's <xref:Orleans.Runtime.GrainType>, equivalent to `GrainId.Type.ToString()`. Each activation and targeted message carries this identity in its grain ID. Explicit grain-type names and constructed generic arguments are part of that identity, including a grain explicitly named `unknown`. The value can differ from the CLR implementation name.

Activation metrics cache the canonical string and population counters per registered type for the silo lifetime. All activations of a type share this metadata. Rejected activation requests reuse cached names when available and otherwise format the target's grain type for that event, keeping cache growth tied to type registration. Working-set insertion/removal and activation-directory registration/removal update the corresponding populations. A scrape enumerates type counters, including zero counts after the last member leaves. Working-set members which expose no grain identity contribute an untagged population. The first idle working-set scan retains a member; eviction removes it. Completed transitions reconcile per-type and untagged sums with the corresponding local population; concurrent scrapes can observe transitions in progress.

Lifecycle counters record additive events. Sum their increases across types and any untagged contributions for the same emitter and interval to recover total event volume. Collection scans and the additional collection-batch shutdown event retain their batch-level scope; individual activation events carry their grain type. A completed activation attempt contributes one latency observation with its outcome and directory tags; a completed deactivation contributes one observation with its shutdown path. Histograms begin timing when enabled at the start of the operation.

Constructed-instance counts are stored in the same per-type metadata and updated atomically during instance assignment and disposal. Management statistics read those counters and combine types sharing a CLR implementation name when collecting a snapshot; metric events retain each canonical grain-type identity.

Population gauges record snapshots. For a cluster total, select the latest fresh observation for each emitting silo and type, then sum across those silos. Preserve distinct emitter identities, expire stale observations, and materialize gauges as current values in the backend. Repeated scrapes contribute successive snapshots. `orleans-grains` instead supplies signed instance deltas which the exporter aggregates from the start of collection.

Catalog populations, working-set populations, and constructed instance counts each retain their own scope. A stateless-worker group contributes one catalog entry and can contain several constructed workers. Activation failures can leave a catalog entry temporarily present while cleanup proceeds. During migration, the source and destination account for their own registrations and removals.

For the `orleans-grains` key/value migration and dashboard updates, see [Activation metric schema update](../../migration-guide.md#activation-metric-schema-update).

## Cluster manifests

| Instrument | Type | Unit | Attributes | Description |
Expand Down
19 changes: 12 additions & 7 deletions docs/site/src/content/docs/host/monitoring/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,17 @@ Message-size instruments `orleans-messaging-sent-messages-size` and `orleans-mes
| Instrument | Type and unit | Meaning and interpretation |
|---|---|---|
| `orleans-scheduler-long-running-turns` | Counter, count | Grain micro-turns whose synchronous execution exceeded <xref:Orleans.Configuration.SchedulingOptions.TurnWarningLengthThreshold>. The default is one second. The lifetime total naturally only increases; alert on its rate and correlate with latency, CPU, thread-pool, traces, and warning logs. |
| `orleans-catalog-activations` | Observable gauge, count | Activations currently registered on this silo. Trend it against memory and traffic. A high value isn't intrinsically unhealthy; unexpected growth, imbalance, or churn is the useful signal. |
| `orleans-catalog-activation-working-set` | Observable gauge, count | Activations in the local active working set. Compare it with total activations to understand the active portion of the catalog. |
| `orleans-grains` | Up/down counter, count; `type` | Current grain instances by grain type after aggregation. Use it to identify placement imbalance or a grain type driving activation growth. |
| `orleans-catalog-activation-created` / `orleans-catalog-activation-destroyed` | Counters, count | Activation lifecycle throughput. High rates in both directions indicate churn even when the current activation gauge is flat. |
| `orleans-catalog-activation-latency` | Histogram, `ms`; `status`, `directory` | Activation duration and outcome. Break down non-success statuses (`canceled`, `directory_error`, `duplicate`, or `error`) and correlate tail latency with directory and storage health. |
| `orleans-catalog-activation-failed-to-activate` | Counter, count | Activation attempts which failed to construct or initialize an activation. A sustained increase is an application availability signal. |
| `orleans-catalog-activations` | Observable gauge, count; `grain_type` | Targets registered in this silo's activation directory, including system targets and stateless-worker group contexts. Trend each type against memory and traffic to identify growth, imbalance, or churn. |
| `orleans-catalog-activation-working-set` | Observable gauge, count; `grain_type` | Recently active activations, retained until working-set eviction or deactivation. Compare per-type trends with catalog populations. |
| `orleans-grains` | Up/down counter, count; `grain_type` | Constructed grain instances after aggregation of instance-assignment and disposal deltas. Each stateless-worker instance contributes individually. |
| `orleans-catalog-activation-created` / `orleans-catalog-activation-destroyed` | Counters, count; `grain_type` | Catalog creation and unregistration throughput. High rates in both directions indicate churn even when the current activation gauge is flat. |
| `orleans-catalog-activation-latency` | Histogram, `ms`; `grain_type`, `status`, `directory` | Activation duration and outcome. Break down non-success statuses (`canceled`, `directory_error`, `duplicate`, or `error`) and correlate tail latency with directory and storage health. |
| `orleans-catalog-deactivation-latency` | Histogram, `ms`; `grain_type`, `via` | Deactivation duration by grain type and shutdown path. Use it to diagnose collection, migration, and shutdown delays. |
| `orleans-catalog-activation-failed-to-activate` | Counter, count; `grain_type` | Activation attempts whose lifecycle initialization failed or was canceled. A sustained increase is an application availability signal. |

The `grain_type` value is the canonical `GrainId.Type.ToString()` identity, including explicit names and constructed generic arguments. Orleans caches this value once per registered type and shares it across activations. Targeted events use the type in their grain ID; batch-level events retain their batch scope. See [Grain-type identity and aggregation](metrics-catalog.md#grain-type-identity-and-aggregation) for population boundaries and aggregation.

For lifecycle throughput, sum event-counter increases across types and any untagged contributions over the same interval. For current cluster populations, sum the latest fresh per-silo gauge values, including untagged working-set members. Configure exporter resource identity and backend freshness rules so each silo contributes one current population. Compare per-host populations using the same observation window and freshness policy.

A long-running turn means Orleans observed one scheduled work item executing synchronously beyond the configured warning threshold. It doesn't by itself prove a deadlock. Common causes include synchronous blocking, lock contention, CPU-heavy work, or blocking I/O. See [Grain turns appear stuck](troubleshooting.md#grain-turns-appear-stuck).

Expand Down Expand Up @@ -108,7 +113,7 @@ Metric identity includes resource attributes and instrument attributes. Set stab

Useful Orleans dimensions are bounded operational categories:

- `grain_type` or `type` for deployed grain types.
- `grain_type` for canonical grain types and `type` for system-target types.
- `Direction`, `ConnectionDirection`, `MessageDirection`, and `Phase`.
- `status`, `directory`, and `via` for lifecycle outcomes.
- `provider_type_name`, `state_name`, and `state_type` for configured storage.
Expand Down
12 changes: 12 additions & 0 deletions docs/site/src/content/docs/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,18 @@ Before changing packages, record the following compatibility contract:

See [Upgrade deployment and rollback](migration/deployment-and-rollback.md) before choosing a deployment strategy.

## Activation metric schema update

**Release note for the next release:** Activation lifecycle counters, latency histograms, and population gauges now include `grain_type`, using the canonical `GrainId.Type.ToString()` identity. `orleans-grains` migrates from the `type` key containing a CLR implementation name to `grain_type` containing that canonical identity. Explicitly named grains and constructed generic grains can therefore have different values as well as a different key.

Update dashboard groupings, filters, recording rules, and OpenTelemetry views to retain `grain_type`. For a mixed-version rollout, use the emitting service version to select the old or new schema. Map old CLR names to canonical names using the deployment's grain-type registrations before combining series. Count each emitter's series once in combined views.

Activation gauges now emit a snapshot for each cached type, including zero after the last activation leaves. Obtain cluster totals by summing the latest fresh observation from each silo and type. Preserve emitter identity through export and configure stale-series expiry in the backend. Lifecycle counters retain additive event semantics: summing their per-type increases over the same interval recovers total event volume.

The shutdown counter retains its historical additional event for each nonempty collection batch with only `via=collection`; individual shutdown events carry their activation's canonical type. Include both typed and untyped contributions when aggregating total shutdown events. The collection-scan counter retains scan-level measurements. `orleans-system-targets` retains its `type` key, and management grain statistics retain CLR display names with per-silo accounting.

See the [metrics catalog](host/monitoring/metrics-catalog.md#grain-type-identity-and-aggregation) for units, lifecycle outcomes, population scopes, and aggregation.

## Package version policy

Keep all `Microsoft.Orleans.*` packages on the same 10.x patch. For solutions that use NuGet Central Package Management, declare the versions in `Directory.Packages.props` and omit versions from project-level `PackageReference` items. Don't copy old provider dependency versions from migration examples; select a current version that is supported by your target runtime and provider.
Expand Down
2 changes: 2 additions & 0 deletions samples/GPSTracker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ Dots will appear on the map in the Web browser and begin moving randomly around

If you're interested in observability, this sample includes some optional logging, metrics, and distributed tracing.

Import `grafana_dash.json` into Grafana to view the Prometheus metrics. Activation panels sum grain-type series per emitting instance: current populations use gauge snapshots, and lifecycle throughput uses counter rates or increases. The Grains panel groups current-schema metrics by canonical `grain_type`. It also shows older Orleans metrics with their CLR `type` names and a `legacy CLR` legend, supporting the sample's published packages and mixed-version upgrades. Map legacy CLR names to canonical grain-type registrations before combining those two schemas into a single type-level total.

```docker
docker run -p 9090:9090 --mount type=bind,source="${PWD}/prometheus.yml",target=/etc/prometheus/prometheus.yml prom/prometheus
```
Expand Down
Loading
Loading