Skip to content

fix(firestore): preserve membership consistency - #11311

Merged
ReubenBond merged 10 commits into
dotnet:mainfrom
ReubenBond:rb-fix-firestore-membership-consistency
Sep 18, 2026
Merged

ReubenBond merged 10 commits into
dotnet:mainfrom
ReubenBond:rb-fix-firestore-membership-consistency

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Preserve Firestore canonical membership consistency while keeping membership operations lightweight.

  • Read each membership view through one native strong snapshot: a single RunQuery stream for the full table, or one BatchGetDocuments request for the version and requested row. Each membership tuple contains that entry's own document ETag; TableVersion carries the version document's separate ETag.
  • Apply canonical mutations with one atomic native Commit, enforcing the table-version CAS and row creation/existence precondition. Full-row updates use the supplied table version; the row ETag remains provider metadata, so heartbeat-only changes preserve original-input update usability.
  • Write the owning silo's heartbeat with one timestamp-only field update. Canonical membership fields and the table version remain stable; the updated row's document ETag can change. Caller cancellation is normalized and native faults propagate.
  • Select Dead rows once with a native status query, evaluate the strict cutoff against the latest start/heartbeat/suspect timestamp, and reuse conditional delete batches of up to 500 rows. Only a conflicted batch is refreshed under native transaction protection; cleanup preserves the canonical version.
  • Cover request/document counts, per-entry ETags, streamed snapshot consumption, canonical conflicts, original-input updates after heartbeats, native cancellation/errors, selected-batch cleanup races, and cross-batch error propagation with native SDK mocked regressions and emulator tests.

The read strategy follows Firestore's native snapshot-read guarantees; Commit writes are atomic with an optional transaction token. This removes redundant client transaction round trips while retaining native atomicity and preconditions.

Includes only the shared legacy cleanup-test prerequisite from #11308 (bede893d290a9035f196f4c3ed58213d102efc6c), cherry-picked as a separate commit. The provider changes are carved from the frozen conformance source and adapted to unchanged-version cleanup and sole-owner, canonically noninterfering heartbeats.

Microsoft Reviewers: Open in CodeFlow

Copilot AI lite review requested due to automatic review settings September 17, 2026 22:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Update-time timestamp comparisons reject valid preconditions, causing writes and regression tests to fail.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity

Open (1)
What changed in this PR

This pull request preserves Firestore membership consistency across cleanup, updates, and heartbeats.

Changes:

  • Adds transactional, race-safe cleanup.
  • Preserves heartbeat monotonicity and handles retired rows.
  • Adds focused Firestore regression tests.
File Summary
test/​Orleans.Runtime.Internal.Tests/​MembershipTests/​MembershipTableTestsBase.cs Updates shared cleanup expectations.
test/​Extensions/​Orleans.Clustering.Firestore.Tests/​FirestoreMembershipHeartbeatTests.cs Adds Firestore consistency and concurrency regression tests.
src/​Google/​Orleans.Clustering.Firestore/​FirestoreMembershipTable.cs Adds transactional cleanup and consistency safeguards.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 17, 2026 23:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Two moderate issues remain in cleanup transaction scope and mocked timestamp comparison.

Review effort: Lite
Findings: 1 High severity

Open (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Cleanup transaction locks unrelated membership rows

src/​Google/​Orleans.Clustering.Firestore/​FirestoreMembershipTable.cs:79

This transaction reads the entire collection, so Firestore's server-side transaction tracks/locks the version document and every active row even though only stale Dead rows can be deleted. A heartbeat or membership update on any of those documents can abort the cleanup (or block until the transaction timeout), making periodic cleanup unreliable in a busy cluster. Restrict the transactional query to Dead rows and limit it to the deletion batch while retaining the per-document update-time preconditions.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request
Lines 82.26% (112,554 / 136,825)
Branches 71.51% (32,461 / 45,391)

Report-only conclusion: current-main baseline stale.

The newest successful coverage run tested 986e5c7, not current main dcf2925.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI review requested due to automatic review settings September 18, 2026 14:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved review comments remain, and focused regression coverage is included.

Review effort: Lite
Findings: None

Resolved since last review (1)

@ReubenBond
ReubenBond marked this pull request as ready for review September 18, 2026 15:11
Copilot AI review requested due to automatic review settings September 18, 2026 15:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Address cancellation translation for heartbeat updates and avoid rescanning all rows during batched cleanup.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)

Copilot AI review requested due to automatic review settings September 18, 2026 16:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A critical test-compilation issue and two moderate implementation issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 2 Medium severity

Open (3)

Copilot AI review requested due to automatic review settings September 18, 2026 16:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cleanup should query only dead rows to avoid live-heartbeat transaction contention and retry exhaustion.

Review effort: Lite
Findings: None

Resolved since last review (3)

Copilot AI review requested due to automatic review settings September 18, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified.

Review effort: Lite
Findings: None

@ReubenBond

Copy link
Copy Markdown
Member Author

Addressed the cleanup observation in review 5250118055 with 690e485.

Cleanup now sends a native Status == Dead equality query before the transactional document reads. Previously the query read the whole collection and only then filtered statuses in memory, unnecessarily adding live heartbeat rows and the version document to its read set. Dead-row compaction preserves the canonical version, so those reads are unnecessary.

Candidate recency still uses the strict cutoff over the latest start, heartbeat, and suspect timestamps. Transactional candidate reads and per-document update-time delete preconditions still protect changes to a candidate's status, heartbeat, or votes. Public ReadAll/ReadRow snapshot transactions are unchanged. The equality filter uses the existing Status field and normal single-field query support; no schema/layout or composite-index changes were added. Native Firestore locking and contention on matching candidates can still occur.

Two native-SDK fake race cases changed a live heartbeat, or live canonical fields plus the version, while cleanup committed. Both failed against the broad query due to an additional retry and pass with the native filter. The tests also assert the exact StructuredQuery predicate and retain all cutoff/candidate-race cases.

Verified on this exact head in CI run 35369437102: both Google Cloud jobs passed. Downloaded TRX results show 102/102 Firestore clustering tests passing on each of net8.0 and net10.0, including 52 native-SDK cases, Dead-row cleanup, and cleanup spanning multiple write batches.

Copilot AI review requested due to automatic review settings September 18, 2026 17:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The cross-cutting Firestore consistency and cleanup changes require final human review.

Review effort: Lite
Findings: None

@ReubenBond

ReubenBond commented Sep 18, 2026 •

Copy link
Copy Markdown
Member Author

ETag correction (05b5444): Read tuples now return each membership document's own physical ETag, separately from the table-version ETag. Canonical updates ignore the row ETag and use only table-version CAS plus native row existence, so an original-input update remains usable after a heartbeat. This supersedes the logical-row-token encoding/agreement statements in the historical reassessment below; request counts and native snapshot guarantees are unchanged.

Active Firestore reassessment

Implemented in c305598, after auditing the membership provider, its shared storage helpers, and gateway reads against freshly fetched main (0f9537dde1c10107d059fc10f1171f6f04a43f19). Those baseline files are identical to the PR's original d515d75e base.

Native guarantees established first

  • Firestore's read architecture documents timestamp-based, strongly consistent standalone reads across splits, with no read locks. This is an actual read-boundary guarantee, not an inference from atomic writes.
  • The RunQuery and BatchGetDocuments protocol default to strong consistency. Their stream contracts keep earlier results valid through subsequent read timestamps. Firestore SDK 4.3.0 Query.GetSnapshotAsync consumes one RunQuery stream; FirestoreDb.GetAllSnapshotsAsync consumes one BatchGet request and restores requested document order. These paths do not paginate with separate requests.
  • Native Commit applies all writes atomically and in order; its transaction token is optional. Document existence and update-time preconditions remain enforced by that atomic commit.

Request and document-read costs

Nominal successful-attempt counts below exclude authentication/transport traffic and SDK retries. N = membership rows, D = Dead rows, K = eligible Dead rows, K_main = the baseline cleanup's broader set of old non-Active rows, B = ceil(K/500).

Method main requests Previous PR requests (690e4854) Reassessed requests Document reads / writes
ReadRowAsync 4: Begin + 2 point reads + Commit 4 1 BatchGet Version + requested row in one strong snapshot; 2 requested documents
ReadAllAsync 3: Begin + query + Commit 3 1 RunQuery N+1 documents; one strong stream
InsertRowAsync 2: Begin + Commit 2 1 Commit 0 client document reads; 2 atomic conditional writes
UpdateRowAsync 2: Begin + Commit 2 1 Commit 0 client document reads; row-exists + version CAS; mismatched logical tokens return contention before a request
UpdateIAmAliveAsync 3: Begin + point read + Commit 1 1 field update 0 client reads; one timestamp field
CleanupDefunctSiloEntriesAsync 1 all-row query + ceil(K_main/500) commits 3(B+1), with repeated Dead scans 1 Dead query + B commits D documents once; up to K conditional deletes
InitializeMembershipTableAsync 1 group read + optional header creation + optional version creation Same Same Preserves existing shared bootstrap metadata/layout and native initialization failures
DeleteMembershipTableEntriesAsync 1 scoped query + chunked commits; another-cluster path also initializes shared group Same Same Existing terminal deletion scope and bootstrap behavior retained
Gateway GetGateways 1 Active-row query Same Same Existing native strong query already matches its boundary

Cleanup retains main's concurrent batch dispatch through Task.WhenAll. Successful batches get no refresh and no final empty query. A stale/missing candidate or native aborted commit refreshes only that selected batch (at most 500 references) inside the existing native transaction mechanism; it then re-evaluates Dead status and the exclusive max(start, heartbeat, suspect-time) cutoff before conditional deletion. With one uncontended refresh this adds Begin + one BatchGet + Commit for that batch, rather than rescanning the collection. Further commit contention follows the SDK's bounded transaction retry policy. Permission/network/resource failures remain observable, including a different batch's permission failure while another batch handles contention.

Retained boundaries and concrete evidence

  • FirestoreMembershipTable.cs:126,162: public reads use native strong snapshots, with missing-version errors preserved.
  • FirestoreMembershipTable.cs:192,234,348: native atomic writes retain canonical version CAS, row creation/existence, and logical-token agreement; version-document ETags remain independent of heartbeat physical timestamps.
  • FirestoreMembershipTable.cs:73: cleanup retains physical candidate update-time predicates because eligibility must still hold at deletion. Only a conflicted selected batch needs a read/write transaction.
  • FirestoreMembershipTable.cs:40,57 and FirestoreDataManager.cs:53,88: shared group bootstrap and scoped terminal deletion remain unchanged. The header read preserves the existing startup/storage-layout behavior; it is not a canonical snapshot fence.
  • FirestoreGatewayListProvider.cs:41: the existing single native query already provides the required gateway read.
  • FirestoreDataManager.cs:439: the existing generic cancellation adapter is now internal for reuse; its implementation and other providers' behavior are unchanged.

All normal public reads and canonical writes now have zero explicit client transactions. The server still coordinates atomic writes internally, and the canonical version document remains a genuine serialization point. This is not a claim of zero backend locking or contention. GAX's existing Commit retries for Unavailable/ResourceExhausted and BatchGet's transient-read retries remain native policy; no provider heartbeat/CAS loop was added.

Tests which forced raw heartbeat clobbering were removed from canonical-token scenarios: lag/overwrite is permitted, not required. No compatibility flags, schema sidecars, markers, rollout protocol, or public API changes were introduced.

Scan checklist and validation

Bounded scan of the three production files: 0 hits for async-void, sync-over-async, Task.Run, ValueTask misuse candidates, BlockingCollection, substring/string-culture conversions, string replacements, params arrays, temporary byte buffers, stackalloc, string.Format, regex constructors/compiled regex, ContainsKey double-lookup candidates, JSON options/serialization, HttpClient construction, static Dictionary/FrozenDictionary, per-call List/Dictionary allocation, and culture comparers. Nine LINQ candidate lines and three unsealed internal classes (0/3 sealed) were inspected; network-operation costs warranted the changes above, not unrelated micro-optimizations. Existing native query/batch usage in gateway reads and shared clear/delete helpers provided local prior art.

Focused local build/tests: 76/76 passed. Exact-head CI run 35374375696: both Google Cloud jobs passed. Downloaded TRX files confirm 126/126 Firestore clustering tests on each of net8.0 and net10.0, including streamed snapshot changes, out-of-order BatchGet responses, 500+1 cleanup batches, targeted refresh, genuine canonical conflicts, native cancellation, cross-batch failure propagation, and existing emulator concurrent-read/update tests. Latest inline-thread audit is clear; the current-head automated review reports no findings and calls for human review. Remaining overall CI jobs were still running at this report; no failed checks were present.

Assessment Count Outcome
Critical findings 0 None identified in the bounded audit
Moderate native-cost findings 3 Removed redundant read transactions, write-only transactions, and repeated cleanup scans
Unmeasured micro-optimization recommendations 0 No unrelated tuning proposed

This is an AI-assisted static/native-operation assessment, not a throughput benchmark. Request counts and correctness checks are verified; production performance and the final design still require human review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The concurrency-sensitive Firestore changes warrant final human review.

Review effort: Lite
Findings: None

Copilot AI review requested due to automatic review settings September 18, 2026 20:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical issues remain with snapshot consistency and the missing row ETag precondition.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity

Open (2)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The broad provider and consistency changes warrant final human review.

Review effort: Lite
Findings: None

Resolved since last review (2)

@ReubenBond
ReubenBond merged commit a5d7ae6 into dotnet:main Sep 18, 2026
73 checks passed
@ReubenBond
ReubenBond deleted the rb-fix-firestore-membership-consistency branch September 18, 2026 22:06
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants