Repository navigation
Description
Failure
The Google Cloud provider .NET 10 CI job for #10693 failed in Orleans.Clustering.Firestore.Tests.FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews.
- Job: https://github.com/dotnet/orleans/actions/runs/37133775317/job/111233985684
- PR head:
d74a614a5745fc460dbe411cc5b1fbfd5f25fd8b - Actual checked-out/tested merge:
4bfa449f52e0c77f31fb2246e3edb3681229628a, merging that head into main44b9905b5a0a62c2bd7c74f43a84446af161f20e. - Tested tree:
3302174b29160ba92f648f16315c87514b0145c4, identical to the PR head tree. - Artifact:
test_output_test-google_net10.0, ID11278020979. - Firestore membership TRX: 155 executed, 154 passed, 1 failed; SHA-256
5A9758EBC9682C74129DF08194DEF3FA07D5AF28A38A6D3E45523F2F3F0E117C.
Membership conformance: provider=FirestoreMembershipTableTests;
guarantee=ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews;
cluster=ctk-a-3184bb68c9b44ad5ac7897ba89c21b35;
handle=A1/A2/B1; seed=17; controlled atomic-read writer failed
MembershipTableTestRunner.<>c__DisplayClass59_2.<<ConcurrentReads>g__Writer|1>d.MoveNext()
MembershipTableTestRunner.cs:723
MembershipTableTestRunner.ConcurrentReads(Boolean pointReads, CancellationToken ct)
MembershipTableTestRunner.cs:765
The failing check asserts that B.UpdateRowAsync(target, before.Row(target.SiloAddress).Etag, before.Next(), ct) returns true. The scenario releases one canonical writer and three point readers together after obtaining the committed snapshot and its tokens. The writer returned false. This result identifies a rejected write; the available assertion does not establish the underlying reason for rejection.
The other Firestore assemblies in the same artifact passed: grain directory 17/17, persistence 37/37, reminders 43/43.
Investigation
Investigate why the controlled canonical update is rejected while concurrent transactional point reads run, and capture the provider/emulator transaction outcome and conflicting tokens when the write returns false. Preserve the atomic committed-view assertions.
This failure is outside the Durable Messaging changes. The current PR's main merge resolved only two documentation package-exception conflicts; its Firestore implementation is inherited from main.
Related work
#11397 tracked a different symptom: a torn table-version/row snapshot. Its fix, #11400, is already included in the actual tested revision (44b9905b5a0a62c2bd7c74f43a84446af161f20e is the #11400 merge commit). Rebasing onto that fix would repeat the same code. #11342 and #11311 are also inherited.
#11389 is open and proposes retiring point-row reads in favor of full snapshots; it is not a landed fix for this failure. Searches for the exact scenario, controlled atomic-read writer, and Firestore issues found no existing issue tracking this rejected-writer outcome.
Activity
ReubenBond commentedon Oct 5, 2026
Resolved by #11389, which retired membership point reads and replaced the reported concurrent point-read conformance scenario with full-snapshot reads. The merged PR's Google Cloud CI exercised the Firestore test assembly on net8.0 and net10.0 with zero failures or skips and no retry. No duplicate patch or PR is needed. If the surviving ConcurrentReadAll_ReturnsOnlyAtomicCommittedViews scenario later reproduces a rejected writer, capture the Firestore RPC status and row/table tokens as documented in the attached investigation.
ReubenBond commentedon Oct 21, 2026
This exact failure recurred on the 10.x maintenance backport PR #11566 at
ab49fd15df7cf1df624bedd4a271ae9b6d7d253c.FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedat the existingUpdateRowAsyncsuccess assertion.957771c27dpassed both Google framework jobs with identical Firestore production code, conformance runner, and Firestore tests. The current net8.0 job passed too.The 10.x branch deliberately retains
ReadRowAsync; #11389's first-partyNotSupportedExceptionbehavior is outside this tactical patch. Therefore closing this issue on main does not establish that the preserved 10.x path is fixed.The artifact identifies a rejected writer, not a torn snapshot. Its native RPC status is absent, so transaction-lock contention remains a hypothesis. Atomic-view assertions and provider semantics are unchanged. The failed jobs are being rerun on the same commit as attempt 2; this comment records the recurrence independently of that retry's outcome.
ReubenBond commentedon Oct 22, 2026
Reopening for the retained 10.x membership point-read path: the same rejected-writer conformance failure has now recurred on both target frameworks of release PR #11566.
Latest occurrence on head
64d09c36a4136dbf0ac47376bde3beec6929b07e:FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failed:UpdateRowAsyncreturned false while concurrent transactional readers ran.a54d67fd39passed both Google jobs with identical Firestore and conformance code; the latest delta only removes migration TTL compensation.The earlier main fix #11389 retires first-party point reads by throwing NotSupportedException, so it does not fix the behavior intentionally retained by the 10.x patch. The earlier net10.0 recurrence is recorded above. No duplicate issue is needed, but this retained path still warrants investigation.
The artifact does not expose the native RPC status mapped to false, so the underlying rejection reason remains unproven. Future diagnosis should capture that status plus row/table tokens and distinguish backend transaction abort from canonical-precondition contention, preserving atomic-read assertions. No assertion weakening, skipped test, blind provider retry, or breaking point-read retirement has been added. The failed job is being rerun on the unchanged head.
ReubenBond commentedon Oct 22, 2026
The latest net8.0 recurrence passed on an unchanged-head job rerun (run39832555515, attempt2): https://github.com/dotnet/orleans/actions/runs/39832555515/job/120028402082 . Fresh TRX evidence: membership155/155, directory17/17, persistence37/37, reminders43/43; total252passed,0failed,0skipped. The CI aggregate gate passed too. This confirms intermittent behavior, not a root-cause fix; keep the reopened10.x investigation active.
ReubenBond commentedon Oct 27, 2026
The retained 10.x point-read path reproduced the same rejected-writer failure on #11593, head
2456226c69d17b0356113b55bb9d19f57b5732d5.FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failed(UpdateRowAsyncreturned false).The backport changes runtime membership observation and leaves the Firestore provider and conformance runner unchanged. The Google net8.0 job passed. No underlying RPC status is exposed in the failure, so this is evidence of recurrence, not a root-cause diagnosis. I will rerun the failed job on the unchanged head after the workflow finishes; assertions and provider behavior remain unchanged.
ReubenBond commentedon Oct 27, 2026
The #11593 occurrence passed on the unchanged-head job rerun (run 40572704345, attempt 2): https://github.com/dotnet/orleans/actions/runs/40572704345 . All checks for head
2456226c69d17b0356113b55bb9d19f57b5732d5are green following the rerun. This confirms intermittency; the 10.x rejected-writer investigation remains open.ReubenBond commentedon Oct 27, 2026
The same rejected-writer flake also reproduced on #11593 final compatibility-fix head
7c52e342de2e8badd5bd658ad776d745d1dfbe34:FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedThe compatibility-fix commit changes snapshot death-history defaults and bidirectional wire tests; Firestore production code and its conformance runner remain unchanged. I am retaining the failure evidence and rerunning only the failed job once its workflow completes. The earlier head passed the same job on its unchanged-head retry; this remains an intermittent rejected-writer investigation, not a diagnosed provider fix.
ReubenBond commentedon Oct 27, 2026
The retained
10.xpoint-read path reproduced the same rejected-writer failure on #11598, head030e8967171e346c3daf6537449a34b3bd5380c1, on both target frameworks:FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedThis backport changes only
ConsistentRingProviderhash ordering/lookup and its focused tests; the Firestore provider and conformance runner are unchanged. The logs do not expose the underlying RPC status, so this records recurrence rather than a root-cause diagnosis. The existing main point-read retirement remains outside this10.xpatch. I will rerun the failed jobs on the unchanged head after the workflow finishes, preserving the conformance assertions and provider behavior.ReubenBond commentedon Oct 27, 2026
The retained
10.xpoint-read path reproduced this same rejected-writer failure on client startup cancellation backport #11605, head0f270f6ea4e99fdba0c0cd1fd22f2295020386d7.FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedThe backport changes client startup cancellation and its focused regressions; the Firestore implementation and conformance runner are unchanged. The Google net10.0 job passed. The log does not expose the native RPC status, so this records recurrence without assigning a root cause. The main point-read retirement remains outside this compatibility-preserving 10.x backport. I will rerun the failed job on the unchanged head after the workflow completes, preserving provider behavior and assertions.
ReubenBond commentedon Oct 27, 2026
The #11598 backport's fresh full CI run is now green on naming-only head
74474ae7b387d766ce80d1b5cbc1b9a54c4720e9, whose tree is identical to the earlier failing head030e8967171e346c3daf6537449a34b3bd5380c1.Both Google Cloud provider partitions passed:
The complete .NET CI workflow also passed. This preserves the failed-run evidence above and confirms intermittent behavior on unchanged code; the retained
10.xrejected-writer investigation remains open.ReubenBond commentedon Oct 27, 2026
Final rebased #11593 head
99f84b464f5eb7cf4f43567ed3fb3140b5c08ed2reproduced the same conformance failure:FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedThe PR has now merged. Its membership patches are unchanged from the previously validated tree; the base advance adds Kubernetes elected-agent monitoring, leaving Firestore and the conformance runner unchanged. The earlier original-head occurrence passed on its unchanged-head job rerun. No assertion weakening or provider retry was introduced; the retained 10.x point-read rejected-writer investigation remains open.
ReubenBond commentedon Oct 27, 2026
The client startup cancellation backport #11605 reproduced the same retained-10.x rejected-writer failure after its follow-up cleanup fixes, on current head
4601ee396d150fe81ac195696c3727241c9c5397:FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViewsseed=17; controlled atomic-read writer failedThe preceding backport head
39ae2ac898b87a2d89dbd5a358f71f6d7dafe9eaalso reproduced it on net8.0: https://github.com/dotnet/orleans/actions/runs/40587248296/job/122481885002 . That obsolete workflow was cancelled when the gateway-refresh shutdown fix was published.The current delta adds cancellation of gateway-manager refresh waits and its regression; Firestore production and conformance code remain unchanged. I will rerun the failing provider job on the unchanged current head once the workflow completes. The native RPC status is absent from the log, so these failures remain recurrence evidence rather than a diagnosed root cause.
ReubenBond commentedon Oct 27, 2026
The current #11605 occurrence passed on its unchanged-head isolated job rerun (run
40590513170, attempt 2, head4601ee396d150fe81ac195696c3727241c9c5397): https://github.com/dotnet/orleans/actions/runs/40590513170/job/122508282171 . The Google net10.0 provider partition completed with 252 passed, 0 failed, 0 skipped. The dependent CI gate and replacement .NET Test Results check are green; the aggregate report has 55,244 passed, 0 failed, 274 skipped. All current-head PR checks are successful except the intentional GitHub Pages deployment skip. This confirms intermittency; the retained-10.x rejected-writer investigation remains open.