Skip to content

test(firestore): investigate rejected writer during concurrent row-read conformance #11402

Open

Description

@ReubenBond

Failure

The Google Cloud provider .NET 10 CI job for #10693 failed in Orleans.Clustering.Firestore.Tests.FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews.

  • Job: https://github.com/dotnet/orleans/actions/runs/37133775317/job/111233985684
  • PR head: d74a614a5745fc460dbe411cc5b1fbfd5f25fd8b
  • Actual checked-out/tested merge: 4bfa449f52e0c77f31fb2246e3edb3681229628a, merging that head into main 44b9905b5a0a62c2bd7c74f43a84446af161f20e.
  • Tested tree: 3302174b29160ba92f648f16315c87514b0145c4, identical to the PR head tree.
  • Artifact: test_output_test-google_net10.0, ID 11278020979.
  • Firestore membership TRX: 155 executed, 154 passed, 1 failed; SHA-256 5A9758EBC9682C74129DF08194DEF3FA07D5AF28A38A6D3E45523F2F3F0E117C.
Membership conformance: provider=FirestoreMembershipTableTests;
guarantee=ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews;
cluster=ctk-a-3184bb68c9b44ad5ac7897ba89c21b35;
handle=A1/A2/B1; seed=17; controlled atomic-read writer failed

MembershipTableTestRunner.<>c__DisplayClass59_2.<<ConcurrentReads>g__Writer|1>d.MoveNext()
  MembershipTableTestRunner.cs:723
MembershipTableTestRunner.ConcurrentReads(Boolean pointReads, CancellationToken ct)
  MembershipTableTestRunner.cs:765

The failing check asserts that B.UpdateRowAsync(target, before.Row(target.SiloAddress).Etag, before.Next(), ct) returns true. The scenario releases one canonical writer and three point readers together after obtaining the committed snapshot and its tokens. The writer returned false. This result identifies a rejected write; the available assertion does not establish the underlying reason for rejection.

The other Firestore assemblies in the same artifact passed: grain directory 17/17, persistence 37/37, reminders 43/43.

Investigation

Investigate why the controlled canonical update is rejected while concurrent transactional point reads run, and capture the provider/emulator transaction outcome and conflicting tokens when the write returns false. Preserve the atomic committed-view assertions.

This failure is outside the Durable Messaging changes. The current PR's main merge resolved only two documentation package-exception conflicts; its Firestore implementation is inherited from main.

Related work

#11397 tracked a different symptom: a torn table-version/row snapshot. Its fix, #11400, is already included in the actual tested revision (44b9905b5a0a62c2bd7c74f43a84446af161f20e is the #11400 merge commit). Rebasing onto that fix would repeat the same code. #11342 and #11311 are also inherited.

#11389 is open and proposes retiring point-row reads in favor of full snapshots; it is not a landed fix for this failure. Searches for the exact scenario, controlled atomic-read writer, and Firestore issues found no existing issue tracking this rejected-writer outcome.

Activity

  1. ReubenBond commented on Oct 5, 2026

    @ReubenBond
    MemberAuthor

    Resolved by #11389, which retired membership point reads and replaced the reported concurrent point-read conformance scenario with full-snapshot reads. The merged PR's Google Cloud CI exercised the Firestore test assembly on net8.0 and net10.0 with zero failures or skips and no retry. No duplicate patch or PR is needed. If the surviving ConcurrentReadAll_ReturnsOnlyAtomicCommittedViews scenario later reproduces a rejected writer, capture the Firestore RPC status and row/table tokens as documented in the attached investigation.

  2. ReubenBond commented on Oct 21, 2026

    @ReubenBond
    MemberAuthor

    This exact failure recurred on the 10.x maintenance backport PR #11566 at ab49fd15df7cf1df624bedd4a271ae9b6d7d253c.

    • Failed job: https://github.com/dotnet/orleans/actions/runs/39687591660/job/119525478704
    • Test: FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews
    • Evidence: seed=17; controlled atomic-read writer failed at the existing UpdateRowAsync success assertion.
    • Membership assembly: 154 passed, 1 failed, 0 skipped. The other Google assemblies passed (17 directory, 37 persistence, 43 reminders).
    • The previous PR head 957771c27d passed both Google framework jobs with identical Firestore production code, conformance runner, and Firestore tests. The current net8.0 job passed too.

    The 10.x branch deliberately retains ReadRowAsync; #11389's first-party NotSupportedException behavior is outside this tactical patch. Therefore closing this issue on main does not establish that the preserved 10.x path is fixed.

    The artifact identifies a rejected writer, not a torn snapshot. Its native RPC status is absent, so transaction-lock contention remains a hypothesis. Atomic-view assertions and provider semantics are unchanged. The failed jobs are being rerun on the same commit as attempt 2; this comment records the recurrence independently of that retry's outcome.

  3. ReubenBond commented on Oct 22, 2026

    @ReubenBond
    MemberAuthor

    Reopening for the retained 10.x membership point-read path: the same rejected-writer conformance failure has now recurred on both target frameworks of release PR #11566.

    Latest occurrence on head 64d09c36a4136dbf0ac47376bde3beec6929b07e:

    • Job: https://github.com/dotnet/orleans/actions/runs/39832555515/job/120012230837 (Google Cloud providers, net8.0)
    • FirestoreMembershipTableTests.ConcurrentReadRow_ReturnsOnlyAtomicCommittedViews
    • seed=17; controlled atomic-read writer failed: UpdateRowAsync returned false while concurrent transactional readers ran.
    • Membership: 154 passed, 1 failed, 0 skipped. Other Google assemblies: directory17/17, persistence37/37, reminders43/43.
    • The same head's net10.0 Google job passed. Prior head a54d67fd39 passed both Google jobs with identical Firestore and conformance code; the latest delta only removes migration TTL compensation.

    The earlier main fix #11389 retires first-party point reads by throwing NotSupportedException, so it does not fix the behavior intentionally retained by the 10.x patch. The earlier net10.0 recurrence is recorded above. No duplicate issue is needed, but this retained path still warrants investigation.

    The artifact does not expose the native RPC status mapped to false, so the underlying rejection reason remains unproven. Future diagnosis should capture that status plus row/table tokens and distinguish backend transaction abort from canonical-precondition contention, preserving atomic-read assertions. No assertion weakening, skipped test, blind provider retry, or breaking point-read retirement has been added. The failed job is being rerun on the unchanged head.

  4. ReubenBond commented on Oct 22, 2026

    @ReubenBond
    MemberAuthor

    The latest net8.0 recurrence passed on an unchanged-head job rerun (run39832555515, attempt2): https://github.com/dotnet/orleans/actions/runs/39832555515/job/120028402082 . Fresh TRX evidence: membership155/155, directory17/17, persistence37/37, reminders43/43; total252passed,0failed,0skipped. The CI aggregate gate passed too. This confirms intermittent behavior, not a root-cause fix; keep the reopened10.x investigation active.

  5. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The retained 10.x point-read path reproduced the same rejected-writer failure on #11593, head 2456226c69d17b0356113b55bb9d19f57b5732d5.

    The backport changes runtime membership observation and leaves the Firestore provider and conformance runner unchanged. The Google net8.0 job passed. No underlying RPC status is exposed in the failure, so this is evidence of recurrence, not a root-cause diagnosis. I will rerun the failed job on the unchanged head after the workflow finishes; assertions and provider behavior remain unchanged.

  6. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The #11593 occurrence passed on the unchanged-head job rerun (run 40572704345, attempt 2): https://github.com/dotnet/orleans/actions/runs/40572704345 . All checks for head 2456226c69d17b0356113b55bb9d19f57b5732d5 are green following the rerun. This confirms intermittency; the 10.x rejected-writer investigation remains open.

  7. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The same rejected-writer flake also reproduced on #11593 final compatibility-fix head 7c52e342de2e8badd5bd658ad776d745d1dfbe34:

    The compatibility-fix commit changes snapshot death-history defaults and bidirectional wire tests; Firestore production code and its conformance runner remain unchanged. I am retaining the failure evidence and rerunning only the failed job once its workflow completes. The earlier head passed the same job on its unchanged-head retry; this remains an intermittent rejected-writer investigation, not a diagnosed provider fix.

  8. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The retained 10.x point-read path reproduced the same rejected-writer failure on #11598, head 030e8967171e346c3daf6537449a34b3bd5380c1, on both target frameworks:

    This backport changes only ConsistentRingProvider hash ordering/lookup and its focused tests; the Firestore provider and conformance runner are unchanged. The logs do not expose the underlying RPC status, so this records recurrence rather than a root-cause diagnosis. The existing main point-read retirement remains outside this 10.x patch. I will rerun the failed jobs on the unchanged head after the workflow finishes, preserving the conformance assertions and provider behavior.

  9. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The retained 10.x point-read path reproduced this same rejected-writer failure on client startup cancellation backport #11605, head 0f270f6ea4e99fdba0c0cd1fd22f2295020386d7.

    The backport changes client startup cancellation and its focused regressions; the Firestore implementation and conformance runner are unchanged. The Google net10.0 job passed. The log does not expose the native RPC status, so this records recurrence without assigning a root cause. The main point-read retirement remains outside this compatibility-preserving 10.x backport. I will rerun the failed job on the unchanged head after the workflow completes, preserving provider behavior and assertions.

  10. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The #11598 backport's fresh full CI run is now green on naming-only head 74474ae7b387d766ce80d1b5cbc1b9a54c4720e9, whose tree is identical to the earlier failing head 030e8967171e346c3daf6537449a34b3bd5380c1.

    Both Google Cloud provider partitions passed:

    The complete .NET CI workflow also passed. This preserves the failed-run evidence above and confirms intermittent behavior on unchanged code; the retained 10.x rejected-writer investigation remains open.

  11. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    Final rebased #11593 head 99f84b464f5eb7cf4f43567ed3fb3140b5c08ed2 reproduced the same conformance failure:

    The PR has now merged. Its membership patches are unchanged from the previously validated tree; the base advance adds Kubernetes elected-agent monitoring, leaving Firestore and the conformance runner unchanged. The earlier original-head occurrence passed on its unchanged-head job rerun. No assertion weakening or provider retry was introduced; the retained 10.x point-read rejected-writer investigation remains open.

  12. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The client startup cancellation backport #11605 reproduced the same retained-10.x rejected-writer failure after its follow-up cleanup fixes, on current head 4601ee396d150fe81ac195696c3727241c9c5397:

    The preceding backport head 39ae2ac898b87a2d89dbd5a358f71f6d7dafe9ea also reproduced it on net8.0: https://github.com/dotnet/orleans/actions/runs/40587248296/job/122481885002 . That obsolete workflow was cancelled when the gateway-refresh shutdown fix was published.

    The current delta adds cancellation of gateway-manager refresh waits and its regression; Firestore production and conformance code remain unchanged. I will rerun the failing provider job on the unchanged current head once the workflow completes. The native RPC status is absent from the log, so these failures remain recurrence evidence rather than a diagnosed root cause.

  13. ReubenBond commented on Oct 27, 2026

    @ReubenBond
    MemberAuthor

    The current #11605 occurrence passed on its unchanged-head isolated job rerun (run 40590513170, attempt 2, head 4601ee396d150fe81ac195696c3727241c9c5397): https://github.com/dotnet/orleans/actions/runs/40590513170/job/122508282171 . The Google net10.0 provider partition completed with 252 passed, 0 failed, 0 skipped. The dependent CI gate and replacement .NET Test Results check are green; the aggregate report has 55,244 passed, 0 failed, 274 skipped. All current-head PR checks are successful except the intentional GitHub Pages deployment skip. This confirms intermittency; the retained-10.x rejected-writer investigation remains open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions