Skip to content

ci: hosted runner loses communication during DynamoDB net8.0 tests #11331

Closed

Description

@ReubenBond

Failure

The AWS DynamoDB provider tests (net8.0) job failed because its hosted runner lost communication with GitHub, before completing the test step or publishing its test artifacts.

The check annotation states:

The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Available evidence

The job started at 2026-09-18T21:46:29Z and was marked failed at 22:41:30Z. Checkout and Start DynamoDB Local succeeded. The Test step started at 21:46:42Z, but its REST metadata still reports in_progress with no completion timestamp; cleanup and post-job steps remain pending.

gh run view --job 105772493932 --log-failed reports log not found. The run has DynamoDB net10.0 output/coverage artifacts, but no corresponding net8.0 artifact. No test assertion or specific test failure could be recovered from the available evidence.

Scope and investigation

The triggering changes concern durable Messaging/Journaling rather than the DynamoDB provider. This is currently classified as a hosted-runner failure with an unlocalized cause, not a confirmed DynamoDB product regression. The annotation alone does not distinguish process termination, CPU/memory starvation, or network loss.

Please investigate runner diagnostics/resource exhaustion and whether an active test or container operation prevented completion and artifact upload. Exact-message and DynamoDB runner/hang searches found no existing matching issue or recently merged fix. No workflow rerun or branch rebase was performed to replace the original failure evidence.

Activity

  1. ReubenBond commented on Sep 19, 2026

    @ReubenBond
    MemberAuthor

    Another occurrence of the same hosted-runner communication-loss annotation, on a different workload/platform:

    • PR#11332 head 1204d79edcbb87a49fa3fd67287344e18edd2500
    • Run35469141901
    • Windows net8 SlowBVT job105966792186
    • Runner GitHub Actions 1006803077
    • Job started2026-09-19T21:01:14Z; marked failed21:51:16Z. Test step started21:01:23Z and still reports in_progress with no completion; post steps remain pending.

    The annotation is exactly “The hosted runner lost communication with the server...” gh run view --log-failed reports log not found; no matching Windows net8 SlowBVT result artifact exists. No specific test assertion or responsible process was recovered, so this is not being attributed to the ZooKeeper-only candidate or asserted to share a root cause with the earlier DynamoDB occurrence.

    No linked merged remedy was found and no unchanged workflow rerun was requested. Preserving the available evidence here rather than opening duplicate unlocalized runner tracking.

  2. ReubenBond commented on Sep 21, 2026

    @ReubenBond
    MemberAuthor

    Completed the end-to-end follow-up. The evidence supports a hang followed by runner disappearance, but it does not localize the initiating fault sufficiently to justify a repository or workflow change.

    Cross-run evidence

    • I scanned 250 recent .NET CI runs, including 162 failed jobs. Only these two jobs have the distinctive state where the active test step remains in_progress, later steps remain pending, and GitHub marks the job failed because the runner lost communication:
    • Both jobs had already exceeded their normal execution time by roughly 35–40 minutes before the runner was declared lost. This is stronger evidence for a stalled test/coordinator or resource-starved host followed by agent loss than for an ordinary test assertion or immediate random disconnect.
    • Both affected partitions target net8.0, but the workloads and operating systems differ. That correlation is notable but not sufficient to attribute the failures to MTP, xUnit, CoreCLR, or the Orleans test harness.

    DynamoDB comparison

    • The same-run net10 job passed in 14.1 minutes.
    • The immediately prior net8 job passed on the same Actions runner version, ubuntu-24.04 image version, and DynamoDB image digest.
    • Later net8 jobs repeatedly passed, including job 105821864641 and job 106167351519. The latest downloaded artifact contains 604/604 passing tests; net10 also contains 604/604.
    • fix(aws): preserve DynamoDB membership consistency #11310 expanded this partition from 425 to 604 discovered tests shortly before the incident, so it increased workload. However, same-run net10 and repeated later net8 executions passed in normal time, so the expansion is not evidence of a deterministic regression.
    • The triggering commit 5e9cafafd49d0654041889e315d5955bc2331fb5 changes only a durable-messaging test and does not touch DynamoDB or CI.

    Independent Windows occurrence

    • The same-run Windows net10 SlowBVT job and the Ubuntu net8/net10 SlowBVT jobs passed. The DynamoDB net8 job in that run also passed 645 tests.
    • The affected Windows net8 SlowBVT partition passed immediately before and after on the same runner/image versions.
    • Its triggering commit 1204d79edcbb87a49fa3fd67287344e18edd2500 changes only ZooKeeper implementation/unit-test files, which are outside the Provider=None SlowBVT partition.
    • The ZooKeeper net8/net10 failures in that run are ordinary, artifact-backed failures and are separately addressed by draft PR fix(zookeeper): make membership reads sequential and resilient #11338; that PR is not a fix for this runner-loss signature.

    Diagnostic boundary and disposition

    • Both job-log REST endpoints return 404 BlobNotFound; no partial terminal output is recoverable.
    • Neither affected partition produced an artifact because the runner disappeared before the always-run archive and cleanup steps could execute. This is not an upload-artifact failure or a Testcontainers lifecycle issue; the DynamoDB job uses a workflow-owned Docker container, and Windows SlowBVT does not share that container path.
    • The shared harness requests full crash/hang dumps with a 10-minute hang threshold. Dump collection or another workload could have amplified resource pressure, but no dump, OOM record, process telemetry, or kernel/runner diagnostic survived. Reducing dump types, adding a job timeout, or changing MTP/xUnit based only on these two losses would be speculative and could remove the diagnostics needed for other known failures.
    • Searches across open and merged PRs found no existing runner-loss fix to advance. Recent MTP lifecycle, dump capture, artifact retry, and retention changes do not cover the boundary where the VM/agent disappears before log/artifact upload.
    • GitHub Status recorded no Actions incident during either window, which rules out a declared platform-wide outage but not an isolated hosted VM failure.

    Disposition: keep this as an evidence-preserving CI reliability issue. No repository/workflow PR is opened because the initiating cause remains unlocalized and every candidate change would be speculative. A recurrence should be correlated with this exact in_progress + pending-post-steps + missing-log-blob signature; actionable next evidence requires off-runner telemetry or a surviving runner diagnostic identifying OOM, dump collection, coordinator/test-host failure, or network loss.

  3. ReubenBond commented on Sep 22, 2026

    @ReubenBond
    MemberAuthor

    Refreshed the investigation after the September 21 merges.

    • Searches across open and merged PRs still find no change which claims or demonstrates a fix for this hang-then-runner-disappearance boundary.
    • fix(zookeeper): make membership reads sequential and resilient #11338 is now merged and makes ZooKeeper membership reads sequential and resilient. test(membership): wire provider conformance suites #11332 is also merged. These address the separate, artifact-backed ZooKeeper failures from the second run; neither identifies or fixes the Windows net8 SlowBVT runner loss recorded here.
    • The only subsequent CI workflow changes are ci: suppress dotnet coverage logo #11340, which suppresses the coverage banner, and ci: remove ZooKeeper failure diagnostics #11341, which removes obsolete ZooKeeper diagnostics. Neither affects net8 hang dumps, test coordinator lifetime, runner resource limits, DynamoDB container lifecycle, or off-runner log/artifact preservation.
    • I inspected the latest 50 .NET CI runs, including 26 failed jobs. There are zero additional jobs with the defining signature: an active step left in_progress, pending post-job steps, and runner communication loss.
    • Recent main and feature-branch CI runs continue to complete the DynamoDB and net8 partitions normally.

    Current disposition remains unchanged: no responsible repository fix is evident, so opening a workflow or product PR would be speculative. The next actionable occurrence needs surviving host/runner evidence which distinguishes a test/MTP stall, dump-induced resource exhaustion, OOM, process termination, or isolated hosted-VM/network loss.

  4. ReubenBond commented on Sep 24, 2026

    @ReubenBond
    MemberAuthor

    September 24 refresh:

    Disposition remains unchanged: current behavior indicates a rare stalled job followed by hosted-agent loss, with no surviving log blob, artifact, OOM record, dump, or process telemetry identifying the initiator. The repository has no demonstrated regression or surgical fix to apply. A code/workflow PR would be speculative; the next actionable evidence requires a recurrence with surviving host or off-runner diagnostics.

  5. ReubenBond commented on Sep 25, 2026

    @ReubenBond
    MemberAuthor

    September 25 refresh:

    Current disposition: the repository reliably runs these partitions in subsequent CI, while the two original runners disappeared after prolonged stalls and before durable diagnostics were uploaded. The available evidence does not establish a product, container-lifecycle, or workflow regression, so there is no deterministic repository fix to implement. Keep this issue as the correlation point for a future recurrence with surviving host or off-runner telemetry.

  6. ReubenBond commented on Sep 26, 2026

    @ReubenBond
    MemberAuthor

    September 26 refresh:

    • I inspected the latest 120 .NET CI runs, including 131 failed jobs. The only job matching the defining stuck-step/runner-loss signature remains the already-recorded September 19 Windows net8 SlowBVT job (105966792186); there are no new occurrences.
    • No open or merged PR since the prior update addresses hosted-runner communication loss, net8 coordinator hangs, resource exhaustion, or preservation of diagnostics after agent/VM disappearance.
    • No relevant change has landed on main in the CI workflow, shared test runner, MTP/xUnit configuration, or DynamoDB test area.
    • The repository continues to complete the affected partitions in subsequent runs. test(membership): wire provider conformance suites #11332/fix(zookeeper): make membership reads sequential and resilient #11338 remain fixes for separate, recoverable ZooKeeper failures and do not explain this signature.

    Disposition: the observed invariant remains a prolonged test step followed by loss of the hosted agent before logs and artifacts can be persisted. Without a surviving OOM record, dump, process trace, or host diagnostic, a repository cause is not established and a workflow/product PR would be speculative. Keep this issue as the correlation point for a future evidence-bearing recurrence.

  7. ReubenBond commented on Sep 27, 2026

    @ReubenBond
    MemberAuthor

    September 27 refresh:

    Disposition remains unchanged: subsequent CI demonstrates normal execution, while the original failures lost their hosted agents only after prolonged stalls and before durable logs/artifacts were persisted. The surviving evidence cannot attribute the initiating fault to Orleans, DynamoDB Local, MTP, dump collection, or the hosted VM. No deterministic repository change is supported, so no skip or speculative PR is warranted.

  8. ReubenBond commented on Sep 28, 2026

    @ReubenBond
    MemberAuthor

    September 28 conclusion:

    The investigation and self-review found no deterministic Orleans-side change which preserves real provider validation and addresses a demonstrated cause. A rubber-duck review challenged the initial “random outage” wording, identified the pre-loss overrun pattern, and prompted the broader historical signature scan; those findings are reflected in this conclusion. It did not establish a defensible code or workflow fix.

    Final disposition: no existing PR fixes #11331, and no repository PR is justified from the available evidence. Keep the issue as the correlation point for a future recurrence with surviving host or off-runner telemetry; do not add a skip or speculative timeout/dump change.

  9. ReubenBond commented on Sep 29, 2026

    @ReubenBond
    MemberAuthor

    September 29 refresh:

    • The latest 180 .NET CI runs, including 148 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
    • No new open or merged PR addresses this signature or its test/MTP/resource boundary, and no relevant main change has landed in CI, shared test execution, MTP/xUnit configuration, or the AWS test partition since the final September 28 review.
    • Subsequent DynamoDB/net8 and unrelated net8 partitions continue to complete normally. No repeated signature or newly merged change establishes that a recent Orleans modification introduced or resolved the two original incidents.

    Actionable diagnostic boundary: a future occurrence must preserve host-level evidence outside the disappearing runner—such as platform runner diagnostics, OOM/process termination telemetry, or another durable external signal—to distinguish a test/coordinator stall from dump-induced resource pressure, network loss, or isolated VM failure. In-run artifact steps cannot provide that evidence after the agent is gone.

    No repository change, skip, broad timeout, or speculative mitigation is justified by the current evidence.

  10. ReubenBond commented on Sep 30, 2026

    @ReubenBond
    MemberAuthor

    September 30 refresh:

    • The latest 200 .NET CI runs, including 153 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
    • Merged test(membership): wire provider conformance suites #11332 adds membership conformance wiring and DynamoDB fixture ownership. Its own description records the Windows net8 runner loss as a separate event without a recovered assertion or root-cause attribution; it does not fix this issue.
    • The only newly opened reliability PR is fix(adonet): prevent SQLite transaction leaks #11354, which fixes a SQLite pooled-connection transaction leak. It is unrelated to DynamoDB, MTP/test coordination, runner resources, or hosted-agent loss.
    • No relevant main change has landed since the prior update in CI, shared test execution, MTP/xUnit configuration, or the AWS test partition.

    The evidence continues to establish the outcome—prolonged test-step stall followed by hosted-agent disappearance—but not the initiating cause. Subsequent affected partitions pass, and no repeated signature or effective PR identifies a repository regression.

    Next action remains evidence acquisition, not mitigation: correlate any future recurrence with durable host/off-runner OOM, process termination, runner, or network telemetry. No skip, broad timeout, dump reduction, or product change is justified without that evidence.

  11. ReubenBond commented on Oct 1, 2026

    @ReubenBond
    MemberAuthor

    October 1 refresh:

    • The latest 220 .NET CI runs, including 159 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
    • New PR ci: mirror pinned provider test images #11356 mirrors/pins CI images to address registry throttling during pre-test image acquisition. For DynamoDB Local it pins sha256:ff89bd48ff32cd8d9be5fee8873b65b8854dc408f1afe881be6eb00247bc0dab on the official registry—the same digest used by the failing job and surrounding passing jobs. The ci: hosted runner loses communication during DynamoDB net8.0 tests #11331 job had already pulled and started that image, passed TCP readiness, and entered the Test step. ci: mirror pinned provider test images #11356 therefore improves deterministic acquisition but does not fix or identify this post-start runner-loss boundary.
    • No other new PR or relevant main change affects MTP/test coordination, hang dumps, runner resources, DynamoDB lifecycle, or diagnostic persistence.

    The available evidence does not support classifying the initiator as purely GitHub infrastructure or purely repository load. It establishes a prolonged test/coordinator stall followed by hosted-agent disappearance. Repository workload or dump collection could have amplified resource pressure, but repeated normal runs on the same image/workload and the independent Windows occurrence leave that unproven; isolated hosted VM/network failure also remains possible.

    Disposition: no existing PR fixes #11331 and no Orleans-side change is justified. #11356 is relevant to image-pull reliability only. The remaining blocker is durable host/off-runner telemetry from a future recurrence.

  12. ReubenBond commented on Oct 3, 2026

    @ReubenBond
    MemberAuthor

    October 3 refresh:

    The evidence continues to distinguish this from image acquisition failures: the #11331 runner had completed DynamoDB pull/start/readiness and was executing tests when it stalled and later disappeared. Subsequent identical provider jobs complete normally.

    Disposition: no repository fix or PR is supported. The blocker remains durable host/off-runner telemetry from a future recurrence; in-run cleanup and artifact upload cannot execute after the hosted agent is lost.

  13. ReubenBond commented on Oct 4, 2026

    @ReubenBond
    MemberAuthor

    A third occurrence now provides a shared, repository-controlled mitigation target:

    • Run 37133134645, Kinesis net10 job 111232095632
    • Ubuntu runner GitHub Actions 1006980211
    • Setup, Ministack, DynamoDB Local, and readiness completed successfully.
    • Test started at 2026-10-03T15:24:52Z, remained in_progress, and the runner was marked lost at 16:13:37Z after 49 minutes.
    • Cleanup/post steps remained pending, the job log endpoint returns 404 BlobNotFound, and only the successful net8 Kinesis artifact exists.
    • Same-run Kinesis net8, DynamoDB net8/net10, and all SlowBVT variants passed. Later runs of the same branch passed both Kinesis frameworks.
    • Triggering commit c98d09767b7e6961c3764d43f3af04e23190ab32 changed only DefaultClientServices.cs, outside Kinesis test infrastructure.

    This occurrence crosses the earlier framework boundary: Linux net10 joins Linux net8 and Windows net8. The common repository-controlled sequence is a stall past the 10-minute inactivity threshold followed by Full hang-dump collection and eventual agent loss. Microsoft documents Full as the largest dump, containing all memory including mapped images; Heap retains threads, stacks, handles, and managed memory while excluding mapped images.

    PR #11405 changes both CI test launch paths to Heap hang dumps, retains Full crash dumps, and adds workflow contract coverage. It preserves test/provider execution and failure semantics while bounding the diagnostic operation which runs during a stall.

  14. ReubenBond commented on Oct 6, 2026

    @ReubenBond
    MemberAuthor

    Closing as mitigated by #11405, merged in cad3eada5e845e1cd0e17df1ea92d9d7a78aa7f2 on October 5.

    The first observed occurrence was the September 18 DynamoDB net8 job recorded above. The surrounding and later identical provider partitions passed, and the triggering change did not touch DynamoDB or CI, so the evidence did not establish a DynamoDB product regression. A September 19 Windows net8 SlowBVT occurrence and the October 3 Kinesis net10 occurrence crossed provider, OS, and framework boundaries. All three shared the repository-controlled sequence of a test stall beyond the 10-minute inactivity threshold followed by Full hang-dump collection and hosted-agent loss before logs, cleanup, or artifacts could persist.

    #11405 changes both standard test launch paths to Heap hang dumps while retaining the 10-minute threshold, Full crash dumps, provider coverage, and failure semantics. Its merge run completed all affected partitions successfully, including DynamoDB net8/net10, Kinesis net8/net10, and SlowBVT across both operating systems/frameworks.

    The first subsequent main failure in DynamoDB net8, run 37385298502/job 112016979267, is useful boundary evidence: the test produced a normal assertion-backed timeout, the Test step completed, cleanup and post steps ran, and the artifact/logs were uploaded. That distinct conformance flake is tracked in #11407. It did not reproduce the runner-loss signature.

    Disposition: the initiating test stalls remain independently diagnosable, while the shared diagnostic resource-pressure amplifier responsible for this issue's lost-runner outcome is bounded. Reopen or correlate a new occurrence if a runner again disappears with the active step left in_progress, pending post steps, and no surviving logs/artifacts after cad3eada.

  15. locked and limited conversation to collaborators on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions