You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: hosted runner loses communication during DynamoDB net8.0 tests #11331
The AWS DynamoDB provider tests (net8.0) job failed because its hosted runner lost communication with GitHub, before completing the test step or publishing its test artifacts.
The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
Available evidence
The job started at 2026-09-18T21:46:29Z and was marked failed at 22:41:30Z. Checkout and Start DynamoDB Local succeeded. The Test step started at 21:46:42Z, but its REST metadata still reports in_progress with no completion timestamp; cleanup and post-job steps remain pending.
gh run view --job 105772493932 --log-failed reports log not found. The run has DynamoDB net10.0 output/coverage artifacts, but no corresponding net8.0 artifact. No test assertion or specific test failure could be recovered from the available evidence.
Scope and investigation
The triggering changes concern durable Messaging/Journaling rather than the DynamoDB provider. This is currently classified as a hosted-runner failure with an unlocalized cause, not a confirmed DynamoDB product regression. The annotation alone does not distinguish process termination, CPU/memory starvation, or network loss.
Please investigate runner diagnostics/resource exhaustion and whether an active test or container operation prevented completion and artifact upload. Exact-message and DynamoDB runner/hang searches found no existing matching issue or recently merged fix. No workflow rerun or branch rebase was performed to replace the original failure evidence.
Job started2026-09-19T21:01:14Z; marked failed21:51:16Z. Test step started21:01:23Z and still reports in_progress with no completion; post steps remain pending.
The annotation is exactly “The hosted runner lost communication with the server...” gh run view --log-failed reports log not found; no matching Windows net8 SlowBVT result artifact exists. No specific test assertion or responsible process was recovered, so this is not being attributed to the ZooKeeper-only candidate or asserted to share a root cause with the earlier DynamoDB occurrence.
No linked merged remedy was found and no unchanged workflow rerun was requested. Preserving the available evidence here rather than opening duplicate unlocalized runner tracking.
Completed the end-to-end follow-up. The evidence supports a hang followed by runner disappearance, but it does not localize the initiating fault sufficiently to justify a repository or workflow change.
Cross-run evidence
I scanned 250 recent .NET CI runs, including 162 failed jobs. Only these two jobs have the distinctive state where the active test step remains in_progress, later steps remain pending, and GitHub marks the job failed because the runner lost communication:
Both jobs had already exceeded their normal execution time by roughly 35–40 minutes before the runner was declared lost. This is stronger evidence for a stalled test/coordinator or resource-starved host followed by agent loss than for an ordinary test assertion or immediate random disconnect.
Both affected partitions target net8.0, but the workloads and operating systems differ. That correlation is notable but not sufficient to attribute the failures to MTP, xUnit, CoreCLR, or the Orleans test harness.
The immediately prior net8 job passed on the same Actions runner version, ubuntu-24.04 image version, and DynamoDB image digest.
Later net8 jobs repeatedly passed, including job 105821864641 and job 106167351519. The latest downloaded artifact contains 604/604 passing tests; net10 also contains 604/604.
fix(aws): preserve DynamoDB membership consistency #11310 expanded this partition from 425 to 604 discovered tests shortly before the incident, so it increased workload. However, same-run net10 and repeated later net8 executions passed in normal time, so the expansion is not evidence of a deterministic regression.
The triggering commit 5e9cafafd49d0654041889e315d5955bc2331fb5 changes only a durable-messaging test and does not touch DynamoDB or CI.
Independent Windows occurrence
The same-run Windows net10 SlowBVT job and the Ubuntu net8/net10 SlowBVT jobs passed. The DynamoDB net8 job in that run also passed 645 tests.
The affected Windows net8 SlowBVT partition passed immediately before and after on the same runner/image versions.
Its triggering commit 1204d79edcbb87a49fa3fd67287344e18edd2500 changes only ZooKeeper implementation/unit-test files, which are outside the Provider=None SlowBVT partition.
Both job-log REST endpoints return 404 BlobNotFound; no partial terminal output is recoverable.
Neither affected partition produced an artifact because the runner disappeared before the always-run archive and cleanup steps could execute. This is not an upload-artifact failure or a Testcontainers lifecycle issue; the DynamoDB job uses a workflow-owned Docker container, and Windows SlowBVT does not share that container path.
The shared harness requests full crash/hang dumps with a 10-minute hang threshold. Dump collection or another workload could have amplified resource pressure, but no dump, OOM record, process telemetry, or kernel/runner diagnostic survived. Reducing dump types, adding a job timeout, or changing MTP/xUnit based only on these two losses would be speculative and could remove the diagnostics needed for other known failures.
Searches across open and merged PRs found no existing runner-loss fix to advance. Recent MTP lifecycle, dump capture, artifact retry, and retention changes do not cover the boundary where the VM/agent disappears before log/artifact upload.
GitHub Status recorded no Actions incident during either window, which rules out a declared platform-wide outage but not an isolated hosted VM failure.
Disposition: keep this as an evidence-preserving CI reliability issue. No repository/workflow PR is opened because the initiating cause remains unlocalized and every candidate change would be speculative. A recurrence should be correlated with this exact in_progress + pending-post-steps + missing-log-blob signature; actionable next evidence requires off-runner telemetry or a surviving runner diagnostic identifying OOM, dump collection, coordinator/test-host failure, or network loss.
The only subsequent CI workflow changes are ci: suppress dotnet coverage logo #11340, which suppresses the coverage banner, and ci: remove ZooKeeper failure diagnostics #11341, which removes obsolete ZooKeeper diagnostics. Neither affects net8 hang dumps, test coordinator lifetime, runner resource limits, DynamoDB container lifecycle, or off-runner log/artifact preservation.
I inspected the latest 50 .NET CI runs, including 26 failed jobs. There are zero additional jobs with the defining signature: an active step left in_progress, pending post-job steps, and runner communication loss.
Recent main and feature-branch CI runs continue to complete the DynamoDB and net8 partitions normally.
Current disposition remains unchanged: no responsible repository fix is evident, so opening a workflow or product PR would be speculative. The next actionable occurrence needs surviving host/runner evidence which distinguishes a test/MTP stall, dump-induced resource exhaustion, OOM, process termination, or isolated hosted-VM/network loss.
No relevant .github, test infrastructure, MTP/xUnit, or DynamoDB test change has landed on main since the prior update.
I inspected the latest 80 .NET CI runs, including 50 failed jobs. There are zero new jobs with the defining signature: the active test step remains in_progress, post-job steps remain pending, and GitHub reports hosted-runner communication loss.
Disposition remains unchanged: current behavior indicates a rare stalled job followed by hosted-agent loss, with no surviving log blob, artifact, OOM record, dump, or process telemetry identifying the initiator. The repository has no demonstrated regression or surgical fix to apply. A code/workflow PR would be speculative; the next actionable evidence requires a recurrence with surviving host or off-runner diagnostics.
I inspected the latest 100 .NET CI runs, including 77 failed jobs. The scan found one job with the defining stuck-step/runner-loss signature, and it is the already-recorded September 19 Windows net8 SlowBVT job (105966792186). There are no new occurrences.
Searches across open and merged PRs still find no fix for hosted-runner communication loss, the net8 coordinator stall, dump-induced resource exhaustion, or the missing-log/artifact boundary. The only newly opened PR is unrelated code-generation work (fix(codegen): escape reserved C# keywords in generated metadata namespace #11349).
No relevant main change has landed since the prior update in CI workflows, MTP/xUnit configuration, DynamoDB tests, or shared test execution.
Current disposition: the repository reliably runs these partitions in subsequent CI, while the two original runners disappeared after prolonged stalls and before durable diagnostics were uploaded. The available evidence does not establish a product, container-lifecycle, or workflow regression, so there is no deterministic repository fix to implement. Keep this issue as the correlation point for a future recurrence with surviving host or off-runner telemetry.
I inspected the latest 120 .NET CI runs, including 131 failed jobs. The only job matching the defining stuck-step/runner-loss signature remains the already-recorded September 19 Windows net8 SlowBVT job (105966792186); there are no new occurrences.
No open or merged PR since the prior update addresses hosted-runner communication loss, net8 coordinator hangs, resource exhaustion, or preservation of diagnostics after agent/VM disappearance.
No relevant change has landed on main in the CI workflow, shared test runner, MTP/xUnit configuration, or DynamoDB test area.
Disposition: the observed invariant remains a prolonged test step followed by loss of the hosted agent before logs and artifacts can be persisted. Without a surviving OOM record, dump, process trace, or host diagnostic, a repository cause is not established and a workflow/product PR would be speculative. Keep this issue as the correlation point for a future evidence-bearing recurrence.
The latest 140 .NET CI runs contain no jobs with the defining signature: an active step left in_progress, pending post-job steps, and hosted-runner communication loss.
Searches across open and merged PRs continue to find no runner-loss or resource-exhaustion fix. No relevant main change has landed in the shared test action, CI workflow, MTP/xUnit configuration, or AWS test partition since the previous update.
Disposition remains unchanged: subsequent CI demonstrates normal execution, while the original failures lost their hosted agents only after prolonged stalls and before durable logs/artifacts were persisted. The surviving evidence cannot attribute the initiating fault to Orleans, DynamoDB Local, MTP, dump collection, or the hosted VM. No deterministic repository change is supported, so no skip or speculative PR is warranted.
The latest 160 .NET CI runs, including 137 failed jobs, contain no job with the defining active-step-in_progress + pending-post-steps + hosted-runner-loss signature.
Git history shows no subsequent relevant main change in the CI workflow, shared test action, MTP/xUnit configuration, or AWS test partition. Surrounding and later DynamoDB/net8 executions pass normally, so a repository regression is not established.
The two original failures both exceeded normal runtime by roughly 35–40 minutes before their agents disappeared, leaving the active test step incomplete and preventing post-job log/artifact upload. The surviving evidence establishes a hang-then-agent-loss outcome but cannot identify whether the initiator was a test/MTP stall, dump-induced resource pressure, OOM/process termination, network loss, or an isolated hosted VM.
The investigation and self-review found no deterministic Orleans-side change which preserves real provider validation and addresses a demonstrated cause. A rubber-duck review challenged the initial “random outage” wording, identified the pre-loss overrun pattern, and prompted the broader historical signature scan; those findings are reflected in this conclusion. It did not establish a defensible code or workflow fix.
Final disposition: no existing PR fixes #11331, and no repository PR is justified from the available evidence. Keep the issue as the correlation point for a future recurrence with surviving host or off-runner telemetry; do not add a skip or speculative timeout/dump change.
The latest 180 .NET CI runs, including 148 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
No new open or merged PR addresses this signature or its test/MTP/resource boundary, and no relevant main change has landed in CI, shared test execution, MTP/xUnit configuration, or the AWS test partition since the final September 28 review.
Subsequent DynamoDB/net8 and unrelated net8 partitions continue to complete normally. No repeated signature or newly merged change establishes that a recent Orleans modification introduced or resolved the two original incidents.
Actionable diagnostic boundary: a future occurrence must preserve host-level evidence outside the disappearing runner—such as platform runner diagnostics, OOM/process termination telemetry, or another durable external signal—to distinguish a test/coordinator stall from dump-induced resource pressure, network loss, or isolated VM failure. In-run artifact steps cannot provide that evidence after the agent is gone.
No repository change, skip, broad timeout, or speculative mitigation is justified by the current evidence.
The latest 200 .NET CI runs, including 153 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
Merged test(membership): wire provider conformance suites #11332 adds membership conformance wiring and DynamoDB fixture ownership. Its own description records the Windows net8 runner loss as a separate event without a recovered assertion or root-cause attribution; it does not fix this issue.
The only newly opened reliability PR is fix(adonet): prevent SQLite transaction leaks #11354, which fixes a SQLite pooled-connection transaction leak. It is unrelated to DynamoDB, MTP/test coordination, runner resources, or hosted-agent loss.
No relevant main change has landed since the prior update in CI, shared test execution, MTP/xUnit configuration, or the AWS test partition.
The evidence continues to establish the outcome—prolonged test-step stall followed by hosted-agent disappearance—but not the initiating cause. Subsequent affected partitions pass, and no repeated signature or effective PR identifies a repository regression.
Next action remains evidence acquisition, not mitigation: correlate any future recurrence with durable host/off-runner OOM, process termination, runner, or network telemetry. No skip, broad timeout, dump reduction, or product change is justified without that evidence.
The latest 220 .NET CI runs, including 159 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
New PR ci: mirror pinned provider test images #11356 mirrors/pins CI images to address registry throttling during pre-test image acquisition. For DynamoDB Local it pins sha256:ff89bd48ff32cd8d9be5fee8873b65b8854dc408f1afe881be6eb00247bc0dab on the official registry—the same digest used by the failing job and surrounding passing jobs. The ci: hosted runner loses communication during DynamoDB net8.0 tests #11331 job had already pulled and started that image, passed TCP readiness, and entered the Test step. ci: mirror pinned provider test images #11356 therefore improves deterministic acquisition but does not fix or identify this post-start runner-loss boundary.
No other new PR or relevant main change affects MTP/test coordination, hang dumps, runner resources, DynamoDB lifecycle, or diagnostic persistence.
The available evidence does not support classifying the initiator as purely GitHub infrastructure or purely repository load. It establishes a prolonged test/coordinator stall followed by hosted-agent disappearance. Repository workload or dump collection could have amplified resource pressure, but repeated normal runs on the same image/workload and the independent Windows occurrence leave that unproven; isolated hosted VM/network failure also remains possible.
Disposition: no existing PR fixes #11331 and no Orleans-side change is justified. #11356 is relevant to image-pull reliability only. The remaining blocker is durable host/off-runner telemetry from a future recurrence.
The latest 250 .NET CI runs, including 337 failed jobs, contain no job with the defining active test step left in_progress, pending post-job steps, and hosted-runner communication loss.
Searches across open, closed, and merged PRs still find no fix for this post-readiness runner-loss boundary. No recent main change establishes an introducing regression or a merged remedy.
The evidence continues to distinguish this from image acquisition failures: the #11331 runner had completed DynamoDB pull/start/readiness and was executing tests when it stalled and later disappeared. Subsequent identical provider jobs complete normally.
Disposition: no repository fix or PR is supported. The blocker remains durable host/off-runner telemetry from a future recurrence; in-run cleanup and artifact upload cannot execute after the hosted agent is lost.
Setup, Ministack, DynamoDB Local, and readiness completed successfully.
Test started at 2026-10-03T15:24:52Z, remained in_progress, and the runner was marked lost at 16:13:37Z after 49 minutes.
Cleanup/post steps remained pending, the job log endpoint returns 404 BlobNotFound, and only the successful net8 Kinesis artifact exists.
Same-run Kinesis net8, DynamoDB net8/net10, and all SlowBVT variants passed. Later runs of the same branch passed both Kinesis frameworks.
Triggering commit c98d09767b7e6961c3764d43f3af04e23190ab32 changed only DefaultClientServices.cs, outside Kinesis test infrastructure.
This occurrence crosses the earlier framework boundary: Linux net10 joins Linux net8 and Windows net8. The common repository-controlled sequence is a stall past the 10-minute inactivity threshold followed by Full hang-dump collection and eventual agent loss. Microsoft documents Full as the largest dump, containing all memory including mapped images; Heap retains threads, stacks, handles, and managed memory while excluding mapped images.
PR #11405 changes both CI test launch paths to Heap hang dumps, retains Full crash dumps, and adds workflow contract coverage. It preserves test/provider execution and failure semantics while bounding the diagnostic operation which runs during a stall.
Closing as mitigated by #11405, merged in cad3eada5e845e1cd0e17df1ea92d9d7a78aa7f2 on October 5.
The first observed occurrence was the September 18 DynamoDB net8 job recorded above. The surrounding and later identical provider partitions passed, and the triggering change did not touch DynamoDB or CI, so the evidence did not establish a DynamoDB product regression. A September 19 Windows net8 SlowBVT occurrence and the October 3 Kinesis net10 occurrence crossed provider, OS, and framework boundaries. All three shared the repository-controlled sequence of a test stall beyond the 10-minute inactivity threshold followed by Full hang-dump collection and hosted-agent loss before logs, cleanup, or artifacts could persist.
#11405 changes both standard test launch paths to Heap hang dumps while retaining the 10-minute threshold, Full crash dumps, provider coverage, and failure semantics. Its merge run completed all affected partitions successfully, including DynamoDB net8/net10, Kinesis net8/net10, and SlowBVT across both operating systems/frameworks.
The first subsequent main failure in DynamoDB net8, run 37385298502/job 112016979267, is useful boundary evidence: the test produced a normal assertion-backed timeout, the Test step completed, cleanup and post steps ran, and the artifact/logs were uploaded. That distinct conformance flake is tracked in #11407. It did not reproduce the runner-loss signature.
Disposition: the initiating test stalls remain independently diagnosable, while the shared diagnostic resource-pressure amplifier responsible for this issue's lost-runner outcome is bounded. Reopen or correlate a new occurrence if a runner again disappears with the active step left in_progress, pending post steps, and no surviving logs/artifacts after cad3eada.
Activity
ReubenBond commentedon Sep 19, 2026
Another occurrence of the same hosted-runner communication-loss annotation, on a different workload/platform:
1204d79edcbb87a49fa3fd67287344e18edd2500GitHub Actions 1006803077The annotation is exactly “The hosted runner lost communication with the server...”
gh run view --log-failedreportslog not found; no matching Windows net8 SlowBVT result artifact exists. No specific test assertion or responsible process was recovered, so this is not being attributed to the ZooKeeper-only candidate or asserted to share a root cause with the earlier DynamoDB occurrence.No linked merged remedy was found and no unchanged workflow rerun was requested. Preserving the available evidence here rather than opening duplicate unlocalized runner tracking.
ReubenBond commentedon Sep 21, 2026
Completed the end-to-end follow-up. The evidence supports a hang followed by runner disappearance, but it does not localize the initiating fault sufficiently to justify a repository or workflow change.
Cross-run evidence
.NET CIruns, including 162 failed jobs. Only these two jobs have the distinctive state where the active test step remainsin_progress, later steps remain pending, and GitHub marks the job failed because the runner lost communication:DynamoDB comparison
ubuntu-24.04image version, and DynamoDB image digest.5e9cafafd49d0654041889e315d5955bc2331fb5changes only a durable-messaging test and does not touch DynamoDB or CI.Independent Windows occurrence
1204d79edcbb87a49fa3fd67287344e18edd2500changes only ZooKeeper implementation/unit-test files, which are outside theProvider=NoneSlowBVT partition.Diagnostic boundary and disposition
404 BlobNotFound; no partial terminal output is recoverable.upload-artifactfailure or a Testcontainers lifecycle issue; the DynamoDB job uses a workflow-owned Docker container, and Windows SlowBVT does not share that container path.Disposition: keep this as an evidence-preserving CI reliability issue. No repository/workflow PR is opened because the initiating cause remains unlocalized and every candidate change would be speculative. A recurrence should be correlated with this exact
in_progress+ pending-post-steps + missing-log-blob signature; actionable next evidence requires off-runner telemetry or a surviving runner diagnostic identifying OOM, dump collection, coordinator/test-host failure, or network loss.ReubenBond commentedon Sep 22, 2026
Refreshed the investigation after the September 21 merges.
.NET CIruns, including 26 failed jobs. There are zero additional jobs with the defining signature: an active step leftin_progress, pending post-job steps, and runner communication loss.Current disposition remains unchanged: no responsible repository fix is evident, so opening a workflow or product PR would be speculative. The next actionable occurrence needs surviving host/runner evidence which distinguishes a test/MTP stall, dump-induced resource exhaustion, OOM, process termination, or isolated hosted-VM/network loss.
ReubenBond commentedon Sep 24, 2026
September 24 refresh:
.github, test infrastructure, MTP/xUnit, or DynamoDB test change has landed onmainsince the prior update..NET CIruns, including 50 failed jobs. There are zero new jobs with the defining signature: the active test step remainsin_progress, post-job steps remain pending, and GitHub reports hosted-runner communication loss.Disposition remains unchanged: current behavior indicates a rare stalled job followed by hosted-agent loss, with no surviving log blob, artifact, OOM record, dump, or process telemetry identifying the initiator. The repository has no demonstrated regression or surgical fix to apply. A code/workflow PR would be speculative; the next actionable evidence requires a recurrence with surviving host or off-runner diagnostics.
ReubenBond commentedon Sep 25, 2026
September 25 refresh:
.NET CIruns, including 77 failed jobs. The scan found one job with the defining stuck-step/runner-loss signature, and it is the already-recorded September 19 Windows net8 SlowBVT job (105966792186). There are no new occurrences.Current disposition: the repository reliably runs these partitions in subsequent CI, while the two original runners disappeared after prolonged stalls and before durable diagnostics were uploaded. The available evidence does not establish a product, container-lifecycle, or workflow regression, so there is no deterministic repository fix to implement. Keep this issue as the correlation point for a future recurrence with surviving host or off-runner telemetry.
ReubenBond commentedon Sep 26, 2026
September 26 refresh:
.NET CIruns, including 131 failed jobs. The only job matching the defining stuck-step/runner-loss signature remains the already-recorded September 19 Windows net8 SlowBVT job (105966792186); there are no new occurrences.mainin the CI workflow, shared test runner, MTP/xUnit configuration, or DynamoDB test area.Disposition: the observed invariant remains a prolonged test step followed by loss of the hosted agent before logs and artifacts can be persisted. Without a surviving OOM record, dump, process trace, or host diagnostic, a repository cause is not established and a workflow/product PR would be speculative. Keep this issue as the correlation point for a future evidence-bearing recurrence.
ReubenBond commentedon Sep 27, 2026
September 27 refresh:
.NET CIruns contain no jobs with the defining signature: an active step leftin_progress, pending post-job steps, and hosted-runner communication loss.Disposition remains unchanged: subsequent CI demonstrates normal execution, while the original failures lost their hosted agents only after prolonged stalls and before durable logs/artifacts were persisted. The surviving evidence cannot attribute the initiating fault to Orleans, DynamoDB Local, MTP, dump collection, or the hosted VM. No deterministic repository change is supported, so no skip or speculative PR is warranted.
ReubenBond commentedon Sep 28, 2026
September 28 conclusion:
.NET CIruns, including 137 failed jobs, contain no job with the defining active-step-in_progress+ pending-post-steps + hosted-runner-loss signature.The investigation and self-review found no deterministic Orleans-side change which preserves real provider validation and addresses a demonstrated cause. A rubber-duck review challenged the initial “random outage” wording, identified the pre-loss overrun pattern, and prompted the broader historical signature scan; those findings are reflected in this conclusion. It did not establish a defensible code or workflow fix.
Final disposition: no existing PR fixes #11331, and no repository PR is justified from the available evidence. Keep the issue as the correlation point for a future recurrence with surviving host or off-runner telemetry; do not add a skip or speculative timeout/dump change.
ReubenBond commentedon Sep 29, 2026
September 29 refresh:
.NET CIruns, including 148 failed jobs, contain no job with the defining active test step leftin_progress, pending post-job steps, and hosted-runner communication loss.Actionable diagnostic boundary: a future occurrence must preserve host-level evidence outside the disappearing runner—such as platform runner diagnostics, OOM/process termination telemetry, or another durable external signal—to distinguish a test/coordinator stall from dump-induced resource pressure, network loss, or isolated VM failure. In-run artifact steps cannot provide that evidence after the agent is gone.
No repository change, skip, broad timeout, or speculative mitigation is justified by the current evidence.
ReubenBond commentedon Sep 30, 2026
September 30 refresh:
.NET CIruns, including 153 failed jobs, contain no job with the defining active test step leftin_progress, pending post-job steps, and hosted-runner communication loss.The evidence continues to establish the outcome—prolonged test-step stall followed by hosted-agent disappearance—but not the initiating cause. Subsequent affected partitions pass, and no repeated signature or effective PR identifies a repository regression.
Next action remains evidence acquisition, not mitigation: correlate any future recurrence with durable host/off-runner OOM, process termination, runner, or network telemetry. No skip, broad timeout, dump reduction, or product change is justified without that evidence.
ReubenBond commentedon Oct 1, 2026
October 1 refresh:
.NET CIruns, including 159 failed jobs, contain no job with the defining active test step leftin_progress, pending post-job steps, and hosted-runner communication loss.sha256:ff89bd48ff32cd8d9be5fee8873b65b8854dc408f1afe881be6eb00247bc0dabon the official registry—the same digest used by the failing job and surrounding passing jobs. The ci: hosted runner loses communication during DynamoDB net8.0 tests #11331 job had already pulled and started that image, passed TCP readiness, and entered the Test step. ci: mirror pinned provider test images #11356 therefore improves deterministic acquisition but does not fix or identify this post-start runner-loss boundary.The available evidence does not support classifying the initiator as purely GitHub infrastructure or purely repository load. It establishes a prolonged test/coordinator stall followed by hosted-agent disappearance. Repository workload or dump collection could have amplified resource pressure, but repeated normal runs on the same image/workload and the independent Windows occurrence leave that unproven; isolated hosted VM/network failure also remains possible.
Disposition: no existing PR fixes #11331 and no Orleans-side change is justified. #11356 is relevant to image-pull reliability only. The remaining blocker is durable host/off-runner telemetry from a future recurrence.
ReubenBond commentedon Oct 3, 2026
October 3 refresh:
.NET CIruns, including 337 failed jobs, contain no job with the defining active test step leftin_progress, pending post-job steps, and hosted-runner communication loss.The evidence continues to distinguish this from image acquisition failures: the #11331 runner had completed DynamoDB pull/start/readiness and was executing tests when it stalled and later disappeared. Subsequent identical provider jobs complete normally.
Disposition: no repository fix or PR is supported. The blocker remains durable host/off-runner telemetry from a future recurrence; in-run cleanup and artifact upload cannot execute after the hosted agent is lost.
ReubenBond commentedon Oct 4, 2026
A third occurrence now provides a shared, repository-controlled mitigation target:
GitHub Actions 10069802112026-10-03T15:24:52Z, remainedin_progress, and the runner was marked lost at16:13:37Zafter 49 minutes.404 BlobNotFound, and only the successful net8 Kinesis artifact exists.c98d09767b7e6961c3764d43f3af04e23190ab32changed onlyDefaultClientServices.cs, outside Kinesis test infrastructure.This occurrence crosses the earlier framework boundary: Linux net10 joins Linux net8 and Windows net8. The common repository-controlled sequence is a stall past the 10-minute inactivity threshold followed by
Fullhang-dump collection and eventual agent loss. Microsoft documents Full as the largest dump, containing all memory including mapped images; Heap retains threads, stacks, handles, and managed memory while excluding mapped images.PR #11405 changes both CI test launch paths to Heap hang dumps, retains Full crash dumps, and adds workflow contract coverage. It preserves test/provider execution and failure semantics while bounding the diagnostic operation which runs during a stall.
ReubenBond commentedon Oct 6, 2026
Closing as mitigated by #11405, merged in
cad3eada5e845e1cd0e17df1ea92d9d7a78aa7f2on October 5.The first observed occurrence was the September 18 DynamoDB net8 job recorded above. The surrounding and later identical provider partitions passed, and the triggering change did not touch DynamoDB or CI, so the evidence did not establish a DynamoDB product regression. A September 19 Windows net8 SlowBVT occurrence and the October 3 Kinesis net10 occurrence crossed provider, OS, and framework boundaries. All three shared the repository-controlled sequence of a test stall beyond the 10-minute inactivity threshold followed by
Fullhang-dump collection and hosted-agent loss before logs, cleanup, or artifacts could persist.#11405 changes both standard test launch paths to
Heaphang dumps while retaining the 10-minute threshold, Full crash dumps, provider coverage, and failure semantics. Its merge run completed all affected partitions successfully, including DynamoDB net8/net10, Kinesis net8/net10, and SlowBVT across both operating systems/frameworks.The first subsequent main failure in DynamoDB net8, run 37385298502/job 112016979267, is useful boundary evidence: the test produced a normal assertion-backed timeout, the Test step completed, cleanup and post steps ran, and the artifact/logs were uploaded. That distinct conformance flake is tracked in #11407. It did not reproduce the runner-loss signature.
Disposition: the initiating test stalls remain independently diagnosable, while the shared diagnostic resource-pressure amplifier responsible for this issue's lost-runner outcome is bounded. Reopen or correlate a new occurrence if a runner again disappears with the active step left
in_progress, pending post steps, and no surviving logs/artifacts aftercad3eada.