System.Text.Json.Tests crashing with segfault in CI

[ViktorHofer]

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1287656&view=ms.vss-test-web.build-test-results-tab&runId=38015200&resultId=183344&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Configuration: net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2741 of 2798 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:    93 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Aug 11 10:28:18 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/console.0d8519e9.log?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Dump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/core.1001.93?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build: https://dev.azure.com/dnceng/public/_build/results?buildId=1287656&view=ms.vss-test-web.build-test-results-tab&runId=38015200&resultId=183344&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

Configuration: net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.Json.Tests (found 2741 of 2798 test cases)
  Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:    93 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Wed Aug 11 10:28:18 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/console.0d8519e9.log?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Dump: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57193-merge-32896558dfa04faba1/System.Text.Json.Tests/1/core.1001.93?sv=2019-07-07&se=2021-08-31T10%3A24%3A36Z&sr=c&sp=rl&sig=sW6tkW0yTybYwRggqu%2FJypbei5TuvxQeYGIra%2BV%2BBa8%3D

Author:	ViktorHofer
Assignees:	-
Labels:	area-System.Text.Json, untriaged
Milestone:	-

[eiriktsarpalis]

Seems like a one-off occurrence so far. We can evaluate if it reappears.

[ViktorHofer]

I think I saw this multiple times. Runfo/Kusto should provide you with the data of how often this has yet happened in CI.

[eiriktsarpalis]

There have been 11 or so System.Text.Json segfaults in the past month for that particular queue, but none occurring in master.

let jobs = Jobs
| where Queued > ago (30d)
// | where Source == "ci/public/dotnet/runtime/refs/heads/main"
| project Name;
WorkItems
| where Started > ago (30d)
| where JobName  in (jobs)
| where QueueName == "ubuntu.1804.armarch.open"
| where ExitCode == 139
| summarize count() by FriendlyName, QueueName

But it seems like 5 of them occurred in the past week:

let crashes =
WorkItems
| where Started > ago (30d)
| where QueueName == "ubuntu.1804.armarch.open"
| where ExitCode == 139;
let min_t = toscalar(crashes | summarize min(Started));
let max_t = toscalar(crashes | summarize max(Started));
crashes
| make-series num=count() on Started in range(min_t, max_t, 1d) by FriendlyName
| limit 5
| render timechart

I'll investigate the dumps.

[eiriktsarpalis]

@ViktorHofer my arm64 WSL2 installation has suddenly stopped working 🤷, so I'm trying to analyze the dump on Windows using the cross DAC binaries. However following the instructions detailed here results in the following error message when I try to dump the stack or any related command:

> clrstack
Failed to load data access module, 0x80004002
Can not load or initialize mscordaccore.dll. The target runtime may not be initialized.

Perhaps I'm copying the DAC artifacts incorrectly, I'm assuming we should be copying the contents of Linux.arm64.Checked/x64 directly into the shared/Microsoft.NETCore.App/6.0.0 folder of the helix payload?

[eiriktsarpalis]

Marking as blocking-release as it's unlikely this will get addressed by the snap deadline. It's relatively low priority but we should still investigate this for 6.0.0.