[Linux][arm] System.Memory.Tests failed with SIGSEGV on rolling build

[krwq]

Rolling build: https://dev.azure.com/dnceng/public/_build/results?buildId=1437467&view=results

  Discovering: System.Memory.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Memory.Tests (found 2366 of 2386 test cases)
  Starting:    System.Memory.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:   100 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Memory.Tests.runtimeconfig.json --depsfile System.Memory.Tests.deps.json xunit.console.dll System.Memory.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Sat Oct 23 08:57:31 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

in case it's needed you can get full log

net7.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673

Tagging subscribers to this area: @GrabYourPitchforks, @dotnet/area-system-memory
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling build: https://dev.azure.com/dnceng/public/_build/results?buildId=1437467&view=results

  Discovering: System.Memory.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Memory.Tests (found 2366 of 2386 test cases)
  Starting:    System.Memory.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:   100 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Memory.Tests.runtimeconfig.json --depsfile System.Memory.Tests.deps.json xunit.console.dll System.Memory.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Sat Oct 23 08:57:31 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

in case it's needed you can get full log

Author:	krwq
Assignees:	-
Labels:	arch-arm32, area-System.Memory, os-linux
Milestone:	-

[danmoseley]

can we get a dump?

[carlossanlop]

❯ runfo get-helix-payload -o "D:\repos\runfo_results" -j "a89b1515-26cd-4906-b64b-bd64b5f31c4b" -w "System.Memory.Tests"
Payload 5c08e442-3e29-432f-92c9-d2c425e436b5.zip => D:\repos\runfo_results\correlation-payload\5c08e442-3e29-432f-92c9-d2c425e436b5.zip
Payload 396d48f2-9cf8-4ec2-a943-dd73c3bb5676.zip => D:\repos\runfo_results\correlation-payload\396d48f2-9cf8-4ec2-a943-dd73c3bb5676.zip
Payload test-runtime-net7.0-Linux-Release-arm64.zip => D:\repos\runfo_results\correlation-payload\test-runtime-net7.0-Linux-Release-arm64.zip
Payload cd9644f0-5899-4135-a7cf-269eca232b22.zip => D:\repos\runfo_results\correlation-payload\cd9644f0-5899-4135-a7cf-269eca232b22.zip

------ Downloading files for: System.Memory.Tests -------

WorkItem System.Memory.Tests => D:\repos\runfo_results\workitems\System.Memory.Tests\System.Memory.Tests.zip
how-to-debug-dump.md => D:\repos\runfo_results\workitems\System.Memory.Tests\how-to-debug-dump.md
core.1001.100 => D:\repos\runfo_results\workitems\System.Memory.Tests\core.1001.100
console.3ef66c74.log => D:\repos\runfo_results\workitems\System.Memory.Tests\console.3ef66c74.log

There's a dump file in this folder:

❯ ls .\runfo_results\workitems\System.Memory.Tests\

        Directory: D:\repos\runfo_results\workitems\System.Memory.Tests


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a---        10/26/2021  12:08 PM          14922    console.3ef66c74.log
-a---        10/26/2021  12:08 PM      575078400    core.1001.100
-a---        10/26/2021  12:03 PM           6301    how-to-debug-dump.md
-a---        10/26/2021  12:03 PM        1298410    System.Memory.Tests.zip

But since it's a Linux arm64 dump file, you need to follow the instructions in the how-to-debug-dump.md file:

Download the Cross DAC Binaries, open it and choose the flavor that matches the dump you are to debug, and copy those files to C:\helix_payload\System.Memory.Tests\shared\Microsoft.NETCore.App\7.0.0.

The C:\helix_payload folder seems to refer to the contents of the zip file located in D:\repos\runfo_results\correlation-payload\test-runtime-net7.0-Linux-Release-arm64.zip. So I extracted it, and then copied the contents of the downloaded DAC Binaries zip file into the right location.

I opened the dump file with WinDBG, I made sure to specify the right symbols folder, and this is what I got:

Error message:

Unable to read memory at <Unavailable>

Callstack:

0:000> k
 # Child-SP          RetAddr               Call Site
00 (Inline Function) --------`--------     libcoreclr!WKS::gc_heap::mark_array_marked+0x14 [/__w/1/s\src/coreclr/gc/gc.cpp @ 8317] 
01 (Inline Function) --------`--------     libcoreclr!WKS::gc_heap::background_mark1+0x14 [/__w/1/s\src/coreclr/gc/gc.cpp @ 22429] 
02 (Inline Function) --------`--------     libcoreclr!WKS::gc_heap::background_mark_simple+0x14 [/__w/1/s\src/coreclr/gc/gc.cpp @ 23647] 
03 0000007e`ef7fad80 0000007f`97aed178     libcoreclr!WKS::gc_heap::background_promote+0xc4 [/__w/1/s\src/coreclr/gc/gc.cpp @ 23735] 
04 0000007e`ef7fada0 0000007f`979131cc     libcoreclr!GcInfoDecoder::EnumerateLiveSlots+0xe44 [/__w/1/s\src/coreclr/vm/gcinfodecoder.cpp @ 938] 
05 0000007e`ef7fb190 0000007f`97a30004     libcoreclr!EECodeManager::EnumGcRefs+0x114 [/__w/1/s\src/coreclr/vm/eetwain.cpp @ 5287] 
06 0000007e`ef7fb340 0000007f`9799da7c     libcoreclr!GcStackCrawlCallBack+0x258 [/__w/1/s\src/coreclr/vm/gcenv.ee.common.cpp @ 310] 
07 0000007e`ef7fb4a0 0000007f`9799dca8     libcoreclr!Thread::MakeStackwalkerCallback+0x98 [/__w/1/s\src/coreclr/vm/stackwalk.cpp @ 833] 
08 0000007e`ef7fb4f0 0000007f`9799e024     libcoreclr!Thread::StackWalkFramesEx+0x184 [/__w/1/s\src/coreclr/vm/stackwalk.cpp @ 914] 
09 0000007e`ef7fb8c0 0000007f`97a2c780     libcoreclr!Thread::StackWalkFrames+0xb8 [/__w/1/s\src/coreclr/vm/stackwalk.cpp @ 996] 
0a 0000007e`ef7fc5d0 0000007f`97a2c50c     libcoreclr!ScanStackRoots+0x1b0 [/__w/1/s\src/coreclr/vm/gcenv.ee.cpp @ 173] 
0b 0000007e`ef7fc680 0000007f`97b61740     libcoreclr!GCToEEInterface::GcScanRoots+0x100 [/__w/1/s\src/coreclr/vm/gcenv.ee.cpp @ 269] 
0c 0000007e`ef7fc6e0 0000007f`97b6030c     libcoreclr!WKS::gc_heap::background_mark_phase+0x890 [/__w/1/s\src/coreclr/gc/gc.cpp @ 33248] 
0d 0000007e`ef7fc7d0 0000007f`97b7824c     libcoreclr!WKS::gc_heap::gc1+0x394 [/__w/1/s\src/coreclr/gc/gc.cpp @ 20321] 
0e 0000007e`ef7fc880 0000007f`97a2f1c0     libcoreclr!WKS::gc_heap::bgc_thread_function+0xb8 [/__w/1/s\src/coreclr/gc/gc.cpp @ 34268] 
0f (Inline Function) --------`--------     libcoreclr!<unnamed-class>::operator()+0x60 [/__w/1/s\src/coreclr/vm/gcenv.ee.cpp @ 1409] 
10 0000007e`ef7fc8e0 0000007f`97d09ac0     libcoreclr!<unnamed-class>::__invoke+0x78 [/__w/1/s\src/coreclr/vm/gcenv.ee.cpp @ 1388] 
11 0000007e`ef7fc920 0000007f`982aefc4     libcoreclr!CorUnix::CPalThread::ThreadEntry+0x1f4 [/__w/1/s\src/coreclr/pal/src/thread/thread.cpp @ 1832] 
12 0000007e`ef7fc9d0 00000000`00000000     libpthread_2_23!_pthread_get_minstack+0x1404

coreclr/gc/gc.cpp:

inline unsigned int gc_heap::mark_array_marked(uint8_t* add)
{
    return mark_array [mark_word_of (add)] & (1 << mark_bit_bit_of (add));
}

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Rolling build: https://dev.azure.com/dnceng/public/_build/results?buildId=1437467&view=results

  Discovering: System.Memory.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Memory.Tests (found 2366 of 2386 test cases)
  Starting:    System.Memory.Tests (parallel test collections = on, max threads = 4)
./RunTests.sh: line 162:   100 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Memory.Tests.runtimeconfig.json --depsfile System.Memory.Tests.deps.json xunit.console.dll System.Memory.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Sat Oct 23 08:57:31 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.
ulimit -c value: unlimited

in case it's needed you can get full log

Author:	krwq
Assignees:	-
Labels:	arch-arm32, os-linux, area-GC-coreclr, untriaged
Milestone:	-

[danmoseley]

Thanks @carlossanlop ! GC folks, do you believe this is GC, or more likely some memory corruption caused by a libraries bug?

[mangod9]

Is this a consistent repro? This looks to be some kind of heap/stack corruption.

[carlossanlop]

@krwq how often is it happening?