Crash in GC

[mfilippov]

Description

.NET Runtime crash in GC with stack:

[0x0]   coreclr!CGCDesc::GetNumPointers + 0x24   
[0x1]   coreclr!WKS::gc_heap::background_mark_simple1 + 0x10262a   
[0x2]   coreclr!WKS::gc_heap::background_mark_simple + 0x74   
[0x3]   coreclr!WKS::gc_heap::background_promote + 0x52   
[0x4]   coreclr!ScanConsecutiveHandlesWithoutUserData + 0x45   
[0x5]   coreclr!BlockScanBlocksWithoutUserData + 0x44   
[0x6]   coreclr!ProcessScanQNode + 0x44   
[0x7]   coreclr!ProcessScanQueue + 0x3f   
[0x8]   coreclr!xxxTableScanQueuedBlocksAsync + 0x5d   
[0x9]   coreclr!xxxAsyncSegmentIterator + 0x55   
[0xa]   coreclr!TableScanHandles + 0x7c   
[0xb]   coreclr!xxxTableScanHandlesAsync + 0x97   
[0xc]   coreclr!HndScanHandlesForGC + 0x10c   
[0xd]   coreclr!Ref_TracePinningRoots + 0x96   
[0xe]   coreclr!GCScan::GcScanHandles + 0xd4   
[0xf]   coreclr!WKS::gc_heap::background_mark_phase + 0x1244   
[0x10]   coreclr!WKS::gc_heap::gc1 + 0x1b5   
[0x11]   coreclr!WKS::gc_heap::bgc_thread_function + 0x90   
[0x12]   coreclr!<lambda_c5a49c859c7ba70587da980d74c056e1>::operator() + 0xa0   
[0x13]   kernel32!BaseThreadInitThunk + 0x14   
[0x14]   ntdll!RtlUserThreadStart + 0x21   

I tried to switch into Server GC but it crashed too:

[0x0]   coreclr!CGCDesc::GetNumPointers + 0x25   
[0x1]   coreclr!SVR::gc_heap::background_mark_simple1 + 0xf4   
[0x2]   coreclr!SVR::gc_heap::background_mark_simple + 0x96   
[0x3]   coreclr!SVR::gc_heap::background_promote + 0x11d   
[0x4]   coreclr!ScanConsecutiveHandlesWithoutUserData + 0x45   
[0x5]   coreclr!BlockScanBlocksWithoutUserData + 0x44   
[0x6]   coreclr!ProcessScanQNode + 0x44   
[0x7]   coreclr!ProcessScanQueue + 0x3f   
[0x8]   coreclr!xxxTableScanQueuedBlocksAsync + 0x5d   
[0x9]   coreclr!xxxAsyncSegmentIterator + 0x55   
[0xa]   coreclr!TableScanHandles + 0x7c   
[0xb]   coreclr!xxxTableScanHandlesAsync + 0x97   
[0xc]   coreclr!HndScanHandlesForGC + 0x10c   
[0xd]   coreclr!Ref_TraceNormalRoots + 0xb8   
[0xe]   coreclr!GCScan::GcScanHandles + 0x218   
[0xf]   coreclr!SVR::gc_heap::background_mark_phase + 0x13ba   
[0x10]   coreclr!SVR::gc_heap::gc1 + 0x136   
[0x11]   coreclr!SVR::gc_heap::bgc_thread_function + 0xe0   
[0x12]   coreclr!<lambda_c5a49c859c7ba70587da980d74c056e1>::operator() + 0xa0   
[0x13]   kernel32!BaseThreadInitThunk + 0x14   
[0x14]   ntdll!RtlUserThreadStart + 0x21   

It happened in our internal build system. When I run build with /m:16 and worked if I run build without /m.

Configuration

Windows: 19042.867
.NET Core SDK: dotnet-sdk-5.0.103-win-x64
x64

Other information

I create two full dumps in Server and Workstation GC mode.
https://drive.google.com/file/d/1RvDtBOVmexXFYRdJuaEIpZUXmWLsRTg7/view?usp=sharing
https://drive.google.com/file/d/1yzqwsb2WXwmYaxBQE7kp702L1YR9qodb/view?usp=sharing

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

.NET Runtime crash in GC with stack:

[0x0]   coreclr!CGCDesc::GetNumPointers + 0x24   
[0x1]   coreclr!WKS::gc_heap::background_mark_simple1 + 0x10262a   
[0x2]   coreclr!WKS::gc_heap::background_mark_simple + 0x74   
[0x3]   coreclr!WKS::gc_heap::background_promote + 0x52   
[0x4]   coreclr!ScanConsecutiveHandlesWithoutUserData + 0x45   
[0x5]   coreclr!BlockScanBlocksWithoutUserData + 0x44   
[0x6]   coreclr!ProcessScanQNode + 0x44   
[0x7]   coreclr!ProcessScanQueue + 0x3f   
[0x8]   coreclr!xxxTableScanQueuedBlocksAsync + 0x5d   
[0x9]   coreclr!xxxAsyncSegmentIterator + 0x55   
[0xa]   coreclr!TableScanHandles + 0x7c   
[0xb]   coreclr!xxxTableScanHandlesAsync + 0x97   
[0xc]   coreclr!HndScanHandlesForGC + 0x10c   
[0xd]   coreclr!Ref_TracePinningRoots + 0x96   
[0xe]   coreclr!GCScan::GcScanHandles + 0xd4   
[0xf]   coreclr!WKS::gc_heap::background_mark_phase + 0x1244   
[0x10]   coreclr!WKS::gc_heap::gc1 + 0x1b5   
[0x11]   coreclr!WKS::gc_heap::bgc_thread_function + 0x90   
[0x12]   coreclr!<lambda_c5a49c859c7ba70587da980d74c056e1>::operator() + 0xa0   
[0x13]   kernel32!BaseThreadInitThunk + 0x14   
[0x14]   ntdll!RtlUserThreadStart + 0x21   

I tried to switch into Server GC but it crashed too:

[0x0]   coreclr!CGCDesc::GetNumPointers + 0x25   
[0x1]   coreclr!SVR::gc_heap::background_mark_simple1 + 0xf4   
[0x2]   coreclr!SVR::gc_heap::background_mark_simple + 0x96   
[0x3]   coreclr!SVR::gc_heap::background_promote + 0x11d   
[0x4]   coreclr!ScanConsecutiveHandlesWithoutUserData + 0x45   
[0x5]   coreclr!BlockScanBlocksWithoutUserData + 0x44   
[0x6]   coreclr!ProcessScanQNode + 0x44   
[0x7]   coreclr!ProcessScanQueue + 0x3f   
[0x8]   coreclr!xxxTableScanQueuedBlocksAsync + 0x5d   
[0x9]   coreclr!xxxAsyncSegmentIterator + 0x55   
[0xa]   coreclr!TableScanHandles + 0x7c   
[0xb]   coreclr!xxxTableScanHandlesAsync + 0x97   
[0xc]   coreclr!HndScanHandlesForGC + 0x10c   
[0xd]   coreclr!Ref_TraceNormalRoots + 0xb8   
[0xe]   coreclr!GCScan::GcScanHandles + 0x218   
[0xf]   coreclr!SVR::gc_heap::background_mark_phase + 0x13ba   
[0x10]   coreclr!SVR::gc_heap::gc1 + 0x136   
[0x11]   coreclr!SVR::gc_heap::bgc_thread_function + 0xe0   
[0x12]   coreclr!<lambda_c5a49c859c7ba70587da980d74c056e1>::operator() + 0xa0   
[0x13]   kernel32!BaseThreadInitThunk + 0x14   
[0x14]   ntdll!RtlUserThreadStart + 0x21   

It happened in our internal build system. When I run build with /m:16 and worked if I run build without /m.

Configuration

Windows: 19042.867
.NET Core SDK: dotnet-sdk-5.0.103-win-x64
x64

Other information

I create two full dumps in Server and Workstation GC mode.
https://drive.google.com/file/d/1RvDtBOVmexXFYRdJuaEIpZUXmWLsRTg7/view?usp=sharing
https://drive.google.com/file/d/1yzqwsb2WXwmYaxBQE7kp702L1YR9qodb/view?usp=sharing

Author:	mfilippov
Assignees:	-
Labels:	area-GC-coreclr, untriaged
Milestone:	-

[jkotas]

Could you please reproduce the crash with these environment variables #45557 (comment) set and share the dump? (Reproducing it with one GC flavor is sufficient.)

[jkotas]

Also, update to 5.0.4 if possible.

[mfilippov]

New crash dump with .NET 5.0.4 and

COMPlus_LogFacility=0x00080001
COMPlus_LogLevel=6
COMPlus_StressLog=1
COMPlus_StressLogSize=2000000
COMPlus_TotalStressLogSize=40000000

https://drive.google.com/file/d/1dS-S2VlrBbe6zBdfmtnMnQQYO1exptRj/view?usp=sharing

[jkotas]

Thank you for the new dump. This is a bug in CGCDesc::GetNumPointers. It does not handle collectible types correctly.

[jkotas]

In addition to that, there is also a problem with missing empty GCDesc for non-canonical generic instantiations.

The type causing the problem is System.WeakReference[[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis]] with Microsoft.CodeAnalysis being loaded into a collectible load context. It should have an empty GCDesc, but it does not.

[mangod9]

Thanks @jkotas for diagnosing. Does this look like a regression in 5?

[jkotas]

It looks like a bug that has been there since forever. The bug just results into occasional small GC inefficiency most of the time. The crash only happens when all of the following is true:

• The workload uses collectible types
• The GC is close to hitting mark stack overflow (ie the object graph is very deep and complex)
• The overflowing objects have collectible types
• The MethodTable of the overflowing objects is allocated at the start of the page and the preceding page is unmapped. (The crash is caused by accesing this unmapped page.)