[QUIC] Abort on cancellation throws QUIC_STATUS_INVALID_PARAMETER

[CarnaViire]

Happens also in CI - see last 30 days - status on 8/25:

Day	Run	Details
8/24	PR 1965733 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open
8/24	PR 1965170 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/24	Rolling run 1964727 (7.0)	net7.0-Linux-Release-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/23	PR 1962069 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open
8/23	PR 1962049 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open
8/22	PR 1959853 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/22	PR 1958051 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/18	PR 1953122 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)Ubuntu.1804.Amd64.Open
8/18	PR 1951538 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/18	PR 1951086 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/17	PR 1950071 (7.0-rc1)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/16	PR 1946224 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/15	PR 1943354 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/14	PR 1942884 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/12	PR 1941440 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/12	PR 1939958 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/9	PR 1931560 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/8	PR 1930321 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/6	PR 1927463 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
8/4	PR 1923922 (main)	net7.0-Linux-Debug-x64-CoreCLR_release-(Alpine.314.Amd64.Open)Ubuntu.1804.Amd64.Open
7/25-8/3	Logs not available

Noticed in HTTP/3 stress tests. Occurrences in period 8/3-8/24:

21x Hits in tests (all crashes), recently 1-2 times per day in PRs

Details

Date	Branch	Log
23.08	PR #73479	log
23.08	PR #74433	log
22.08	PR #74376	log
22.08	PR #74322	log
19.08	PR #74215	log
18.08	PR #74147	log
18.08	PR #74002	log
17.08	PR #74098	log
16.08	PR #73981	log
16.08	PR #73768	log
15.08	PR #67049	log
15.08	PR #73547	log
14.08	PR #73907	log
13.08	PR #73817	log
13.08	PR #73697	log
12.08	PR #73745	log
12.08	PR #73748	log
09.08	PR #72934	log
08.08	PR #73586	log
06.08	PR #73515	log
04.08	PR #73305	log

2x Hits in stress tests

Details

1x in Run #20220803.6 on PR #72746
1x in Run #20220810.1 on main

System.AggregateException: One or more errors occurred. (One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER))
client_1  |  ---> System.AggregateException: One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER)
client_1  |  ---> System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
client_1  |    at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 422
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 108
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in /app/ClientOperations.cs:line 313
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 224

{
    "ErrorMessage":  "An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER"
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Occurrences:

1x in Run #20220803.6 on PR #72746
1x in Run #20220810.1 on main

System.AggregateException: One or more errors occurred. (One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER))
client_1  |  ---> System.AggregateException: One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER)
client_1  |  ---> System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
client_1  |    at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 422
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    --- End of inner exception stack trace ---
client_1  |    at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
client_1  |    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in /app/ClientOperations.cs:line 108
client_1  |    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in /app/ClientOperations.cs:line 313
client_1  | --- End of stack trace from previous location ---
client_1  |    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 224

Author:	CarnaViire
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[karelz]

Triage: We should look into that.
@CarnaViire is running stress locally to get a repro, but couldn't reproduce it just yet. We might punt to 8.0 it if we won't be able to find anything useful.

Putting it into Low Priority bucket as we cannot reproduce it locally and it has only rare hit (1x per week so far).

[karelz]

Moving it to 8.0 as we were unable to reproduce it and make it actionable. Impact seems to be low as well.

[rzikm]

Reproduced in #73817 at https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-73817-merge-3f1c7c257c234ddebf/System.Net.Quic.Functional.Tests/1/console.8d88137e.log?helixlogtype=result

Unhandled exception. System.AggregateException: One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER)
 ---> System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
   at System.Net.Quic.ThrowHelper.ThrowIfMsQuicError(Int32 status, String message) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ThrowHelper.cs:line 123
   at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 422
   at System.Net.Quic.QuicStream.<>c.<.ctor>b__23_0(Object target) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 75
   at System.Net.Quic.ResettableValueTaskSource.<>c.<TryGetValueTask>b__12_0(Object obj, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 85
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 874
   at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1104
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
   at System.Threading.CancellationTokenSource.TimerCallback(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 35
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 703
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 666
   at System.Threading.TimerQueue.FireNextTimers() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 330
   at System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/TimerQueue.Portable.cs:line 139
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 984
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105

My suspicion is that we may have some race which could result in calling StreamShutdown twice.

[carlossanlop]

Hit it in this RC1 backport PR: #74098

Log

Unhandled exception. System.AggregateException: One or more errors occurred. (An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER)
 ---> System.Net.Quic.QuicException: An internal error has occurred. StreamShutdown failed: QUIC_STATUS_INVALID_PARAMETER
   at System.Net.Quic.ThrowHelper.ThrowIfMsQuicError(Int32 status, String message) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ThrowHelper.cs:line 123
   at System.Net.Quic.QuicStream.Abort(QuicAbortDirection abortDirection, Int64 errorCode) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 422
   at System.Net.Quic.QuicStream.<>c.<.ctor>b__23_0(Object target) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 75
   at System.Net.Quic.ResettableValueTaskSource.<>c.<TryGetValueTask>b__12_0(Object obj, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 85
   at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 874
   at System.Threading.CancellationTokenSource.CallbackNode.ExecuteCallback() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 1104
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
   --- End of inner exception stack trace ---
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 726
   at System.Threading.CancellationTokenSource.TimerCallback(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/CancellationTokenSource.cs:line 35
   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 703
   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 666
   at System.Threading.TimerQueue.FireNextTimers() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Timer.cs:line 330
   at System.Threading.TimerQueue.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/TimerQueue.Portable.cs:line 139
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 984
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105

[CarnaViire]

I suspect that the reason is not a race but a native heap corruption, note the test suite also crashed with SIGABRT as in #72696