Poll based datapaths can hang sends in race condition with kernel

[thhous-msft]

Describe the bug

For sends in the datapath, we don't enable the edge triggers until after a socket write returns EAGAIN or EWOULDBLOCK. However, if sends in the kernel become enabled between calling send() and epoll_ctl (Or equivalent in other paths), the epoll event would never tirgger, and the sends would be hung forever for that socket.

Affected OS

• All
• Windows Server 2022
• Windows 11
• Windows Insider Preview (specify affected build below)
• Ubuntu
• Debian
• Other (specify below)

Additional OS information

kqueue and epoll datapaths, in addition to PSN on windows

MsQuic version

main

Steps taken to reproduce bug

Race condition

Expected behavior

Sends can't hang

Actual outcome

Sends can hang

Additional details

The solution to this will be to always keep the edge triggers enabled. Then before we send, enqueue the send into the pending queue. If the send passes, dequeue it, otherwise just continue, and the datapath thread will pick up the send later.