Switch epoll and kqueue to level triggers

[thhous-msft]

Description

kqueue and epoll datapaths have a race condition upon pended sends whereas they might not get trigger notifications. By switching to level triggers, we solve this issue.

Closes #2654

Testing

Existing tests cover the change.

Documentation

N/A

[thhous-msft]

Don't merge this yet, I want to do more local testing. Its just easy for CI to test everything as well.

[nibanks]

I still have my reservations about making changes here, because we've never seen an issue with the current code. Is there any test we can write (maybe at the datapath unit test layer) to prove/disprove this change?

[thhous-msft]

I did a lot more research, and some independent testing, and at least on epoll, as long as pending doesn't happen with a basically empty kernel buffer, the event will still be triggered. Its a test that was only reproducable using TCP (It was the only way I could actually force EAGAIN or EWOULDBLOCK to occur), so adding it to the tests isn't really feasable.

However, if we ever want to implement PSN, It does require this change. PSN is well defined in its behavior here, and different from epoll and kqueue. Also the code is written as if level shifting was the mode we were using.

[nibanks]

How do we validate this with testing though?

[thhous-msft]

Well we validate that it works, and doesn't break performance. If it was going to be problematic, it would have triggered an issue with those cases. And using a TCP test, validated that the logic holds.

The issue is its basically impossible to trigger EAGAIN on write for UDP. I've tried looking if theres a way to force it, but I can't find any.

[nibanks]

Ok. I leave you in charge of the final decision!

[thhous-msft]

The discussion created in #2662 is another reason for doing this change. Level triggered mode is required to make it so the entire receive buffer does not have to be read every time an event is received.