Issues in environments with intermittent connectivity

[gracenoah]

What version of gRPC are you using?

1.20.1 and upgrading to any more recent version

What version of Go are you using (go version)?

1.12.7

What operating system (Linux, Windows, …) and version?

Linux, various

What did you do?

We use GRPC in an environment with intermittent connectivity where packets are silently dropped for a prolonged period of time, leaving only a 1 minute gap when packets can be sent through.

What did you expect to see?

To be able to continue to use GRPC after the upgrade past 1.20.1

What did you see instead?

Excessive errors and delays, messages often completely fail to get through or get stuck for a part of the 1 minute window of opportunity.

I think that the issue is caused by #2760 and the 30 second wait that it introduces. I don't know the code well enough, so I'm currently experimenting with adding an option to disable the rate limit. If that works, I'll make a pull request with my changes that add this option. Let me know if my theory makes sense or if there are other potential issues in environments with intermittent connectivity.

I'll update this issue as new information becomes known.

[menghanl]

What error messages and logs do you get when the failure happens?
(Logs can be turned on by https://github.com/grpc/grpc-go#how-to-turn-on-logging)

The issue you referred to is a change to the name resolver. So if your backends aren't changing a lot, it in theory shouldn't have affected you.

[dfawley]

Ping @gracenoah

[gracenoah]

Sorry, I really want to get to the bottom of this because we have pinned our GRPC version to 1.20.1 and are too scared to upgrade without thorough testing and debugging this issue. Unfortunately, I have some really tight deadlines coming up in the next month or two so it's very difficult to allocate time to testing this. I'll update the issue when we get a chance to root cause this and/or confirm that the DNS change is causing the failures.

[dfawley]

No problem @gracenoah - if the issue is locked by the time you get a chance to look at this, feel free to file a new issue. Otherwise, comment here and we can reopen it. Thanks!