Why gRPC Server closes a connection at the 2nd GOAWAY.

[hiyosi]

In my under standing, the gRPC Server closes a connection at sending the 2nd GOAWAY frame if there is no active streams.

https://github.com/grpc/grpc-go/blob/v1.53.0/internal/transport/http2_server.go#L1346-L1350
https://github.com/grpc/grpc-go/blob/v1.53.0/internal/transport/http2_server.go#L336-L342

But is there a data race between Flush() and Close() ?
Because I sometimes find the error connection rest by peer around the 2nd GOAWAY.

If some data (e.g. PING frame) is received between Flush() and Close(), the server close the connection by sending the RST packet instead of FIN packet isn't it? and an error occur at client side.

As described at

grpc-go/test/end2end_test.go

Lines 900 to 906 in 3151e83

	// 1. If the call to ss.S.Stop() causes the server's sockets to close while
	// there's still in-fight data from the client on the TCP connection, then
	// the kernel can send an RST back to the client (also see
	// https://stackoverflow.com/questions/33053507/econnreset-in-send-linux-c).
	// Note that while this condition is expected to be rare due to the
	// test httpServer start synchronization, in theory it should be possible,
	// e.g. if the client sends a BDP ping at the right time.

I would like to know the reason that the connection is closed from the server instead of waiting for the client to close the connection after the 2nd GOAWAY.
On the other hand, the same process is followed in the case of Too Many Pings, but I do not consider that to be a problem.

[easwars]

Are you running into any issue because of this behavior? If so, could you please explain that in detail.

The GOAWAY behavior that we implement is based on https://httpwg.org/specs/rfc7540.html#rfc.section.6.8. When the server sends the first GOAWAY, it usually gives some time to the client to terminate the connection gracefully, before sending the second GOAWAY (at which point, the connection is hard closed).

[hiyosi]

@easwars Thank you for replying.

In my environment, I sometimes get the connection rest by peer error while client gRPC calls and it is happening around GOAWAY.
And I captured the packets, I found out the server close the connection by sending RST packets at that case.

e.g. client logs (v1.51.0)

 error="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: read tcp ${CLIENT_IP}:${CLIENT_PORT}->${SERVER_IP}:${SERVER_PORT}: read: connection reset by peer\", received prior goaway: code: NO_ERROR

e.g. server logs (v1.51.0)

ERROR: [transport] transport: loopyWriter.run returning. Err: transport: Connection closing
WARNING: [transport] transport: http2Server.HandleStreams failed to read frame: read tcp ${SERVER_ADDR}:${SERVER_PORT}->${CLIENT_ADDR}:${CLIENT_PORT}: use of closed network connection
INFO: [transport] transport: error closing conn during Close: use of closed network connection
INFO: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

Also I found the following successful cases.

• The client receives the 1st GOAWAY, it closes the existing connection by sending FIN packets and starts a new one. (no errors)
• The client receives the 2st GOAWAY, and the server closes the existing connection by sending FIN packets. (no errors)

The gRPC server is configured as below.

grpc.NewServer(
	grpc.UnaryInterceptor(unaryInterceptor),
	grpc.StreamInterceptor(streamInterceptor),
	grpc.Creds(credentials.NewTLS(tlsConfig)),
	grpc.KeepaliveParams(keepalive.ServerParameters{
		MaxConnectionAge: 3 * time.Minute,
	}),
)

The gRPC client is configured as below.

grpc.DialContext(ctx, address,
	grpc.WithDefaultServiceConfig(`{ "loadBalancingConfig": [ { "round_robin": {} } ] }`),
	grpc.FailOnNonTempDialError(true),
	grpc.WithBlock(),
	grpc.WithReturnConnectionError(),
	grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
)

The configuration is intended for load balancing connections.
I have created similar issues before and understand the behavior of GOAWAY ( #5969 )

So I am wondering if there is a possibility race in case of the server closes the connection like described the comments of test case.

grpc-go/test/end2end_test.go

Lines 900 to 906 in 3151e83

	// 1. If the call to ss.S.Stop() causes the server's sockets to close while
	// there's still in-fight data from the client on the TCP connection, then
	// the kernel can send an RST back to the client (also see
	// https://stackoverflow.com/questions/33053507/econnreset-in-send-linux-c).
	// Note that while this condition is expected to be rare due to the
	// test httpServer start synchronization, in theory it should be possible,
	// e.g. if the client sends a BDP ping at the right time.

In addition, I have disabled dynamic windows and BDP in the client settings as shown in #5358, I have not seen any errors at this time.

[easwars]

In my environment, I sometimes get the connection rest by peer error while client gRPC calls and it is happening around GOAWAY. And I captured the packets, I found out the server close the connection by sending RST packets at that case.

Do you feel that the server is closing the connection when it should not have? Or do you feel that the server is not giving enough time for the client to gracefully close the connection from its end?

Because I still don't understand what problem you are facing.

[hiyosi]

I am not sure, why the server send RST packets.
I suspect that the problem may be caused by timing.

e.g. If the server closed the connection while there were still unhandled data in the socket buffer it will send a RST packet.

Is there a possibility like above around GOAWAY?

[dfawley]

I would like to know the reason that the connection is closed from the server instead of waiting for the client to close the connection after the 2nd GOAWAY.

We used to rely on the client closing the connection, but that is a bug. The server needs to protect itself from a client that does not close the connection, and there are clients in the wild that don't do this properly. This was actually the case in v1.51, which you were using. It was only fixed in #5968.

Can you test this again with v1.53? It's possible the changes made since v1.51 fixes a problem you may be seeing.

[hiyosi]

@dfawley

Yes, I will test with v1.53. Please wait for some days due to low reproducibility.

[easwars]

I am not sure, why the server send RST packets. I suspect that the problem may be caused by timing.

Yes, I think so too. And this is not under the control of the gRPC server. This is under the control of the TCP implementation in the kernel.

e.g. If the server closed the connection while there were still unhandled data in the socket buffer it will send a RST packet. Is there a possibility like above around GOAWAY?

gRPC server flushes all data in its write buffer before closing the TCP connection.

Also, what I feel is more important in this scenario is that the gRPC server should stop handling new streams on the connection once it sends the second GOAWAY frame. And it is currently doing that, and I was able to verify the same.

[hiyosi]

@dfawley
The error is occurred even in v1.53.

@easwars
Thank you for replying.
Is the only error mitigation I can do on gRPC layer is to "disable BDP estimation"? Any other good mitigations?