health, grpc: Deliver health service updates through the health listener

[arjan-bal]

As part of the dualstack changes described in A61, client side health check updates need to be delivered through the health listener instead of the raw connectivity listener. This PR creates a health producer that manages the health service client and allows registration of a listener for receiving updates. The RegisterHealthListener method in acBalancerWrapper decides whether client side health checking is enabled, if it is, it registers the health listener with the health producer. acBalancerWrapper wraps the LB policy's listener to synchronize updates from the health producer.

Existing health check tests are run against the health producer when the GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST is set to true.

RELEASE NOTES: N/A

[codecov]

Codecov Report

Attention: Patch coverage is 89.02439% with 9 lines in your changes missing coverage. Please review.

Project coverage is 82.04%. Comparing base (317271b) to head (5cf90ee).
Report is 10 commits behind head on master.

Files with missing lines	Patch %	Lines
health/producer.go	86.66%	4 Missing and 2 partials ⚠️
balancer_wrapper.go	91.89%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7900      +/-   ##
==========================================
- Coverage   82.13%   82.04%   -0.10%     
==========================================
  Files         377      378       +1     
  Lines       38179    38258      +79     
==========================================
+ Hits        31360    31389      +29     
- Misses       5525     5561      +36     
- Partials     1294     1308      +14     

Files with missing lines	Coverage Δ
balancer_wrapper.go	85.27% <91.89%> (-0.83%)	⬇️
health/producer.go	86.66% <86.66%> (ø)

... and 19 files with indirect coverage changes

[dfawley]

Consider a grpcsync.Event instead of having two fields here.

[arjan-bal]

I was using the two fields, one to trigger the cancellation of the context that is passed to the health client and the second to wait till the health check client to return. Waiting for the client close isn't required. The health checking in addrConn also cancels the context and lets the client shut down asynchronously. I got rid of the channel to wait for the client to return here.

grpc-go/clientconn.go

Lines 1342 to 1354 in c1b6b37

	onClose := func(r transport.GoAwayReason) {
	ac.mu.Lock()
	defer ac.mu.Unlock()
	// adjust params based on GoAwayReason
	ac.adjustParams(r)
	if ctx.Err() != nil {
	// Already shut down or connection attempt canceled. tearDown() or
	// updateAddrs() already cleared the transport and canceled hctx
	// via ac.ctx, and we expected this connection to be closed, so do
	// nothing here.
	return
	}
	hcancel()

[dfawley]

Would it make sense to implement a nopHealthListener that is actually a real implementation that only reports Ready, once, to avoid special-case handling here?

[arjan-bal]

Refactored to make this method return either the heallh producer's register listener function or a no-op register listener function.