It is impossible to set a custom polling frequency in DNS resolver

[kop]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.7.2

What version of Go are you using (go version)?

go1.9.2 darwin/amd64

What did you do?

I was trying to play with new Load Balancing APIs for gRPC services running in K8S.

What did you expect to see?

I expected to see an option to configure default DNS resolver and set custom frequency of polling the DNS server.

What did you see instead?

Resolver has a hardcoded value of 30 minutes that is absolutely unusable for dynamic environments (like K8S).

[menghanl]

With the current code, right, there's no way to set a custom polling frequency for DNS.
An easy workaround would be to add a new function dns.NewBuilderWithFreq(freq time.Duration) (similar to this). Users can create a new DNS builder with this, and register it to override the default DNS builder.

Another problem here is, it's very hard to find the perfect polling frequency.

I'm working on a change to make resolver re-resolve the name whenever a connection goes down. With this, we plan to remove the polling frequency in resolver.
But the resolver won't re-resolve if all connections are working. Does this work for K8S?

[kop]

An easy workaround would be to add a new function dns.NewBuilderWithFreq(freq time.Duration)

Yes, this sounds good.

I'm working on a change to make resolver re-resolve the name whenever a connection goes down.
But the resolver won't re-resolve if all connections are working. Does this work for K8S?

I'm not sure it will work great. I mean it will handle cases when pod is destroyed/recreated but it will not handle cases when service is scaled up and new pods are ready to serve requests, isnt?

I think polling frequency should stay. It will be possible to set much higher values for polling frequency if targets will be reevaluated on every disconnect, but we still need a mechanism to discover new instances of the service.

[menghanl]

it will not handle cases when service is scaled up and new pods are ready to serve requests

IMO a better solution for this would be to push the updates instead of polling. But that's out of the scope of DNS.

cc @ejona86 for more inputs.

[kop]

IMO a better solution for this would be to push the updates instead of polling.

Totally agreed.

But that's out of the scope of DNS.

DNS is a native service discovery mechanism in K8S and i think it would be really great if it will work with gRPC out of the box :)

[ejona86]

I mean it will handle cases when pod is destroyed/recreated but it will not handle cases when service is scaled up and new pods are ready to serve requests, isnt?

The solution for this is MAX_CONNECTION_AGE. This has the advantage that the service owner controls the configuration. (It also works with L4 proxies where the address doesn't change.)

[trusch]

DNS in k8s only returns the Virtual IP of the service and not all IP's of the replicas. This will not work when reusing a single client conn.

I saw a k8s resolver implementation which resolves natively via the k8s api.

[kop]

@trusch but we can use Headless Services - they return multiple A records, this should solve the problem.

[ejona86]

DNS in k8s only returns the Virtual IP of the service and not all IP's of the replicas. This will not work when reusing a single client conn.

@trusch, what do you mean by "not work?" It should "work" as in function, in a basic sense. It would also load balance multiple clients to multiple servers and rebalance the clients over time (via MAX_CONNECTION_AGE). But I would agree it does not help distributing load from a single client to multiple servers. Is this what you mean?

Our (internal-discussion only?) proposed solution there is to create multiple ClientConns and round-robin over them. To make this easier, we could enhance grpc to create the multiple ClientConns behind a single ClientConn so it becomes mostly transparent. The extra connections would not be guaranteed to get different backends, but that is a normal property of L4 LBs.

The discussions/work to do this stalled. But this work can be re-opened if necessary. But either being okay with what you have (because, let's say, you don't a single client causing a substantial amount of load) or exposing IPs to your clients to avoid the L4 LB sounds superior in this case.

[trusch]

@ejona86 yes, I got a setup with gateways and workers and would like to have one ClientConn to the workers which loadbalances on a per call basis. I think using headless services like @kop suggested and the new DNS resolver (in combination with MAX_CONN_AGE) will do the job for me.

[dfawley]

Resolving with "working as intended", since we don't intend to add support for custom polling frequencies to the default DNS resolver -- please let us know if you don't think that's appropriate.