Synchornize etcd cluster in registrator on service registration

[hawka]

In our system, we've been seeing issues where, when etcd becomes unreachable to registrator, it needs to be kicked manually to publish its local service entries to etcd again. We believe that by using etcd client's SyncCluster functionality, we can synchronize the set of etcd servers with registrator and avoid this situation. Our proposal is that a client should initially use the local proxy, then via a proxied request, it discovers the current cluster topology. That topology is subsequently used directly, and the proxy is no longer used. We think this will buy us better retry behavior against etcd servers.

[progrium]

This is great, and sounds reasonable. Has this improved the situation you initially described?

[hawka]

We haven't deployed to production yet, as we wanted to get your input first, and we're also planning to make a similar patch on our copy of confd as well. However, we did perform testing on a test etcd cluster, and the results were positive.

[progrium]

I don't see anything wrong with it, but I'll leave this open for anybody else to comment until you want to confirm you're happy with it.

[progrium]

@hawka any update on this working out?

[hawka]

@progrium I wanted to make sure we were running successfully in production before I updated this PR, and we've been running in production for about two weeks now. This patch has fixed the error we were seeing, with no negative side effects. I'm confident in asking for it to be committed, if you are amenable!

[progrium]

@hawka that's great news! Looked it over again and it still looks great.