HA - Automatic Failover

[eveiga]

Hi! First of all, thanks for the proxy, it has been really helpful :)

I'm in need of a decent solution for automatic failover and already stated that twemproxy doesn't support it. Any thoughts or ideas on it?

I was thinking on a external process that would leverage the use o redis-sentinel and on a master-switch event updates the IP address on nutcracker.conf and restarts the service.

[manjuraj]

Glad you liked it @eveiga

I believe using the external process the way you described makes sense. In fact you can have two twemproxy processes running - one routing the traffic to all the masters and the other to all the slaves. On a failover event, you switch from one twemproxy to the other

[bmatheny]

@manjuraj @eveiga that's what we do for memcache when there are events like a total failure (external process). Works quite well.

[eveiga]

@manjuraj I've already thought on that solution. Can I use the slaves cluster to perform read operations? Or the hashes wont pair with the ones for the master cluster?

@bmatheny are you using my sugestion or manjuraj one?

[bmatheny]

@eveiga the one you recommended. When the topology needs to change the config is updated by an external process and twem gets restarted.

[eveiga]

@bmatheny sorry for the boring questions :) dont you experience a window of downtime during that restart? If yes, How do you cope with that?

BTW, are you using any pool of twemproxy just with slaves for reading?

[eveiga]

Humm, I forgot you are using it with memcache, dont know if the last question fits your use case!

[bmatheny]

We do see a short burst of errors. The error type is detected by the app and retried, so we generally don't 'lose' writes, and reads will fall back to the DB.

[eveiga]

@bmatheny Thanks for the tips, I'll go on with that solution!

[matschaffer]

@eveiga thanks for the redis-sentinel reminder. So far it looks like this will work well.

Has anyone built the bits to update twemproxy when redis-sentinel finishes a failover?

@manjuraj would you recommend anything more graceful than simply rewriting the twemproxy config and restarting it?

[eveiga]

@matschaffer Yes, I've developed a simple service that attaches a handler to the "master-switch" event emitted by redis-sentinel, updates twemproxy.conf with the new info and restartes the service. So far so good with the tests, I'll put it in production in a short time.

[matschaffer]

@eveiga any chance of sharing what you've come up with?

[eveiga]

No problem. It's on node.js and a bit tight with our structure, still want it?

[matschaffer]

Sure! Even just a gist is great. Always nicer to have some collaboration. :)

On Feb 26, 2013, at 9:47, eveiga notifications@github.com wrote:

No problem. It's on node.js and a bit tight with our structure but, still
want it?

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/67#issuecomment-14118050
.

[eveiga]

https://gist.github.com/eveiga/5039007

As I said, it's pretty tight with our structure (init scripts path, mails, etc) and could be a lot configurable, but it can give you a starting point.

Sugestions are welcome!

[manjuraj]

it you guys can make this generic enough, we can check this into the scripts/ folder of twemproxy