Random thoughts of a warped mind…

September 17, 2014

Reconfig for service discovery

Filed under: Amazon EC2,Chef,Git,Linux,Redhat OpenShift,Ruby,Virtualization — Srinivas @ 09:49

Imagine a constantly changing fleet of servers… New servers (or virtual machines) going online as capacity is needed or being taken offline as load drops… Maybe you already use Chef/Puppet to bootstrap your servers and use their node attributes to populate N servers in a load balancer (Haproxy for example) configuration. 

Now adding/removing a new server (a backend, maybe a rails/tomcat server, whatever) would mean that chef-client has to run on all the haproxy boxes so they know about the new backend (or the one that went away). This would work if you  run chef-client every 5 minutes or so? But why? Chef/Puppet are primarily meant to bootstrap your servers and not to sync state. Enter Reconfig and service discovery.

Reconfig uses as its backend an etcd cluster (Thanks to awesome folks at CoreOS!). Etcd is a lightweight http accessible zookeeper alternative which uses Raft protocol to provide a highly available key/value store. Reconfig rides on top on etcd and provides a mechanism to watch specific etcd keys (that your backends would set/unset when they go online/offline – +1 for TTL based keys) and uses changes in keys (or a tree of keys) to trigger a config file change – the “new” config being built from a template you provide and the key/value list from etcd. 

This means service discovery can be done almost as soon your new node comes online… No lengthy chef/puppet runs… And while Reconfig is meant to complement Chef/any-other-config-management-system, it can also be used by itself. Does’nt necessarily have to manage only  haproxy/varnish configs, could very well use it to force config changes for other daemons or simply file level updates too.

Try it out – I am looking for other contributors to make this better! – Reconfig is at https://github.com/srinivasmohan/reconfig . Reconfig is work in progress and expect more changes coming soon!

p.s. – Reconfig was inspired by confd – Liked its core concept but wanted more. And why do Go when theres Ruby! :-)

Powered by WordPress