Random thoughts of a warped mind…

August 29, 2012

Consolidate and graph statistics from multiple HAProxy servers in Nagios

Filed under: Development,Linux,Nagios,Perl — Srinivas @ 14:01

This code is applicable if you have multiple haproxy servers as backends to a load balancer (AWS/hardware, whatever)… HA Proxy allows you to setup a stats view where you can login via a browser and see statistics across different services and the frontend/backend traffic stats etc. There is also a CSV format of this to allow for easy parsing by scripts.

The configuration and the perl script discussed in further sections allow you to grab statistics from each haproxy service frontend, summarize it by service name to fetch core statistics like Bytes In/Out, Session info etc (modify the perl script to add other parameters as you deem fit) and finally dump this as performance data that Nagios can graph (assuming you have the some graph plugin setup – I recommend Nagios Graph highly).

Heres how the Service state info would look in Nagios.

Heres how the graphed data would look like (via Nagios Graph).

Enable Stats view on HA Proxy servers

Assuming you have HA Proxies lb1, lb2 and lb3 configured with make the stats data available on their port 8080 (config below):

Its assumed that all haproxies in this “group” share the same setup i.e. the services frontended for, admin stats config etc.

Nagios check script

Copy this script as /etc/nagios3/scripts/check_haproxy_all.pl and make sure Nagios user can execute this script (chmod u+x).

Note: The hash keys in %watch correspond to the HA Proxy backends I want to track stats for – these are typically the names following “backend” in haproxy.cfg. You will need to tweak these to match your backends name(s). Also, if you any of your backends is another single haproxy/load balancer, then the tag “FRONTEND” would not be available for this in the HA Proxy stats. You will need to set %watch to have a key=>value pair of the backend name in haproxy.cfg and the value matching the server name in that haproxy definition i.e. if you have “server XXXX <weight> <N> <check>” in backend beanstalk then add key/value pair beanstalk=>XXXX to the hash.

Setup Nagios configuration to use this script

Note that I set up the password for the HA Proxy stats view to be in variable USER6 in my Nagios config. Set/change this to suit your configuration. Bounce Nagios and see some graphed data in a few minutes.

Things to note

  1. HA Proxy restart will reset all counters to zero – So you may see sudden drops in graphed values. This is not a problem for me so I have’nt bothered to make this more intelligent( I like it this way!)
  2. Make sure not to open up access to HA Proxy port 8080 from everywhere – Just to your management host / nagios box.
  3. I pick up the lines matching FRONTEND from the stats CSV data (but if this is not available e.g. when the backend service is also load balanced, I name it explicity and have exceptions for this in my code, just see the hash %watch in the program and modify as needed.
  • Bahi

    Srini, I need you to email me! I have something of yours that I need to get back to you and I can’t find your email anywhere online. Hope you’re doing well buddy! -Bahi

Powered by WordPress