Showing posts with label graphite. Show all posts
Showing posts with label graphite. Show all posts

Sunday, February 3, 2013

Graphite: Vim vs Emacs

Stardate: 90697.76
Graph all the things! Graphite is a real time graphing tool. It allows sysadmins like myself to visualize statistics about our environment and change over time. We can also, using the graphite dashboard, easily map different data sources onto each other to try to find correlations or just to look at differences in use.
Nightfly, a co-worker of mine, has developed a script to run against our college's general login boxes. These boxes are used by the CS, ECE, and other departments. It provides a good picture of what people are using against time and against each other. Obviously the first order of business is to prove which editor is more popular:
The program to collect and submit this data is on github. The botnet element of this is hacked together with cron and ssh.
Big thanks to Nightfly for making the tech behind this post.

Monday, January 28, 2013

PuppetDB/Storeconfigs Cache expiry

Stardate: 90682.98
After a couple of weeks of getting frustrated with puppet's Storeconfigs/puppetdb features, I have emerged victorious. PuppetDB is the newer, better, postgressier backend for puppet Storeconfigs. PuppetDB sports some really nice features including a fancy status/metrics web dashboard:
As you can see this is some interesting and potentially beneficial feedback. It is updated live and is mobile browser compatible. Personally, I'm happy to get graphs of this data any way I can, but I would prefer not to be locked into their dashboard. I would rather be able to get these data out of an often updated file or udp port so that I could send it to graphite for real time graphing and correlation with other metrics. I also don't see the point of having it be mobile friendly, since most everyone will have their puppetmaster/puppetdb server firewalled heavily and mobile devices have no business on the internal network. Some of the metrics can lead to actually tuning and performance boosts: mostly this is in the increasing the number of threads and the max jvm heap size.
The punchline here is that with
storeconfigs = true
in puppet.conf you can do exported/collected resource magics. When doing this with nagios resources I've been able to export and collect resources flawlessly. The problems came up when I tried to modify a resource. Since we make heavy use of dynamic git environments with puppet I was running something like
 puppet agent --test --environment=nagios 
on a host at random and
 puppet agent --test --environment=nagios 
on the nagios server, hoping to collect exported resources. The problem was they were not changing. As it turns out puppetdb can cache old exported resources for up to an hour. My advice for others having problem getting nagios or other exported resources to change or purge is to give it time. Run a big ssh for loop or use mcollective to hit all your boxes and hit the coffee cart for a quick pick me up. Chances are good you just need to give it time.