Just What Is OpenStack Infra?

I work for HP doing two things. By day I work inside the HP firewall setting up and running a CI system for testing HP's OpenStack technology. We call this system Gozer. (By the way we are hiring). By night I work upstream (in the Open Source world) with the OpenStack Infrastructure Team setting up and running a CI system for OpenStack developers.

This blog post concerns my work upstream.

One of my chief initiatives since joining the team two years ago is to make the Puppet codebase used by infra more in-line with standards, more reusable, and generally better. I have never attempted to use infra as a testbed for experimental uses of Puppet, I've always tried to apply the best practices known in the community. Of course there are exceptions to this (see all the Ansible stuff). This initiative is codified in a few different specifications accepted by the team (you don't need to read these):

• Puppet codebase Refactor
• Git Repo per Module
• Move core configuration files to their own repo
• Create a public hiera directory in the main repo
• Puppet 4 GO GO GO
• Functional Testing of Puppet (beaker-rspec)

One mark of the success of this ongoing initiative is that I am now in a place where I am recommending parts of our code to other people in my community. Those are the people for whom I intend this blog post. Someone sees a neat part of the Puppet OpenStack 'stuff' and wants to use it, but it needs a patch or a use case covered. This blog post is supposed to provide a high level overview of what we do, who 'we' are, and the bigger pieces and how they interact with each other. We'll start with a long series of names and definitions.

Naming things is hard

So what is OpenStack? OpenStack is an Open Source software collection based around providing cloud software. The OpenStack Foundation is a nonprofit organization that provides centralized resources to support the effort, this comes in both technical (sysadmins) and other forms (legal, conference organizing, etc). OpenStack is made up of many components, the simplest is that 'nova' provides a compute layer to the cloud i.e. kvm or xen management.

OpenStack can be installed with Puppet. The Puppet code that does this is called "OpenStack Puppet Modules." These modules install OpenStack services such as nova, glance, and cinder. Their source code is available by searching for openstack/puppet-*. The team that develops this code is called the OpenStack Puppet Module Team. This team uploads to the forge under the namespaces 'openstack' or 'stackforge.'

I do not work with these modules on a daily basis.

I work with the OpenStack Infrastructure Team. This team deploys and maintains the CI system used by OpenStack upstream developers. We have our own set of Puppet modules that are completely unrelated to the OpenStack Puppet Modules. Their source code can be found by searching for openstack-infra/puppet-*. These modules are uploaded under the forge namespaces 'openstackci' and 'openstackinfra.' We use these modules to deploy services like Gerrit, Jenkins, and Drupal. We also have a number of utility modules useful for generic Linux administration. We have Precise, Trusty, Centos 6, and various Fedora flavors in our infrastructure, so our modules often have good cross-platform support.

Central Nexus

All the openstack-infra/puppet-* modules are consumed from master by our 'central nexus' repository: system-config. System-config uses a second repository for flat-files: project-config. System-config contains node definitions, public hiera data(soon), a few utility scripts, a modules.env file, a single module to stick 'roles' in called 'openstack_project'. The more 'core' roles in openstack_project call out to another repo called: puppet-openstackci. The secrets are stored in a hiera directory that is not public.

Crude Drawing

The crude drawing above shows a typical flow. A node definition lives in site.pp, which include a role class from openstack_project, which includes a role class from the openstackci module, which then uses resources and classes from the other modules, in this case puppet-iptables.

There are other code paths too. Sometimes, often in fact, an openstack_project role will include openstack_project::server or openstack_project::template, these classes wrap up most of the 'basics' of linux administration. Template or server will go on to include more resources.

There are multiple places to integrate here. At the most basic, a Puppet user could include our puppet-iptables module in their modulepath and start using it. An individual who wants a jenkins server or another server like ours could use openstackci and it's dependencies and write their own openstack_project wrapper classes to include openstackci classes.

We do not encourage site.pp or openstack_project classes to be extended at this time, we instead encourage features or compatibility extensions to be put into openstackci or the service-specific modules themselves. This is a work in progress and some important logic still lives in openstack_project and should be moved out. A stretch-goal is to move to a place where all of openstack infra runs out of openstackci, providing only a hiera yaml file to set parameters.

Continuous Deployment

A note about modules.env: OpenStack-infra has a modules.env file instead of a Puppetfile. This file contains the location, name, and ref of git repositories to put inside the modulepath on the Puppetmaster. OpenStack infra deploys all of its own Puppet modules from master, so any change to any module can break the whole system. We counteract this danger by having lots of testing and code review before any change goes through.

A note about project-config: One of the patterns we use in OpenStack Infra is to push our configuration into flat files as much as possible. We have one repository, project-config, which holds files that control the behaviour of our services, Puppet's job is only to copy files out of the repo and into the correct location. This makes it easier for people to find these often-changed files, and means we can provide more people access to merge code there than we would with our system-config repository.

A note about puppet agent: We run puppet-agent, but it is fired from the Puppetmaster by an ansible run. We hope to move to puppet apply triggered by ansible soon.

The part where I give you things

There are two modules right now that you might be interested in using yourself. The first is our puppet-httpd module. This module was forked from puppetlabs-apache at version 0.0.4. It has seen some minor improvements from us but nothing major, other than a name change from 'apache' to 'httpd'. You can see why we forked in the Readme of the project but the kicker is that this module allows you to use raw 'myhost.vhost.erb' templates with apache. You no longer need to know how to translate the apache syntax you want into puppetlabs-apache parameters. Let's see what this looks like:

openstack_project/templates/status.vhost.erb:
# ************************************
# Managed by Puppet
# ************************************

NameVirtualHost <%= @vhost_name %>:<%= @port %>
<VirtualHost <%= @vhost_name %>:<%= @port %>>
  ServerName <%= @srvname %>
<% if @serveraliases.is_a? Array -%>
<% @serveraliases.each do |name| -%><%= " ServerAlias #{name}\n" %><% end -%>
<% elsif @serveraliases != '' -%>
<%= " ServerAlias #{@serveraliases}" %>
<% end -%>
  DocumentRoot <%= @docroot %>

  Alias /bugday /srv/static/bugdaystats
  <Directory /srv/static/bugdaystats>
      AllowOverride None
      Order allow,deny
      allow from all
  </Directory>

  Alias /reviews /srv/static/reviewday
  <Directory /srv/static/reviewday>
      AllowOverride None
      Order allow,deny
      allow from all
  </Directory>

  Alias /release /srv/static/release

  <Directory <%= @docroot %>>
    Options <%= @options %>
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  # Sample elastic-recheck config file, adjust prefixes
  # per your local configuration. Because these are nested
  # we need the more specific one first.
  Alias /elastic-recheck/data /var/lib/elastic-recheck
  <Directory /var/lib/elastic-recheck>
      AllowOverride None
      Order allow,deny
      allow from all
  </Directory>

  RedirectMatch permanent ^/rechecks(.*) /elastic-recheck
  Alias /elastic-recheck /usr/local/share/elastic-recheck
  <Directory /usr/local/share/elastic-recheck>
      AllowOverride None
      Order allow,deny
      allow from all
  </Directory>


  ErrorLog /var/log/apache2/<%= @name %>_error.log
  LogLevel warn
  CustomLog /var/log/apache2/<%= @name %>_access.log combined
  ServerSignature Off
</VirtualHost>


::httpd::vhost { 'status.openstack.org':
  port     => 80,
  priority => '50',
  docroot  => '/srv/static/status',
  template => 'openstack_project/status.vhost.erb',
  require  => File['/srv/static/status'],
}


If you don't need a vhost and just want to serve a directory, you can:

::httpd::vhost { 'tarballs.openstack.org':
  port     => 80,
  priority => '50',
  docroot  => '/srv/static/tarballs',
  require  => File['/srv/static/tarballs'],
}

The second is puppet-iptables, which provides the ability to spit direct iptables rules into a Puppet class and have those rules set. You can also specify the ports to open up. Again this is an example of weak modeling. Concat resources around specific rules are coming soon in this change. Let's see what using the iptables module looks like:

class { '::iptables':
  public_tcp_ports => ['80', '443', '8080'],
  public_udp_ports => ['2003'],
  rules4           => ['-m state --state NEW -m tcp -p tcp --dport 8888 -s somehost.openstack.org -j ACCEPT'],
  rules6           => ['-m state --state NEW -m tcp -p tcp --dport 8888 -s somehost.openstack.org -j ACCEPT'],
} 

This enables you to manage iptables the way you view iptables. It is easy to debug, easy to reason about, and extensible. We think it provides a significant advantage over the puppetlabs-firewall module. Unfortunately, the puppet-iptables module currently is hardcoded to open up certain openstack hosts, that should be fixed very soon (possibly by you!). Both of these modules try to be as simple as possible.

Getting these modules right now is done through git. If you don't want to ride the 'master' train with us, you can hop in #openstack-infra on freenode and ask for a tag to be created at the revision you need. We're working on getting forge publishing in to the pipeline, it's not a priority for us right now but if you need it you can ask for it and we can see about increasing focus there.

There are two generic modules that advance the puppet ecosystem coming out of OpenStack Infra and we hope there will be more to come. If you'd like to help us develop these modules we'd love the help. You can start learning how to contribute to OpenStack here.

Inspecting Puppet Module Metadata

Last week at #puppethack, @hunner helped me land a patch to stdlib to add a load_module_metadata function. This function came out of several Puppet module triage sessions and a patch from @raphink inspired by a conversation with @hirojin.

The load_module_metadata function is available in master of puppetlabs-stdlib, hopefully it will be wrapped up into one of the later 4.x releases, but will almost certainly make it into 5.x.

On it's own this function doesn't do much, but it is composable. Let's see some basic usage:

$: cat metadata.pp

$metadata = load_module_metadata('stdlib')

notify { $metadata['name']: }

$: puppet apply --modulepath=modules metadata.pp
Notice: Compiled catalog for hertz in environment production in 0.03 seconds
Notice: puppetlabs-stdlib
Notice: /Stage[main]/Main/Notify[puppetlabs-stdlib]/message: defined 'message' as 'puppetlabs-stdlib'
Notice: Finished catalog run in 0.03 seconds

As you can see this isn't the most amazing thing ever. However access to that information is very useful in the following case:


$apache_metadata = load_module_metadata('apache')

case $apache_metadata['name'] {
  'puppetlabs-apache': {
    # invoke apache as required by puppetlabs-apache
   }
  'example42-apache': {
    # inovke apache as required by example42-apache
  }
  default: {
    fail("Apache module author not recognized, please add it here")
  }
}

This is an example of Puppet code that can inspect the libraries loaded in the modulepath, then make intelligent decisions about how to use them. This means that module authors can support multiple versions of 'library' modules and not force their users into one or the other.

This is a real problem in Puppet right now. For every 'core' module there are multiple implementations, with the same name. Apache, nginx, mysql, archive, wget, the list goes on. Part of this is a failure of the community to band behind a single module, but we can't waste time finger pointing now. The cat is out of the bag and we have to deal with it.

We've had metadata.json and dependencies for a while now. However, due to the imperfectness of the puppet module tool, most advanced users do not depend on dependency resolution from metadata.json. At my work we simply clone every module we need from git, users of r10k do much the same.

load_metadata_json enables modules to enforce that their dependencies are being met. Simply put a stanza like this in params.pp:

$unattended_upgrades_metadata = load_module_metadata('unattended_upgrades') 
$healthcheck_metadata = load_module_metadata('healthcheck')

if versioncmp($healthcheck_metadata['version'], '0.0.1') < 0 {
  fail("Puppet-healthcheck is too old to work")
}
if versioncmp($unattended_upgrades_metadata['version'], '2.0.0') < 0 {
  fail("Puppet-unattended_upgrades is too old to work")
}

As we already saw, modules can express dependencies on specific implementations and versions. They can also inspect the version available and use that. This is extremely useful when building a module that depends on another module, and that module is crossing a symver major version boundary. In the past, in the OpenStack modules, we passed a parameter called 'mysql_module_version' to each class which allowed that class to use the correct invocation of the mysql module. Now classes anywhere in your puppet code base can inspect the mysql module directly and determine which invocation syntax to use.

$mysql_metadata = load_module_metadata('mysql')

if versioncmp($mysql_metadata['version'], '2.0.0') <= 0 {
  # Use mysql 2.0 syntax
} else {
  # Use mysql 3.0 syntax
}

Modules can even open up their own metadata.json, and while it is clunky, it is possible to dynamically assert that dependencies are available and in the correct versions.

I'm excited to see what other tricks people can do with this. I'm anticipating it will make collaboration easier, upgrades easier, and make Puppet runs even more safe. If you come up with a neat trick, please share it with the community and ping me on twitter(@nibalizer) or IRC: nibalizer.

Managing patchset stacks with git-review

In OpenStack, we use gerrit and git-review to propose changes to the repository. The workflow for that is pretty complicated, and downright confusing if you are coming from the github workflow.

One of the places where it gets hard/complicated/annoying to use our workflow is if you have multiple dependent changes. I have a technique I use, that I will present below.

The situation: You have two patches in a stack. There is a bug in the first patchset that you need to fix.

The simple play: Checkout the patchset with 'git review -d <review number>', ammend and git-review. The problem with this is that now you need to go rebase all dependent patchsets against this new review. Sometimes you can get away with using the 'rebase' button but sometimes you cannot.

What I do: I use 'git rebase -i HEAD~2' and use 'edit' to change the commit that needs to be changed, rebase goes ahead and auto-rebases everything above it (pausing if needed for me to fix things), then I can 'git review' and it will update all the patchsets that need to be changed.

This approach works for any sized stack, but using it on a two-stack is the simplest example that works.



The git log before we start:

commit e394aba4321f6d30131793e69a4f14b011ce5560
Author: Spencer Krum <nibz@spencerkrum.com>
Date:   Wed May 6 15:43:27 2015 -0700

    Move afs servers to using o_p::template
    
    This is part of a multi-stage process to merge o_p::server and
    o_p::template.
    
    Change-Id: I3bd3242a26fe701741a7784ae4e10e4183be17cf

commit 3e592608b4d369576b88793377151b7bfaacd872
Author: Spencer Krum <nibz@spencerkrum.com>
Date:   Wed May 6 15:38:23 2015 -0700

    Add the ability for template to manage exim
    
    Managing exim is the one thing sever can do that template cannot.
    This is part of a multi stage process for merging server and template.
    
    Change-Id: I354da6b5d489669b6a2fb4ae4a4a64c2d363b758

Note that we have two commits and they depend on each other. The bug is in 3e592608b4d369576b88793377151b7bfaacd872. We start the interactive rebase below, first with a vim session then with output on the command line. The vim session:

$ git rebase -i HEAD~2


  1 e 3e59260 Add the ability for template to manage exim
  2 pick e394aba Move afs servers to using o_p::template
  3 
  4 # Rebase af02d02..e394aba onto af02d02
  5 #
  6 # Commands:
  7 #  p, pick = use commit
  8 #  r, reword = use commit, but edit the commit message
  9 #  e, edit = use commit, but stop for amending
 10 #  s, squash = use commit, but meld into previous commit
 11 #  f, fixup = like "squash", but discard this commit's log message
 12 #  x, exec = run command (the rest of the line) using shell
 13 #
 14 # These lines can be re-ordered; they are executed from top to bottom.
 15 #
 16 # If you remove a line here THAT COMMIT WILL BE LOST.
 17 #
 18 # However, if you remove everything, the rebase will be aborted.
 19 #
 20 # Note that empty commits are commented out

Note that the 'top' commit in the rebase view is the 'bottom' commit in the git log view, because git is stupid. We change the 'pick' to 'e' for 'edit' meaning stop at that point for ammending. And the shell output:

Stopped at 3e592608b4d369576b88793377151b7bfaacd872... Add the ability for template to manage exim

You can amend the commit now, with

  git commit --amend

Once you are satisfied with your changes, run

  git rebase --continue

 (master|REBASE-i 1/2)$: git st
rebase in progress; onto af02d02
You are currently editing a commit while rebasing branch 'master' on 'af02d02'.

  (use "git commit --amend" to amend the current commit)
  (use "git rebase --continue" once you are satisfied with your changes)



nothing to commit, working directory clean

Then we make changes to modules/openstack_project/manifests/template.pp (not shown) and continue the rebase:

 (master *|REBASE-i 1/2)$: git st
rebase in progress; onto af02d02

You are currently editing a commit while rebasing branch 'master' on 'af02d02'.

  (use "git commit --amend" to amend the current commit)
  (use "git rebase --continue" once you are satisfied with your changes)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  modified:   modules/openstack_project/manifests/template.pp

no changes added to commit (use "git add" and/or "git commit -a")

 (master *|REBASE-i 1/2)$: git add modules/openstack_project/manifests/template.pp
 (master +|REBASE-i 1/2)$: git rebase --continue
[detached HEAD 6ca26e9] Add the ability for template to manage exim
 1 file changed, 7 insertions(+)
Successfully rebased and updated refs/heads/master.

Then we publish our changes with git-review:

(master u+2)$: git review

You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits. Otherwise you should consider squashing your changes into one
commit before submitting.

The outstanding commits are:

2bc78a8 (HEAD, master) Move afs servers to using o_p::template
6ca26e9 Add the ability for template to manage exim

Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: yes

remote: Resolving deltas: 100% (4/4)
remote: Processing changes: updated: 2, refs: 2, done    
To ssh://krum-spencer@review.openstack.org:29418/openstack-infra/system-config.git
 * [new branch]      HEAD -> refs/publish/master

With that we have changed a commit deep in the stack, rebased any commits above it, and published our changes to the gerrit server.

Debain packaging

Stardate: 90784.6

Git-sync is a script in ruby we use at work for managing git repos. It is covered in an earlier post. I got tired of ensuring it as a file in puppet and decided to make a debian package. Here is the summary of how to make a simple debian package containing just a single file. Note that the answer to this stack overflow question is the source of most of my knowledge, so this will just be annotations and extensions to that.

Debian/Ubuntu packaging (on an ubuntu system) required me to install a single package: devscripts.

At a high level, debian packaging involves creating a 'debian' folder in your source tree and putting several metadatafiles in it. Figuring out the precise contents of these files is the challenge of packaging. I recommend you use the 'apt-get source git' command to get the source of a working package (git in this case) to compare to your own metadatafiles.

Debian/Ubuntu packaging using debuild creates files one level above your current working directory(wtf debian). So the first step is to build a build directory like:

cd ~/devel
mkdir git-sync-build

Procure the source:

nibz@darktemplar:~/devel/git-sync-build$ git clone git@github.com:pdxcat/git-sync
nibz@darktemplar:~/devel/git-sync-build$ ls
git-sync
nibz@darktemplar:~/devel/git-sync-build$ cd git-sync
nibz@darktemplar:~/devel/git-sync-build/git-sync$ mkdir debian

All of the metadata files that debuild, the utility that will actually build the .deb, needs are going to be in the debian directory. 

The first file to create is the debian/changelog file. This file is created with the dch utility. Run it from the git-sync directory. It will open vim and it will look like this. Many fields here need to be changed.
dch --create

PACKAGE (VERSION) UNRELEASED; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- Spencer Krum   Thu, 07 Mar 2013 01:40:18 -0800


PACKAGE refers to the name of the package. Replace the word PACKAGE with the name you want your package to register itself as. In my git-sync case I will use 'git-sync'. The package name must be lower case. The VERSION must be replaced with a version number. I'm using 1.0.1 for this, since it is the second release of git-sync, but the changes are very minor. There are long articles on the internet about version numbering. It's not my place to comment here. The RELEASE variable needs to be replaced with a debian or ubuntu codename such as 'precise' or 'wheezy'. I have no idea what urgency is, but setting it to low doesn't seem to hurt anything. Maybe this is how you tell apt/dpkg about security updates. The initial release stuff is fine. The name is a bit tricky. Later on we will gpg sign the package. Make sure the name and email in the changelog match exactly the name and email on your gpg key, else the debuild utility won't attempt to have you gpg sign it at all. My changelog looks like this:
git-sync (1.0.0) precise; urgency=low

  * Initial release. 

 -- Spencer Krum   Wed, 06 Mar 2013 16:46:14 -0800


Next create a debian/copyright file:
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: myScript
Upstream-Contact: Name, 

Files: *
Copyright: 2011, Name, 
License: (GPL-2+ | LGPL-2 | GPL-3 | whatever)
 Full text of licence.
 .
 Unless there is a it can be found in /usr/share/common-licenses


I elected for the apache2 license and to use the two paragraph version of that license. I also gave credit where it was due here. Fill out this file as you see fit.

Next create a debain/compat file: 
nibz@darktemplar:~/devel/git-sync-build/git-sync/debian$ echo 7 > compat

Next create the rules file. This file seems to be the work-doer in debian packaging. It is evaluated by make, which is picky, so make sure that indented line is a real tab(copying from my blog will probably fail). The --with python is... well I have no idea. I traced it to a python.pem (pem is a perlism) deep within /usr. Since I am packaging a ruby script I just removed it.

Example from stackoverflow
#!/usr/bin/make -f

%:
    dh $@ --with python2

git-sync version
#!/usr/bin/make -f

%:
	dh $@

Next make the control file. Make the natural substitutions here. I guessed on section and it just sorta worked.
nibz@darktemplar:~/devel/git-sync-build/git-sync/debian$ cat control 
Source: git-sync
Section: ruby
Priority: optional
Maintainer: Spencer Krum, 
Build-Depends: debhelper (>= 7),
               ruby (>= 1.8.7)
Standards-Version: 3.9.2
X-Ruby-Version: >= 1.8.7


Package: git-sync
Architecture: all
Section: ruby
Depends: ruby, ${misc:Depends}, ${python:Depends}
Description: Git syncing script, pull based
  Git-sync allows git repositories to be kept in sync via git
  hooks or other means. Pull based, able to handle force pushes
  and submodules

Next make the install file. I went with the default in the stackoverflow post. I attempted to make some simple modifications to it(moving the file to /usr/local/bin) and that made it fail so this file is evidently pretty finicky.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ cat debian/install 
git-sync usr/bin


Now you can build the debian package.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ debuild --no-tgz-check

If all went well, it should ask you to decrypt your gpg key twice and build a package in the directory one level up.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ ls ..
git-sync                git-sync_1.0.0_amd64.build    git-sync_1.0.0.dsc
git-sync_1.0.0_all.deb  git-sync_1.0.0_amd64.changes  git-sync_1.0.0.tar.gz

You now have a shiny .deb file that can be installed with dpkg -i git-sync_1.0.0_all.deb

It is easy to put this in a launchpad PPA if you have a launchpad account. From your launchpad homepage (a shortcut is https://launchpad.net/~ if you are signed in). Press the "Create new PPA". Fill out the form. 

Next build a source package. Launchpad PPAs take source packages and build binary packages on launchpad servers. Build it with:
nibz@darktemplar:~/devel/git-sync-build/git-sync$ debuild -S

It should go through the gpg motions again and build a source file.
Then you should be able to run something like(with your launchpad username and name of PPA):
dput ppa:krum-spencer/git-sync-ppa git-sync_1.0.0_source.changes


Happy Packaging!

Git-sync

Stardate: 90784.4428183

Where I work(read as: play) we use a lot of git. As an operator we often have a service running with its configs in git. A common pattern we use is to have a post-receive hook on the git repository set up to update the git checkout on a remote server. We accomplish this through a post-receive hook that sshes into the remote server and calls a script called git-sync with some options. The git-sync script github project forked from the puppet-sync script project that we use specifically for puppet dynamic git environments. Hunner <3 More dynamic git environments with puppet. Finch <3.

A hook for a project goes in the hooks/post-receive file of the git server's bare repo. Lets look at one now:

Example git post-receive hook

#!/bin/sh
## File: akwardly incorrect

PWD=`pwd`
REPONAME=`basename $PWD | sed 's/.git$//'`
REPO="git@gitserver.cat.pdx.edu:$REPONAME.git"
DEPLOY="/etc/nagios3/conf.d/flat"
SSH_ARGS="-i /shadow/home/git/.ssh/git@gitserver.cat.pdx.edu"
GITSYNC="nagios@nagios.cat.pdx.edu"
SYNC_COMMAND="/usr/local/bin/git-sync"

while read oldrev newrev refname
do
  BRANCH=`echo $refname | sed -n 's/^refs\/heads\///p'`
  if [ $BRANCH != "master" ]; then 
    echo "Branch is not master, therefore not pushing to nagios"
    exit 0
  fi
  [ "$newrev" -eq 0 ] 2> /dev/null && DELETE='--delete' || DELETE=''
  ssh $SSH_ARGS "$GITSYNC" "$SYNC_COMMAND" \
    --branch "$BRANCH" \
    --repository "$REPO" \
    --deploy "$DEPLOY" \
    $DELETE

done

ssh nagios@gitserver.cat.pdx.edu '/etc/init.d/nagios3 reload'

The hook will exit before doing anything if the branch is not

master

and if it is, will run the git-sync script remotely on the nagios host, then go back in to bounce the nagios service.

The git-sync script essentially performs a

git fectch; git checkout HEAD

It doesn't worry itself with merging, and it is submodule aware.

A file, .git-sync-stamp, must be created by the administrator of the system. This is how git-sync knows it is in charge of managing the repository. It is definitely not recommended that you add this file to git. However, that should more or less work if you never want to think about it. I also wrote this puppet defined type to manage the stamp, initial vcsrepo, and public_key_file for you.

A puppet defined type to initalize gitsync managed folders

define gitsync::gitsync(
  $deploy,
  $repo,
  $user,
  $source,
  $public_key,
  $public_key_type
){

  ssh_authorized_key { "${user}-${name}-gitsync":
    user   => $user,
    ensure => $present,
    type   => $public_key_type,
  }

  vcsrepo { $deploy:
    ensure    => present,
    provider  => git,
    user      => $user,
    source    => $source
    require   => Ssh_authorized_key["${user}-${name}-gitsync"],
  }

  file { "${deploy}/.git-sync-stamp":
    ensure  => present,
    owner   => $user,
    mode    => 0644,
    require => Vcsrepo[$deploy],
  }

The last thing to note is that I didn't write git-sync. I've modified it but it was mostly written by Reid Vandewielle and others. Marut <3 Enjoy

PuppetDB/Storeconfigs Cache expiry

Stardate: 90682.98
After a couple of weeks of getting frustrated with puppet's Storeconfigs/puppetdb features, I have emerged victorious. PuppetDB is the newer, better, postgressier backend for puppet Storeconfigs. PuppetDB sports some really nice features including a fancy status/metrics web dashboard:

As you can see this is some interesting and potentially beneficial feedback. It is updated live and is mobile browser compatible. Personally, I'm happy to get graphs of this data any way I can, but I would prefer not to be locked into their dashboard. I would rather be able to get these data out of an often updated file or udp port so that I could send it to graphite for real time graphing and correlation with other metrics. I also don't see the point of having it be mobile friendly, since most everyone will have their puppetmaster/puppetdb server firewalled heavily and mobile devices have no business on the internal network. Some of the metrics can lead to actually tuning and performance boosts: mostly this is in the increasing the number of threads and the max jvm heap size.
The punchline here is that with

storeconfigs = true

in puppet.conf you can do exported/collected resource magics. When doing this with nagios resources I've been able to export and collect resources flawlessly. The problems came up when I tried to modify a resource. Since we make heavy use of dynamic git environments with puppet I was running something like

 puppet agent --test --environment=nagios 

on a host at random and

 puppet agent --test --environment=nagios 

on the nagios server, hoping to collect exported resources. The problem was they were not changing. As it turns out puppetdb can cache old exported resources for up to an hour. My advice for others having problem getting nagios or other exported resources to change or purge is to give it time. Run a big ssh for loop or use mcollective to hit all your boxes and hit the coffee cart for a quick pick me up. Chances are good you just need to give it time.