Simple Openssl recepie

The Openssl command has always been very opaque to me. Whenever I am doing cert operations I feel like I am a monk in the middle ages, copying scrolls I cannot read. Last night, I learned a simple command to inspect x509 certificate files that is short enough to commit to memory. I encourage everyone to use this command whenever they encounter .pem files and I encourage you to memorize it as well.

The command syntax is:

 openssl openssl x509 -in  -text

A full example:

nibz@host $ openssl x509 -in /etc/ssl/certs/Verisign_Class_1_Public_Primary_Certification_Authority.pem  -text
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            3f:69:1e:81:9c:f0:9a:4a:f3:73:ff:b9:48:a2:e4:dd
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=VeriSign, Inc., OU=Class 1 Public Primary Certification Authority
        Validity
            Not Before: Jan 29 00:00:00 1996 GMT
            Not After : Aug  2 23:59:59 2028 GMT
        Subject: C=US, O=VeriSign, Inc., OU=Class 1 Public Primary Certification Authority
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:e5:19:bf:6d:a3:56:61:2d:99:48:71:f6:67:de:
                    b9:8d:eb:b7:9e:86:80:0a:91:0e:fa:38:25:af:46:
                    88:82:e5:73:a8:a0:9b:24:5d:0d:1f:cc:65:6e:0c:
                    b0:d0:56:84:18:87:9a:06:9b:10:a1:73:df:b4:58:
                    39:6b:6e:c1:f6:15:d5:a8:a8:3f:aa:12:06:8d:31:
                    ac:7f:b0:34:d7:8f:34:67:88:09:cd:14:11:e2:4e:
                    45:56:69:1f:78:02:80:da:dc:47:91:29:bb:36:c9:
                    63:5c:c5:e0:d7:2d:87:7b:a1:b7:32:b0:7b:30:ba:
                    2a:2f:31:aa:ee:a3:67:da:db
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha1WithRSAEncryption
         58:15:29:39:3c:77:a3:da:5c:25:03:7c:60:fa:ee:09:99:3c:
         27:10:70:c8:0c:09:e6:b3:87:cf:0a:e2:18:96:35:62:cc:bf:
         9b:27:79:89:5f:c9:c4:09:f4:ce:b5:1d:df:2a:bd:e5:db:86:
         9c:68:25:e5:30:7c:b6:89:15:fe:67:d1:ad:e1:50:ac:3c:7c:
         62:4b:8f:ba:84:d7:12:15:1b:1f:ca:5d:0f:c1:52:94:2a:11:
         99:da:7b:cf:0c:36:13:d5:35:dc:10:19:59:ea:94:c1:00:bf:
         75:8f:d9:fa:fd:76:04:db:62:bb:90:6a:03:d9:46:35:d9:f8:
         7c:5b
-----BEGIN CERTIFICATE-----
MIICPDCCAaUCED9pHoGc8JpK83P/uUii5N0wDQYJKoZIhvcNAQEFBQAwXzELMAkG
A1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFz
cyAxIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTk2
MDEyOTAwMDAwMFoXDTI4MDgwMjIzNTk1OVowXzELMAkGA1UEBhMCVVMxFzAVBgNV
BAoTDlZlcmlTaWduLCBJbmMuMTcwNQYDVQQLEy5DbGFzcyAxIFB1YmxpYyBQcmlt
YXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQDlGb9to1ZhLZlIcfZn3rmN67eehoAKkQ76OCWvRoiC5XOooJskXQ0f
zGVuDLDQVoQYh5oGmxChc9+0WDlrbsH2FdWoqD+qEgaNMax/sDTXjzRniAnNFBHi
TkVWaR94AoDa3EeRKbs2yWNcxeDXLYd7obcysHswuiovMaruo2fa2wIDAQABMA0G
CSqGSIb3DQEBBQUAA4GBAFgVKTk8d6PaXCUDfGD67gmZPCcQcMgMCeazh88K4hiW
NWLMv5sneYlfycQJ9M61Hd8qveXbhpxoJeUwfLaJFf5n0a3hUKw8fGJLj7qE1xIV
Gx/KXQ/BUpQqEZnae88MNhPVNdwQGVnqlMEAv3WP2fr9dgTbYruQagPZRjXZ+Hxb
-----END CERTIFICATE-----

Cisco IOS 15 Public-key authentication

With the release of IOS 15 from Cisco, we can now use ssh public keys to authenticate to Cisco devices. It's the technology of the late nineties, today!

First create yourself a rather small key:

ssh keygen -t rsa -b 1024

It will ask you some questions, hopefully you've seen this dialog before. If you need help please feel free to comment or privately message me.

After the key has been created, copy the public string into your copybuffer.

> cat .ssh/nibz_cisco@shadow.cat.pdx.edu.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDMuKvC5ZVRuQw6YF5xnMZLopBVbQv5jxgHcR6BWfws3lTaqfSrKUlp3BulxA7P2snphcavf4TS+bNHFd9PKGRVpoQ8ERZtXn1+f008XUN3cxYMZXLB18ae7kfm8Sxk/bO4xWGaQAKc7jkIQY4OLIE0TsKTZGux241N6BNeLGmuLQ== nibz@shadow.cat.pdx.edu

Now add the key to cisco. This assumes the user has already been created properly. It also assumes you are running the following version of IOS:

*    1 52    WS-C3750G-48TS     15.0(2)SE             C3750-IPBASEK9-M

I have tried this on a 15.0(1) and it didn't work. Configuration commands:

fab6017a#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
fab6017a(config)#ip ssh pubkey
fab6017a(config)#ip ssh pubkey-chain 
fab6017a(conf-ssh-pubkey)#username nibz
fab6017a(conf-ssh-pubkey-user)#key-string
fab6017a(conf-ssh-pubkey
data)#$snphcavf4TS+bNHFd9PKGRVpoQ8ERZtXn1+f008XUN3cxYMZXLB18ae7kfm8Sxk/bO4xWGaQAKc7jkIQY4OLIE0TsKTZGux241N6BNeLGmuLQ== nibz@shadow.cat.pdx.edu
fab6017a(conf-ssh-pubkey-data)#exit
fab6017a(conf-ssh-pubkey-user)#end

Some notes on the above: Paste the whole public key once you get the (conf-ssh-pubkey-data) prompt. This includes the 'ssh-rsa' header and comment footer. Use the exit keyword on the (conf-ssh-pubkey-data) line, any other word will be sandwiched onto the end of the key. You can use this feature to split your key into multiple lines and input it that way. After this, cisco will hash your key and the configuration will look like:

  username nibz
   key-hash ssh-rsa 2F33A5AE2F505B42203276F9B2313138 nibz@shadow.cat.pdx.edu

This configuration can be put in other cisco configs elsewhere in your infrastructure. Happy hacking. This was performed on a Cisco3750G running IOS 15.0(2)SE

Ganeti Migration

It is common to want to do a 'Texas two-step' with your ganeti nodes. The idea being that you migrate everything off of a node, reboot it for patches, then migrate everything back, and do this around your cluster until you've patched all your hypervisors. Easier said than done!

The fast way:

# move primaries off node
gnt-node migrate $node

# move secondaries off node
gnt-node evacuate --secondary-only $node

Sometimes nodes fail to migrate. I don't know why. I just use the gnt-instance failover command to move them which requires a reboot. Sometimes the secondary disks don't move either. This is more annoying because hail crashes before making a plan for you. I have written the following script to, stupidly, move all the secondaries off of one node onto other nodes so you can reboot a node in your cluster without fear.

#!/bin/bash

evac_node=$1
if [ -z $evac_node ]; then
echo "Please specify a node to evac"
exit 1
fi

echo "Evacuating secondaries from $evac_node"

HYPERVISORS=`wget --no-check-certificate -O-  https://localhost:5080/2/nodes 2>/dev/null | grep id | awk '{print $NF}' | tr -d \",`

for instance in `gnt-instance list -o name,snodes  | grep $evac_node | cut -d " " -f 1`
do     
        current_primary=`gnt-instance list -o name,pnode | grep $instance | awk '{print $NF}'`
        target_node=`echo $HYPERVISORS | sed 's/ /\n/g' | grep -v $evac_node | grep -v $current_primary | sort -R | head -n 1`
        echo "gnt-instance replace-disks -n $target_node $instance"
        gnt-instance replace-disks -n $target_node $instance
done

Happy hacking!

Ops School: The coolest thing since sliced bread

Stardate 90915.3

I work at an organization that teaches people new to computers how to administer them. I've give a talk with blkperl, my boss, many times called Zero to Root. Teaching in our organization is very personal and doesn't really translate to other environments. We can't say 'Here, use our curriculum, its baws.'

Ops School strives to be the curriculum for people who want to become operations people. It doesn't assume any previous knowledge, but doesn't hide the details from the users either. Its kindof like a tldp.org for this generation of hackers. I have been slowing in my blog postings because some of that energy I use to write down things for other people has been siphoned off into opsschool.

I encourage everyone out there to contribute, there are huge chunks of this document that still need to be written.

Solaris redundant nfs mounts

Stardate 90840.8 Nfs servers with lots of disks, sharing files out to multiple clients, is a pattern followed everywhere. Eventually your users begin to not only use these remote files, but start to put nfs directories into their PATH variable. This causes problems whenever you need to patch or reboot the nfs server because all shells launched by users with PATHs that look into the nfs directories will hang forever(this does assume you are mounting hard).

You can minimize the effect of this by using redundant mount information.

Before:

/usr/local -ro,hard,intr,suid  bunny.cat.pdx.edu:/disk/forest/local

After:

/usr/local -ro,hard,intr,suid  bunny.cat.pdx.edu:/disk/forest/local,caerbannog.cat.pdx.edu:/volumes/cave/misc/usr-local

It is best to maintain an idea of a primary and secondary, at least for administration. Modify only the primary, and rsync to the secondary. Use readonly mounting to mount. This mount appears in ``mount`` like this:

/usr/local on caerbannog.cat.pdx.edu:/volumes/cave/misc/usr-local,bunny.cat.pdx.edu:/disk/forest/local remote/read only/setuid/devices/rstchown/hard/intr/xattr/dev=5a0000e on Wed Mar 27 15:35:21 2013

Note that this is mounted on both servers. Packets get sent to both servers and the first to respond with valid information is reported to the system. This can make for some bizarre weirdness if you use readwrite mounts.

It is totally possible to use something like drbd between nfs servers (not on Solaris, obviously), to make this doable with readwrite mounts. I have not done this personally.

Cascadia IT Conf

Stardate 90814.44

This weekend we attended CasItConf13. I had a blast at met a lot of really cool people. I attended presentations on Logstash, IPv6, Chef and more. Jordan Sissel, in particular, did a great job of presenting Logstash. After his talk we met up and had a neat conversation. He showed me an app he had created called fingerpoken. Its a bit out of date and we had to do some hacks but I was able to get it up and running in a half-hour lunch break and still have time to demolish some tasty lunch provided by the wonderful folks over at Puppet Labs. Fingerpoken is an app that lets you send mouse and keyboard events to a computer with a smartphone.

And thats really what its all about. Is the tool simple and easy enough that you can get it going in a crunch? Are all the nonintuitive parts ripped out and replaced with sane defaults and the tool just 'goes'? In fingerpoken's case not really. We had to do some:

sudo ln -s /usr/lib/libxdo.so.2 /usr/lib/libxdo.so

But, what is the point of having the author of your tool nearby if not to tell you to do that? And yes, the abi is evidently close enough to just work in that case.

I am very impressed that I was able to get such high-level functionality out of a tool in a short period of time and under pressure. If your tool passes the 'setup at lunch at a conference' test, you're doing pretty dang good. If it doesn't, look for places to streamline it. I'm happy to test your random tool, please let me know.

My talk, on the Computer Action Team/Braindump, is available on my github and you can download the pdf from here.

In other news, it seems that github no longer allows you to download the raw files out of repositories if they are above a certain size. Possibly more on that later.

Debain packaging

Stardate: 90784.6

Git-sync is a script in ruby we use at work for managing git repos. It is covered in an earlier post. I got tired of ensuring it as a file in puppet and decided to make a debian package. Here is the summary of how to make a simple debian package containing just a single file. Note that the answer to this stack overflow question is the source of most of my knowledge, so this will just be annotations and extensions to that.

Debian/Ubuntu packaging (on an ubuntu system) required me to install a single package: devscripts.

At a high level, debian packaging involves creating a 'debian' folder in your source tree and putting several metadatafiles in it. Figuring out the precise contents of these files is the challenge of packaging. I recommend you use the 'apt-get source git' command to get the source of a working package (git in this case) to compare to your own metadatafiles.

Debian/Ubuntu packaging using debuild creates files one level above your current working directory(wtf debian). So the first step is to build a build directory like:

cd ~/devel
mkdir git-sync-build

Procure the source:

nibz@darktemplar:~/devel/git-sync-build$ git clone git@github.com:pdxcat/git-sync
nibz@darktemplar:~/devel/git-sync-build$ ls
git-sync
nibz@darktemplar:~/devel/git-sync-build$ cd git-sync
nibz@darktemplar:~/devel/git-sync-build/git-sync$ mkdir debian

All of the metadata files that debuild, the utility that will actually build the .deb, needs are going to be in the debian directory. 

The first file to create is the debian/changelog file. This file is created with the dch utility. Run it from the git-sync directory. It will open vim and it will look like this. Many fields here need to be changed.
dch --create

PACKAGE (VERSION) UNRELEASED; urgency=low

  * Initial release. (Closes: #XXXXXX)

 -- Spencer Krum   Thu, 07 Mar 2013 01:40:18 -0800


PACKAGE refers to the name of the package. Replace the word PACKAGE with the name you want your package to register itself as. In my git-sync case I will use 'git-sync'. The package name must be lower case. The VERSION must be replaced with a version number. I'm using 1.0.1 for this, since it is the second release of git-sync, but the changes are very minor. There are long articles on the internet about version numbering. It's not my place to comment here. The RELEASE variable needs to be replaced with a debian or ubuntu codename such as 'precise' or 'wheezy'. I have no idea what urgency is, but setting it to low doesn't seem to hurt anything. Maybe this is how you tell apt/dpkg about security updates. The initial release stuff is fine. The name is a bit tricky. Later on we will gpg sign the package. Make sure the name and email in the changelog match exactly the name and email on your gpg key, else the debuild utility won't attempt to have you gpg sign it at all. My changelog looks like this:
git-sync (1.0.0) precise; urgency=low

  * Initial release. 

 -- Spencer Krum   Wed, 06 Mar 2013 16:46:14 -0800


Next create a debian/copyright file:
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: myScript
Upstream-Contact: Name, 

Files: *
Copyright: 2011, Name, 
License: (GPL-2+ | LGPL-2 | GPL-3 | whatever)
 Full text of licence.
 .
 Unless there is a it can be found in /usr/share/common-licenses


I elected for the apache2 license and to use the two paragraph version of that license. I also gave credit where it was due here. Fill out this file as you see fit.

Next create a debain/compat file: 
nibz@darktemplar:~/devel/git-sync-build/git-sync/debian$ echo 7 > compat

Next create the rules file. This file seems to be the work-doer in debian packaging. It is evaluated by make, which is picky, so make sure that indented line is a real tab(copying from my blog will probably fail). The --with python is... well I have no idea. I traced it to a python.pem (pem is a perlism) deep within /usr. Since I am packaging a ruby script I just removed it.

Example from stackoverflow
#!/usr/bin/make -f

%:
    dh $@ --with python2

git-sync version
#!/usr/bin/make -f

%:
	dh $@

Next make the control file. Make the natural substitutions here. I guessed on section and it just sorta worked.
nibz@darktemplar:~/devel/git-sync-build/git-sync/debian$ cat control 
Source: git-sync
Section: ruby
Priority: optional
Maintainer: Spencer Krum, 
Build-Depends: debhelper (>= 7),
               ruby (>= 1.8.7)
Standards-Version: 3.9.2
X-Ruby-Version: >= 1.8.7


Package: git-sync
Architecture: all
Section: ruby
Depends: ruby, ${misc:Depends}, ${python:Depends}
Description: Git syncing script, pull based
  Git-sync allows git repositories to be kept in sync via git
  hooks or other means. Pull based, able to handle force pushes
  and submodules

Next make the install file. I went with the default in the stackoverflow post. I attempted to make some simple modifications to it(moving the file to /usr/local/bin) and that made it fail so this file is evidently pretty finicky.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ cat debian/install 
git-sync usr/bin


Now you can build the debian package.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ debuild --no-tgz-check

If all went well, it should ask you to decrypt your gpg key twice and build a package in the directory one level up.
nibz@darktemplar:~/devel/git-sync-build/git-sync$ ls ..
git-sync                git-sync_1.0.0_amd64.build    git-sync_1.0.0.dsc
git-sync_1.0.0_all.deb  git-sync_1.0.0_amd64.changes  git-sync_1.0.0.tar.gz

You now have a shiny .deb file that can be installed with dpkg -i git-sync_1.0.0_all.deb

It is easy to put this in a launchpad PPA if you have a launchpad account. From your launchpad homepage (a shortcut is https://launchpad.net/~ if you are signed in). Press the "Create new PPA". Fill out the form. 

Next build a source package. Launchpad PPAs take source packages and build binary packages on launchpad servers. Build it with:
nibz@darktemplar:~/devel/git-sync-build/git-sync$ debuild -S

It should go through the gpg motions again and build a source file.
Then you should be able to run something like(with your launchpad username and name of PPA):
dput ppa:krum-spencer/git-sync-ppa git-sync_1.0.0_source.changes


Happy Packaging!