OSCON Talk: "Pro Puppet"

I'm giving a talk at OSCON 2014! It's called Pro Puppet and will cover the techniques I think anyone can use to get the most out of using Puppet. To celebrate and promote, we've created a word cloud of all the words in Pro Puppet 2'nd Edition. Please come watch me at 4:10 pm on Wednesday!

CephFS as a replacement for NFS: Part 1

This is the first in a series of posts about CephFS. The overall goal is to evaluate and characterize the behavior of CephFS and determine if it can be a reliable replacement for NFS.

The current use case of NFS is 400G-1T 'stashes' shared from an NFS server to hundreds of Linux/Unix clients in an academic setting. In some cases these stashes are accessed by a single user on a single machine, in some cases dozens of users access them across dozens of machines.

Drawbacks to the current situation are the same as any situation involving NFS:

• Security is a joke
• Single über-powerful NFS filers present a SPOF
• Bigger and bigger filers get more and more expensive
• Forced to use proprietary and expensive ZFS on Solaris
• Backing up is becoming a problem as total dataset size becomes more than a tape backup system can really hold
• No tiering of storage. The whole dataset either goes on the fast disks or the slow disks

There are also some advantages of this system:

• NFS is old faithful
• Every operating system supports it, and usually pretty well
• NFS ipv6's like a champ
• It's already working
• Integrates well with pam, autofs, ldap
• Vendor, while expensive, is really good at fixing it
• ZFS allows 'thin provisioning' so that we can over subscribe. 
• ZFS allows full nfsv4 acls to be used (This could also go in the drawbacks section because extended acls cause much pain)

Some key advantages we hope to achieve with ceph:

• Clustering
• Replication of data at the ceph layer instead of RAID
• Authentication
• Tiering of disks/storage
• Setting different replication levels for different storage sets

The CephFS remote filesystem has capabilities roughly analogous to NFS. There is a single 'volume', it can be simultaneously mounted by multiple clients, it respects unix groups.

In the follow up posts to this one we will build out a test ceph cluster, build filesystems on it, mount them, and generally attempt to build feature parity with an NFS system.

Hipchat + Bitlbee + Irssi

I use irssi for open source. I use hipchat for work. It's pretty obvious that they should be combined.

There are a number of posts around the internet covering how to do this but I will throw my hat into the ring. Suffice to say, I'm standing on the shoulders of giants.

I use Centos 6 as my jump host. After a lot of head scratching I eventually discovered that the version of bitlbee in centos/epel is too old to work correctly. I thus had to compile my own. Yippie. Why aren't we just using FreeBSD again?

 ./configure --prefix=$HOME/local --ssl=openssl

Architecture: Linux

Configuration done:
  Debugging disabled.
  Building PIE executable
  Binary stripping enabled.
  Off-the-Record (OTR) Messaging disabled.
  systemd disabled.
  Using event handler: glib
  Using SSL library: openssl
  Building with these protocols: msn jabber oscar yahoo twitter

After  make and make install the software is installed in my home directory. Bitlbee installs to $PREFIX/sbin so be sure that is in your $PATH.

We can now fire up bitlbee:

[nibz@destiny]~% mkdir ~/.config/bitlbee
[nibz@destiny]~% bitlbee  -D -i 127.0.0.1 -p 2121 -d /home/nibz/.config/bitlbee
Warning: Unable to read configuration file `/home/nibz/local/etc/bitlbee/bitlbee.conf'.

Despite this troubling error bitlbee is running correctly:

[nibz@destiny]~% ps -ef | grep bitlbee                                        
nibz     31276     1  0 00:07 ?        00:00:00 bitlbee -D -i 127.0.0.1 -p 2121 -d /home/nibz/.config/bitlbee
nibz     31386  9444  0 00:07 pts/20   00:00:00 grep bitlbee

With this up we have bitlbee listening on localhost port 2121. Connect to this with your favorite irc client, I use irssi.
  

/server add -auto -network hipchat 127.0.0.1 2121
/connect 127.0.0.1

Now you must join the bitlbee control channel and proceed to do the quickstart tutorial stuff. Choose a very secure password because anyone able to connect to the bitlbee socket can attempt to log into your account. As far as I know there is no rate limiting.

/join bitlbee
help quickstart
help quickstart2

Next we follow the instructions here, here, here and here. Don't worry. I have my set below. You'll need more than just your hipchat credentials. You need to get your hipchat metadata from here. I'm not sure what the security status of that information is, but I've gone ahead and anonamized mine.


On the jabber xmpp settings page of your hipchat account, there is a 'username', a 'jabber id', and a 'conference (muc) domain'.

My username is of the format 8489_938318@chat.hipchat.com.

The first before the underscore, 8489, is your organization id. The second number is essentially your number.

Users on your domain will look like <Bob Dole> 8489_38239@chat.hipchat.com.

Channels/Rooms will look like #ScienceLab 8489_ScienceLab@conf.hipchat.com.

Notice that the channels are suffixed by the 'conf.hipchat.com' and the users are suffixed by 'chat.hipchat.com'. I believe this corresponds to the 'conference (mux) domain.'

Anyhoo this is how you connect:

account add jabber 8489_384193@chat.hipchat.com
account jabber set tag hipchat
account hipchat set nick_source full_name

account hipchat set resource bot

Now you have to feed it your password. Use the /oper command on your irc client, as if you were assuming irc network operator privileges. This will prompt you for your password, and your password will not be echoed to you. I do not know to what extent the password is encrypted on disk.

Now connect to the network:

account hipchat on

You'll see a long list of 'people' join. I think this is everyone in your domain. I think this is also the place where you private message people. As in, any messages coming to you in this domain channel are private messages from the person sending it and any messages you send are messages you send privately.




Now join your channel and set your nick to present as your real name to appease the gods.

chat add hipchat 8489_ScienceLab@conf.hipchat.com #ScienceLab
channel #ScienceLab set nick 'James Kirk'
/join #ScienceLab

And just like that. Boom. Hipchat through Irssi.


 

Giving a great talk

These are a few of my thoughts on how to make any talk you're giving great.

• Practice! (something like 9/10 people don't do their presentation ONCE before giving it)
• Practice doesn't just mean run through the whole thing. It means trying little paragraph-sized parts over and over again until you get the timings right. This can be done in the shower, while running, on the commute, whenever! These short, punchy parts of your talk will have the biggest impact on your audience. 
• The title of your talk should be short and to the point. "Intermediate Graphite" is good. "Automating metrics with graphite and collectd: How I learned to stop worrying and love the data" is bad. For one, every talk these days seems to be titled off of a movie. For two, your talk title will be printed on a tightly-jammed conference schedule, compressed into 30 pixels on a mobile app, and the very top of your title slide, making it longer makes it harder to tell what's going on. For three, shorter phrases just pack more punch. You don't have to tell everyone what will be covered in your talk in the title, you can let the title be mysterious. This will both bring in more audience members who want to see what is going on, and increase your chances of being accepted to talk because conference organizers will be curious as well. 
• Carry in your own water, a bottle, a pitcher, several cups. Whatever. Just make sure you are well prepped for hydration. Rule of thumb is 1 8 oz cup per half hour of talking. This water should be room temperature. Don't drink cold water before or during your talk. Cold water freezes the vocal chords. 
• Take water sips in between paragraphs or sections. Stopping mid point or sentence to drink pulls your audience out of the narrative. 
• Print your speaker notes. I'm serious. Kill a tree. You do not want to be at the mercy of your laptop, conference wifi, or power problems. 
• Most projector systems are powered of VGA. Make sure your laptop has VGA out or you have the appropriate dongle. Don't loan this dongle to anyone. Guard it with your life. 
• Give your laptop a full charge and resist using it until showtime for your talk. 
• Have a local copy of your talk on your laptop, if using google-docs, download a pdf and present from that. 
• Email yourself a copy of the talk. If your laptop totally sparks up on the day of, you can use your phone to quickly forward the talk to a buddy who can set it up on their laptop. 
• Use the restroom before your talk. Wash your hands, splash some water on your face. Get pumped. Men: sit down for this bathroom break, this is no time to tempt fate. 
• Bring a friend to the talk. If you're alone at a conference, make a friend and have them come to the talk. Install this person in the front row nearest to you. 
• If this person is paying rapt attention, the people behind them will instinctively follow suit. The same is true if that person is playing with their phone. 
• This person can ask a question at the end of the talk, even if no one else does. 
• If you have projector trouble, after 3 minutes of working on it, you can begin your talk from your printed notes and your buddy can fight with the projector. 
• Think about the best teacher you had in high school or college. What did they do in their lectures to engage you? 
• Slides suck. But you have to use slides. Most audience members expect slides and if you literally don't have any, the audience will be confused, and it may become unclear when the talk is starting and stopping. 
• I am a big fan of split talks. Start the presentation with 5-10 slides. These slides introduce you and your content, define terms, provide links and generally set up the talk. Then you can go to the whiteboard/chalkboard or draw on your tablet, or step out from behind the podium and give a more personal, engaging talk. 
• Start the slide deck with a graphic or gif slide that fills the whole screen, usually this is a funny picture or a gif of a fireplace. This allows everyone to relax, including you. It is obvious that the talk hasn't officially started yet. It provides a good color and sizing match for when you are setting up the projector. If there is trouble with the projector, the humorous image you are trying to present will take the stress off of everyone. 
• The next slide should be the title page of your talk. This should include your name, your contact information, and your affiliation (work, school, foundation, etc.) Some percent (0.5% or so) of people who will see your talk will find something wrong with it and NEED to correct you. Providing a twitter handle or an email address gives people an outlet to contact you with questions, comments, and criticism. This is vastly preferable to someone raging on the comments section of the youtube/slideshare or whatevers.
• You should expressly invite feedback in your dialogue, because we all need feedback as a speaker.
•  Depending on the context and topic of your talk, you should provide the audience with your slides. For open source conference talks, this is as easy as putting your slides up on slideshare.net.  For more private talks, emailing out a pdf of your slides is appropriate.  
• During question time, always repeat the question. This will take time and practice. In todays world, most talks are video recorded and put up online. Repeating the question is the only way for the at-home viewer to know what the question you are responding to is.  
• At question time, you don't need to be an expert at everything. Often people will ask you a question you don't know the answer to or know that some friend of yours in the audience is the person to ask. You can call on this person. This has the nice effect of making the talk feel more like a dialogue. Still, you should give this person a few seconds to prepare and swallow their food and so forth. Say something like "You know that question is really best answered by my friend <friendname>. We've known each other since we worked at <company> together, where we developed <something random>. Since then they're now working at <new company name> and has become an expert in <question>. <friendname>, do you have an answer to <repeat question>."
• Your last slide should be 'Questions?' and your name/contact information/affiliation.