OpenStack Swift on HP Cloud

OpenStack Swift is the Object Storage component of OpenStack. It is roughly analogous to Amazon S3. HP Cloud (full disclaimer: I work at hp and get cloud resources for free) has an Object Storage component. This post will be about getting basic functionality out of it.

A very long document on HP's object storage can be found here. Reading as much of it as I have has permanently damaged my soul, so I am posting here to share my story, hopefully you won't have to spend so much time kicking it.

Usually when doing things on HP Cloud's OpenStack services I just poke around the command line utility's until I am happy. Let's look at basic tooling with the python-swiftclient tool:

Check deps:

$: pip install -U python-swiftclient
Requirement already up-to-date: python-swiftclient in /disk/blob/nibz/corepip/lib/python2.7/site-packages
Requirement already up-to-date: six>=1.5.2 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: futures>=2.1.3 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: requests>=1.1 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: simplejson>=2.0.9 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)  

Check creds:

$: [ -z $OS_PASSWORD ] && echo set password
$: [ -z $OS_TENANT_NAME ] && echo set tenant name
$: echo $OS_AUTH_URL
https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0/

With that set, we can use the swift command line client to upload, list, download, and delete files. All swift objects are put in containers, the equivalent of s3 buckets.

$: swift list craigslist
x220t.jpg
$: swift upload craigslist r61e.jpg
r61e.jpg
$: swift download craigslist x200t.jpg
Object 'craigslist/x200t.jpg' not found
$: swift list craigslist
r61e.jpg
x220t.jpg
$: time swift list craigslist
r61e.jpg
x220t.jpg

real    0m1.913s
user    0m0.283s
sys    0m0.051s
$: swift download craigslist x220t.jpg
x220t.jpg [auth 6.971s, headers 13.029s, total 13.314s, 0.002 MB/s]
$: swift delete craigslist r61e.jpg
r61e.jpg
$: time swift list craigslist
x220t.jpg

real    0m2.330s
user    0m0.299s
sys    0m0.059s


As you can see from the timing information, some of the operations are fast and some are slow. Swift download provides it's own timing information, which is nice. All upload/download operations above are authenticated through keystone. To provide a 'fake cdn' service like amazon s3, swift uses tempurls. This is how tempurls are typically used:

$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg
Usage: swift tempurl <method> <seconds> <path> <key>
Generates a temporary URL for a Swift object.

Positions arguments:
  [method]              An HTTP method to allow for this temporary URL.
                        Usually 'GET' or 'PUT'.
  [seconds]             The amount of time in seconds the temporary URL will
                        be valid for.
  [path]                The full path to the Swift object. Example:
                        /v1/AUTH_account/c/o.
  [key]                 The secret temporary URL key set on the Swift cluster.
                        To set a key, run 'swift post -m
                        "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"'
$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg supersecret
/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300

Then, given this information you prepend the root url of the swift service and curl at it (don't forget single quotes!):

$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg supersecret
/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300
$: keystone catalog | grep -i object
Service: object-store
|  publicURL  | https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504 |
| versionInfo |        https://region-b.geo-1.objects.hpcloudsvc.com/v1/        |
| versionList |          https://region-b.geo-1.objects.hpcloudsvc.com          |
$: curl 'https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300'
401 Unauthorized: Temp URL invalid


After poking this for some time I found this tibit buried in the HP Cloud docs on swift, er excuse me, Object Storage:

2.7.5.2 Differences Between Object Storage and OpenStack Swift TempURL Signature Generation

There are two differences between Object Storage and OpenStack Swift TempURL signature generation:

• OpenStack Swift Temporary URLs (TempURL) required the X-Account-Meta-Temp-URL-Key header be set on the Swift account. In Object Storage you do not need to do this. Instead we use Access Keys to provide similar functionality.
• Object Storage Temporary URLs require the user's Project ID and Access Key ID to be prepended to the signature. OpenStack Swift does not.

So this basically means you can't use the python-swiftclient tool with hpcloud object storage. At least nearby they provide a code snipit. I've created my own script.


$: python hpcloud_swift.py GET 3600 /v1/10724706841504/craigslist/x220t.jpg $OS_ACCESS_KEY_SECRET https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=10724706841504:KAPYAYFHCH51B3ZCCB4W:044cefcdf58d4bccea7da3a4836145488a7c8496&temp_url_expires=1423453940 


This depends on setting OS_ACCESS_KEY_ID and OS_ACCESS_KEY_SECRET in your environment. These are nonstandard environment variables. Once this is set, anyone will be able to prepend the object storage root url to the front of the url output by hpcloud_swift.py and make their own janky-cdn.

The script to perform the hpcloud fakery can be found here.

Irc Bots in Twisted with Invite-only Channels

Stardate: 90672.08

I'm kind of obsessed with writing irc bots in python using twisted.words.protocols. A longer example of how to do that may come later but for now I want to show you one of the best ways to debug your twisted irc bot and a vector to get really cool behavior not intended by the twisted developers. On an IRC server I frequent channels that are secured by forcing users to first login with NickServ then to ask ChanServ for an invite. The problem is you must join the channel after your receive the invitation from ChanServ. My solution to this problem is below, using irc_UNKNOWN and the ircLogBot.py example in the twisted.words documentation:

class BeerBot(irc.IRCClient):
    """A logging IRC bot."""
    """That also does beer things."""

...


    def signedOn(self):
        """Called when bot has succesfully signed on to server."""
        self.logger.log('identifying to nickserv')
        self.msg("NickServ", "identify %s" % config.password)
        self.logger.log('requesting channel invite')
        self.msg("ChanServ", "invite %s" % config.channel)
        self.join(self.factory.channel)
        self.logger.log('channel joinied')

...


    def irc_unknown(self, prefix, command, params):
        self.logger.log("{0}, {1}, {2}".format(prefix, command, params))
        if command == "INVITE":
          self.join(params[1])

The irc_unknown is great because it simulates the 'window 1' on most irc clients(well most command line irc clients[and by that I mean weechat and irssi{and by that I mean irssi-for-life!}]). You can add if statements to grab the other 'named' irc control messages. The others are numbered and you can split those out as well. One of the bad things about irc is different irc servers behave differently. It must be a frustrating and thankless task for the maintainers of irssi/weechat/t.p.w to provide such a universal interface to all irc servers. (lol jk, irssi hasn't had an update since like 2010.) [but no really, thank you irssi devs *internet hug*].

The source code for beerbot can be found at my github.