Sunday, May 10, 2015

Managing patchset stacks with git-review

In OpenStack, we use gerrit and git-review to propose changes to the repository. The workflow for that is pretty complicated, and downright confusing if you are coming from the github workflow.

One of the places where it gets hard/complicated/annoying to use our workflow is if you have multiple dependent changes. I have a technique I use, that I will present below.

The situation: You have two patches in a stack. There is a bug in the first patchset that you need to fix.

The simple play: Checkout the patchset with 'git review -d <review number>', ammend and git-review. The problem with this is that now you need to go rebase all dependent patchsets against this new review. Sometimes you can get away with using the 'rebase' button but sometimes you cannot.

What I do: I use 'git rebase -i HEAD~2' and use 'edit' to change the commit that needs to be changed, rebase goes ahead and auto-rebases everything above it (pausing if needed for me to fix things), then I can 'git review' and it will update all the patchsets that need to be changed.

This approach works for any sized stack, but using it on a two-stack is the simplest example that works.



The git log before we start:

commit e394aba4321f6d30131793e69a4f14b011ce5560
Author: Spencer Krum <nibz@spencerkrum.com>
Date:   Wed May 6 15:43:27 2015 -0700

    Move afs servers to using o_p::template
    
    This is part of a multi-stage process to merge o_p::server and
    o_p::template.
    
    Change-Id: I3bd3242a26fe701741a7784ae4e10e4183be17cf

commit 3e592608b4d369576b88793377151b7bfaacd872
Author: Spencer Krum <nibz@spencerkrum.com>
Date:   Wed May 6 15:38:23 2015 -0700

    Add the ability for template to manage exim
    
    Managing exim is the one thing sever can do that template cannot.
    This is part of a multi stage process for merging server and template.
    
    Change-Id: I354da6b5d489669b6a2fb4ae4a4a64c2d363b758


Note that we have two commits and they depend on each other. The bug is in 3e592608b4d369576b88793377151b7bfaacd872. We start the interactive rebase below, first with a vim session then with output on the command line. The vim session:

$ git rebase -i HEAD~2


  1 e 3e59260 Add the ability for template to manage exim
  2 pick e394aba Move afs servers to using o_p::template
  3 
  4 # Rebase af02d02..e394aba onto af02d02
  5 #
  6 # Commands:
  7 #  p, pick = use commit
  8 #  r, reword = use commit, but edit the commit message
  9 #  e, edit = use commit, but stop for amending
 10 #  s, squash = use commit, but meld into previous commit
 11 #  f, fixup = like "squash", but discard this commit's log message
 12 #  x, exec = run command (the rest of the line) using shell
 13 #
 14 # These lines can be re-ordered; they are executed from top to bottom.
 15 #
 16 # If you remove a line here THAT COMMIT WILL BE LOST.
 17 #
 18 # However, if you remove everything, the rebase will be aborted.
 19 #
 20 # Note that empty commits are commented out


Note that the 'top' commit in the rebase view is the 'bottom' commit in the git log view, because git is stupid. We change the 'pick' to 'e' for 'edit' meaning stop at that point for ammending. And the shell output:

Stopped at 3e592608b4d369576b88793377151b7bfaacd872... Add the ability for template to manage exim

You can amend the commit now, with

  git commit --amend

Once you are satisfied with your changes, run

  git rebase --continue

 (master|REBASE-i 1/2)$: git st
rebase in progress; onto af02d02
You are currently editing a commit while rebasing branch 'master' on 'af02d02'.

  (use "git commit --amend" to amend the current commit)
  (use "git rebase --continue" once you are satisfied with your changes)



nothing to commit, working directory clean


Then we make changes to modules/openstack_project/manifests/template.pp (not shown) and continue the rebase:


 (master *|REBASE-i 1/2)$: git st
rebase in progress; onto af02d02

You are currently editing a commit while rebasing branch 'master' on 'af02d02'.

  (use "git commit --amend" to amend the current commit)
  (use "git rebase --continue" once you are satisfied with your changes)
 
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  modified:   modules/openstack_project/manifests/template.pp
no changes added to commit (use "git add" and/or "git commit -a")
 
 (master *|REBASE-i 1/2)$: git add modules/openstack_project/manifests/template.pp
 (master +|REBASE-i 1/2)$: git rebase --continue
[detached HEAD 6ca26e9] Add the ability for template to manage exim
 1 file changed, 7 insertions(+)
Successfully rebased and updated refs/heads/master.


Then we publish our changes with git-review:
(master u+2)$: git review

You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits. Otherwise you should consider squashing your changes into one
commit before submitting.

The outstanding commits are:

2bc78a8 (HEAD, master) Move afs servers to using o_p::template
6ca26e9 Add the ability for template to manage exim

Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel: yes

remote: Resolving deltas: 100% (4/4)
remote: Processing changes: updated: 2, refs: 2, done    
To ssh://krum-spencer@review.openstack.org:29418/openstack-infra/system-config.git
 * [new branch]      HEAD -> refs/publish/master

With that we have changed a commit deep in the stack, rebased any commits above it, and published our changes to the gerrit server.

Overview of Puppet in OpenStack Infra

Last week I gave this presentation at the PDX Puppet Users group. It is an overview of how we use Puppet in the OpenStack Infra project. There is no video or audio recording.

Presentation

Friday, March 6, 2015

Checking out Servo

Servo is an experimental web browser from Mozilla. It was the impetus and driver for early development of the Rust language. I'm excited to ditch firefox because of its performance issues and I don't want to run google anything these days. Blogging on blogger, I know.

I got it built from the instructions on the github, here are some screencaps of what it can do.




Sunday, February 15, 2015

EFI boot on HP EliteBook 840

After entirely too long with my HP EliteBook 840, I have made it boot successful without human interaction. After installing ubuntu my typical power-on process looked like this:

  •  Power button
  •  Computer tries and fails to boot, dumping to diagnostics
  •  Power button
  •  Interupt boot process at the right time with f-9
  •  Select 'boot from efi file'
  •  Select a disk
  •  Drill into the filesystem and select 'shimx64.efi'
This was super annoying. I finally got fed up and went exploring settings. There is a section in settings for setting a custom efi path. The interface is a bit derpy, but its eventually possible to get to a text input box.

I put in my text box:  
\EFI\ubuntu\shimx64.exe

At this point, I saved and rebooted. The machine was able to come up into ubuntu with no human intervention. My next work is to enable the 'Custom Logo at boot' component.

Wednesday, February 11, 2015

Rocket: First steps

Rocket is a container runtime for CoreOS. In this post we will do some basic tasks with Rocket: installing it, creating an ACI, publishing that ACI to OpenStack Swift, and pulling it down.

wget https://github.com/coreos/rocket/releases/download/v0.3.1/rocket-v0.3.1.tar.gz
tar xzvf rocket-v0.3.1.tar.gz
cd rocket-v0.3.1
./rkt help
 
I moved 'rkt' and 'stage1.aci' to ~/bin for ease of use.

We also need actool:

derp@myrkt:~$ git clone https://github.com/appc/spec.git
Cloning into 'spec'...
remote: Counting objects: 1604, done.
remote: Compressing objects: 100% (20/20), done.
Receiving objects: 100% (1604/1604), 614.19 KiB | 0 bytes/s, done.
remote: Total 1604 (delta 7), reused 1 (delta 0)
Resolving deltas: 100% (924/924), done.
Checking connectivity... done.
derp@myrkt:~$ cd spec/
derp@myrkt:~/spec$ ls
ace  aci  actool  build  CONTRIBUTING.md  DCO  discovery  examples  Godeps  LICENSE  pkg  README.md  schema  SPEC.md  test  VERSION
derp@myrkt:~/spec$ ./build 
Building actool...
go linux/amd64 not bootstrapped, not building ACE validator
derp@myrkt:~/spec$ ls bin/actool 
bin/actool
derp@myrkt:~/spec$ ./bin/actool -h
Usage of actool:
  -debug=false: Print verbose (debug) output
  -help=false: Print usage information and exit 
 
 
Now we need a simple go application:
 
package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        log.Printf("OHAIDER: request from %v\n", r.RemoteAddr)
        w.Write([]byte("hello\n"))
    })
    log.Fatal(http.ListenAndServe(":5000", nil))
} 
 
And a manifest.json file:
 
 {
    "acKind": "ImageManifest",
    "acVersion": "0.2.0",
    "name": "nializer/daemon",
    "labels": [
        {
            "name": "version",
            "value": "1.0.0"
        },
        {
            "name": "arch",
            "value": "amd64"
        },
        {
            "name": "os",
            "value": "linux"
        }
    ],
    "app": {
        "user": "root",
        "group": "root",
        "exec": [
            "/bin/daemon"
        ],
        "ports": [
        {
            "name": "www",
            "protocol": "tcp",
            "port": 5000
        }
        ]
    },
    "annotations": [
        {
        "name": "authors",
        "value": "Kelsey Hightower , Spencer Krum "
        },
        {
            "name": "created",
            "value": "2014-10-27T19:32:27.67021798Z"
        }
    ]
}
 
 
And a file structure:
 
root@myrkt:~/app# find daemon-layout/
daemon-layout/
daemon-layout/rootfs
daemon-layout/rootfs/bin
daemon-layout/rootfs/bin/daemon
daemon-layout/manifest
 
 
Then we can build(and verify) this image:
 
root@myrkt:~/app# find daemon-layout/
daemon-layout/
daemon-layout/rootfs
daemon-layout/rootfs/bin
daemon-layout/rootfs/bin/daemon
daemon-layout/manifest
root@myrkt:~/app# actool build daemon-layout/ daemon-static.aci
root@myrkt:~/app# actool --debug validate daemon-static.aci 
daemon-static.aci: valid app container image 
 
 
Then we can run the image (this doesn't work):
 
root@myrkt:~/app# rkt run daemon-static.aci
/etc/localtime is not a symlink, not updating container timezone.
Error: Unable to open "/lib64/ld-linux-x86-64.so.2": No such file or directory
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Unmounting /proc/sys/kernel/random/boot_id.
All filesystems unmounted.
Halting system. 
 
 
The issue is that our go binary is not statically compiled:
 
root@myrkt:~/app# ldd daemon
        linux-vdso.so.1 =>  (0x00007fff2d2cd000)
        libgo.so.5 => /usr/lib/x86_64-linux-gnu/libgo.so.5 (0x00007ff809e45000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff809c2f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff809868000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff80aca3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff80964a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff809344000) 
 
 
But this is okay because we can just add these files (plus a few more) to our ACI:
 
root@myrkt:~/app# find daemon-layout/
daemon-layout/
daemon-layout/rootfs
daemon-layout/rootfs/lib64
daemon-layout/rootfs/lib64/libc.so.6
daemon-layout/rootfs/lib64/ld-linux-x86-64.so.2
daemon-layout/rootfs/bin
daemon-layout/rootfs/bin/daemon
daemon-layout/rootfs/usr
daemon-layout/rootfs/usr/lib
daemon-layout/rootfs/usr/lib/x86_64-linux-gnu
daemon-layout/rootfs/usr/lib/x86_64-linux-gnu/libgo.so.5
daemon-layout/rootfs/lib
daemon-layout/rootfs/lib/x86_64-linux-gnu
daemon-layout/rootfs/lib/x86_64-linux-gnu/libpthread.so.0
daemon-layout/rootfs/lib/x86_64-linux-gnu/libc.so.6
daemon-layout/rootfs/lib/x86_64-linux-gnu/libgcc_s.so.1
daemon-layout/rootfs/lib/x86_64-linux-gnu/libm.so.6
daemon-layout/manifest 
 
 
Now we can re-build, verify, run:
 
 
root@myrkt:~/app# actool build --overwrite  daemon-layout/ daemon.aci                                                                                          
root@myrkt:~/app# actool --debug validate daemon.aci                                                                                                           
daemon.aci: valid app container image
root@myrkt:~/app# du -sh daemon.aci
5.6M    daemon.aci
root@myrkt:~/app# rkt run daemon.aci
/etc/localtime is not a symlink, not updating container timezone.
2015/02/11 13:32:58 OHAIDER: request from 127.0.0.1:55024

This means everything is working. You can exit by pressing ^] three times.

We can then post the daemon file to swift, using the tempurl system from a previous post. Then using a tiny url service, we can run the aci from the network:


root@myrkt:~/app# rkt fetch http://l.pdx.cat/nibz_daemon.aci
rkt: fetching image from http://l.pdx.cat/nibz_daemon.aci
Downloading aci: [============================================ ] 5.81 MB/5.84 MB
Downloading signature from http://l.pdx.cat/nibz_daemon.sig
EOF
root@myrkt:~/app# rkt run nibz_daemon
rkt only supports http or https URLs (nibz_daemon)
root@myrkt:~/app# rkt run http://l.pdx.cat/nibz_daemon.aci
rkt: fetching image from http://l.pdx.cat/nibz_daemon.aci
Downloading aci: [=====================================        ] 4.89 MB/5.84 MB
Downloading signature from http://l.pdx.cat/nibz_daemon.sig
EOF 
 
 
Okay, so that didn't work. Maybe later on I will figure that part out. I am particularly excited to use the swfit meta tags to add the security meta tags used by rocket for collecting signatures.
 
 
 

Rocket: Containers without Docker

Rocket is a container runtime from CoreOS. It is a response to Docker's feature creep. Simultaneously with Rocket, the CoreOS team released the App Container Spec, a specification of the image format consumed by a container runtime. Multiple container runtimes could then be written and could all consume the same images. In this post I will talk about my experience with it and what I like and don't like so far. Note that I don't have a ton of experience with this tool at this point.

There are a couple of things inside the app container spec/rocket ecosystem that are just fantastic(actually I'm pumped about basically the whole thing):

Security is a first class concern

Rocket uses gpg to verify the authenticity of App Container Images(aci). It does this by allowing the administrator to trust keys, then containers signed by those keys are trusted. Rocket maintains its own keyring with trust levels. This borrows from the techniques used to secure Linux packaging. Rocket/ACI also use sha512sums to uniquely identify ACIs.

Built on core unix utilities

The main operations (actually all operations) involved in creating, signing, verifying, and sharing ACI's are composed out of the standard unix utilites: tar, gpg, gzip. Any helper utilities just serialize these functions. actool is one such utility. This keeps ACI's simple, doesn't tie anyone to custom tooling, increases debugability and hackability. Particularly in the signing and verification components, this means no one has to trust CoreOS or anyone else.

Emphasis on pushing exactly what you want into the container

With ACI, you copy the files you want into the container and stop there. This encourages small images, and encourages administrators to know exactly what is going in their images.

Networking just works

No crazy -p port:port:ipaddress nonsense, you specify the listen port in the configuration file, boom done. Listens on all interfaces.

Configuration file makes sense, extendable

When you build an ACI, you bake a manifest.json into it. This is a configuration file with a combination of runtime settings and overall metadata. I am already comparing and contrasting with Puppet's metadata.json. Both of these files contain basic metadata such as authorship information. And both are young formats with tooling and use still growing up. JSON's schemalessness allows users and devs to rapidly prototype and try out new information and structures in these files.

HTTP

Http is used as the primary transport. ACI's are simple files. These ACI's can be pushed into webservers or s3 and pulled out with wget or the rocket utility itself.
This is a massive improvement over the current docker-hub situation. Rocket has some rudimentary support for inferring github http locations from names such as 'coreos.com/etcd'

Sunday, February 8, 2015

OpenStack Swift on HP Cloud

OpenStack Swift is the Object Storage component of OpenStack. It is roughly analogous to Amazon S3. HP Cloud (full disclaimer: I work at hp and get cloud resources for free) has an Object Storage component. This post will be about getting basic functionality out of it.

A very long document on HP's object storage can be found here. Reading as much of it as I have has permanently damaged my soul, so I am posting here to share my story, hopefully you won't have to spend so much time kicking it.

Usually when doing things on HP Cloud's OpenStack services I just poke around the command line utility's until I am happy. Let's look at basic tooling with the python-swiftclient tool:

Check deps:

$: pip install -U python-swiftclient
Requirement already up-to-date: python-swiftclient in /disk/blob/nibz/corepip/lib/python2.7/site-packages
Requirement already up-to-date: six>=1.5.2 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: futures>=2.1.3 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: requests>=1.1 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)
Requirement already up-to-date: simplejson>=2.0.9 in /disk/blob/nibz/corepip/lib/python2.7/site-packages (from python-swiftclient)  

Check creds:

$: [ -z $OS_PASSWORD ] && echo set password
$: [ -z $OS_TENANT_NAME ] && echo set tenant name
$: echo $OS_AUTH_URL
https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0/

With that set, we can use the swift command line client to upload, list, download, and delete files. All swift objects are put in containers, the equivalent of s3 buckets.

$: swift list craigslist
x220t.jpg
$: swift upload craigslist r61e.jpg
r61e.jpg
$: swift download craigslist x200t.jpg
Object 'craigslist/x200t.jpg' not found
$: swift list craigslist
r61e.jpg
x220t.jpg
$: time swift list craigslist
r61e.jpg
x220t.jpg

real    0m1.913s
user    0m0.283s
sys    0m0.051s
$: swift download craigslist x220t.jpg
x220t.jpg [auth 6.971s, headers 13.029s, total 13.314s, 0.002 MB/s]
$: swift delete craigslist r61e.jpg
r61e.jpg
$: time swift list craigslist
x220t.jpg

real    0m2.330s
user    0m0.299s
sys    0m0.059s


As you can see from the timing information, some of the operations are fast and some are slow. Swift download provides it's own timing information, which is nice. All upload/download operations above are authenticated through keystone. To provide a 'fake cdn' service like amazon s3, swift uses tempurls. This is how tempurls are typically used:

$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg
Usage: swift tempurl <method> <seconds> <path> <key>
Generates a temporary URL for a Swift object.

Positions arguments:
  [method]              An HTTP method to allow for this temporary URL.
                        Usually 'GET' or 'PUT'.
  [seconds]             The amount of time in seconds the temporary URL will
                        be valid for.
  [path]                The full path to the Swift object. Example:
                        /v1/AUTH_account/c/o.
  [key]                 The secret temporary URL key set on the Swift cluster.
                        To set a key, run 'swift post -m
                        "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"'
$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg supersecret
/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300

Then, given this information you prepend the root url of the swift service and curl at it (don't forget single quotes!):

$: swift tempurl GET 3600 /v1/10724706841504/craigslist/x220t.jpg supersecret
/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300
$: keystone catalog | grep -i object
Service: object-store
|  publicURL  | https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504 |
| versionInfo |        https://region-b.geo-1.objects.hpcloudsvc.com/v1/        |
| versionList |          https://region-b.geo-1.objects.hpcloudsvc.com          |
$: curl 'https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=7388703d11f6a3362dff1008a2f5dd3b2fd31293&temp_url_expires=1423453300'
401 Unauthorized: Temp URL invalid


After poking this for some time I found this tibit buried in the HP Cloud docs on swift, er excuse me, Object Storage:

2.7.5.2 Differences Between Object Storage and OpenStack Swift TempURL Signature Generation
There are two differences between Object Storage and OpenStack Swift TempURL signature generation:
  • OpenStack Swift Temporary URLs (TempURL) required the X-Account-Meta-Temp-URL-Key header be set on the Swift account. In Object Storage you do not need to do this. Instead we use Access Keys to provide similar functionality.
  • Object Storage Temporary URLs require the user's Project ID and Access Key ID to be prepended to the signature. OpenStack Swift does not.

So this basically means you can't use the python-swiftclient tool with hpcloud object storage. At least nearby they provide a code snipit. I've created my own script.


$: python hpcloud_swift.py GET 3600 /v1/10724706841504/craigslist/x220t.jpg $OS_ACCESS_KEY_SECRET https://region-b.geo-1.objects.hpcloudsvc.com/v1/10724706841504/craigslist/x220t.jpg?temp_url_sig=10724706841504:KAPYAYFHCH51B3ZCCB4W:044cefcdf58d4bccea7da3a4836145488a7c8496&temp_url_expires=1423453940 


This depends on setting OS_ACCESS_KEY_ID and OS_ACCESS_KEY_SECRET in your environment. These are nonstandard environment variables. Once this is set, anyone will be able to prepend the object storage root url to the front of the url output by hpcloud_swift.py and make their own janky-cdn.

The script to perform the hpcloud fakery can be found here.