Monday, February 8, 2016

Testing Ansible with Inspec

Background Information & References

Ansible testing is something I think we're going to start seeing more of in the future. Tonight I took to testing it with inspec. Inspec is an infrastructure testing tool from Chef. Inspec is created in the style of Serverspec and even throws a shoutout to that software and to mizzy specifically in the readme. There are a bunch of other tools in this space: testinfra is written in python, beaker to some extent fills this role, and goss is a go version, nicely covered by Dean Wilson.

At a high level, I feel strongly that these tools can be used to make systems operations code better. However, I think it is easy to fall into writing your code twice, once in Puppet and once in serverspec. Our limited use of Beaker at work started strong and has atrophied, mostly from frustration with this duplication and a shared sense that it wasn't providing value.

I also worry that we will end up coding to the bugs in implementations. Puppet's core types have remained untouched for years, and their behavior at this time is essentially an API contract. Each of these tools has specific implementations that have bugs and decisions baked in, and we'll be stuck with them until someone boldy breaks compatibility or we simply move to the next tool.

I was drawn to Inspec by Stuart Preston's presentation at Config Mgmt Camp 2016. What drew me in further was the use of inspec on a remote host without software installed on it. This 'agentless' mode neatly pairs with the Ansible methods, so using them together seemed reasonable. This presentation on inspec is also exceptional from Dominik Richter.

Procedure & Results

We can start with a simple ansible inventory file:

[openstack] ansible_user=root

From this we can perform a list hosts and a ping.

$: ansible --version

  config file = /etc/ansible/ansible.cfg
  configured module search path = /usr/share/ansible
$: ansible -i simple_inventory all --list-hosts

  hosts (1):
$: ansible -i simple_inventory all -m ping | SUCCESS => {
    "changed": false,
    "ping": "pong"


At this point we can set up a simple ansible playbook:

$: cat simple_ansible_playbook.yml
- hosts: all
  user: root
  - debug: msg="debug {{inventory_hostname}}"
  - apt: name=apache2 state=present

This playbook does very little, installing only the apache2 package.

Then we can write the inspec tests for it. Inspec is written in a DSL, the best introductory docs seem to be here. Inspec tests (as far as I can tell) must be contained in a 'control' block. See the example below:

control "spencer" do
  impact 0.7
  title "Test some simple resources"
  describe package('apache2') do
      it { should be_installed }

  describe port(80) do
    it { should be_listening }
    its('processes') {should eq ['apache2']}


This reads like any block of Puppet, Ansible, or other systems tooling that has been around, with a sprinkling of rspec or rspec-puppet. There is a long list of available resources in the inspec documentation.

Inspec easily installs with gem.

$: inspec version

Inspec can detect the remote machine and give you its operatingsystem version. I don't really see the direct value in this, but it is a nice ping/pong subcommand to test the connection.

$: inspec detect  -i ~/.ssh/insecure_key  -t ssh://root@

Inspec right now does not have the ability to ask my ssh-agent for permission to use my key, I have a less-secure key (though by no means totally insecure) key that I use in instances like this. The -t connection infromation flag has a fairly straightforward syntax. Like any agentless tool inspec supports a number of flags allowing it to connect as an unprivileged user and to use sudo to achieve root permissions.

We can now run our test (before our ansible playbook has run, to validate that we are getting failures).

$: inspec exec  -i ~/.ssh/insecure_key  -t ssh://root@  inspec.rb


  1) System Package apache2 should be installed
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected that `System Package apache2` is installed
     # inspec.rb:6:in `block (3 levels) in load'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/inspec/runner_rspec.rb:55:in `run'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/utils/base_cli.rb:52:in `run_tests'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'

  2) Port  80 should be listening
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected `Port  80.listening?` to return true, got false
     # inspec.rb:10:in `block (3 levels) in load'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/inspec/runner_rspec.rb:55:in `run'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/utils/base_cli.rb:52:in `run_tests'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'

  3) Port  80 processes should eq ["apache2"]
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected: ["apache2"]
            got: nil

       (compared using ==)
     # inspec.rb:11:in `block (3 levels) in load'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/inspec/runner_rspec.rb:55:in `run'
     # /tmp/file1otIsd/gems/inspec-0.10.1/lib/utils/base_cli.rb:52:in `run_tests'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
     # /tmp/file1otIsd/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'

Finished in 0.30626 seconds (files took 3.26 seconds to load)
3 examples, 3 failures

Failed examples:

rspec  # System Package apache2 should be installed
rspec  # Port  80 should be listening
rspec  # Port  80 processes should eq ["apache2"]

$: echo $?

Seeing partial tracebacks is not great, but the output is easy enough to understand. We had three things we were asserting and none of them are true.

Let's now run our ansible playbook

$: ansible-playbook -i simple_inventory simple_ansible_playbook.yml



TASK [setup] *******************************************************************
ok: []

TASK [debug] *******************************************************************
ok: [] => {
    "msg": "debug"

TASK [apt] *********************************************************************
changed: []

PLAY RECAP *********************************************************************              : ok=3    changed=1    unreachable=0    failed=0  

The colors are lost here, but the 'ok' output is green and the 'changed' output is yellow.

We can reasonably expect that the apache package was installed and that the service is running. I am usually tempted at this point to run the playbook again to verify idempotence but for such a simple playbook that isn't necessary.

Now we can re-run our inspec tests:

$: inspec exec  -i ~/.ssh/insecure_key  -t ssh://root@  inspec.rb

Finished in 0.46273 seconds (files took 3.37 seconds to load)
3 examples, 0 failures

$: echo $?

A much prettier output, to be sure.


With this we have two tools, one to make changes and the other to verify state. Unfortunately there is a lot of code duplication. We could probably write a simple tool to parse ansible and dump inspec code, but I feel like that would ultimately only be a tool that would reveal implementation differences between the two tools.

One issue I ran into in my brief use of this was around this snippit:

    its('processes') {should eq ['apache2']}

It initially read:

    its('processes') {should include 'apache'}

I could not get this check to pass. Eventually I dug into the code and discovered that its('processes') returns an array of strings, each string containing a process name. In unix, only one process can hold a port like that, so this is a bit of an odd choice for a return type. I had expected a single string to be returned and for 'include' to do a substring match on 'apache'. Not the least important result of this brief debugging was that inspec resources are very small and readable.

Using these tools together means having both ruby and python set up. This isn't a problem for me, because both of those are already in my development and testing environments. For others, using a single runtime would be more valuable than picking up a new tool. There are a lot of tools in this space, and all seem passable.

This kind of testing still requires a unix node of some type. Both ansible and inspec can test using docker, but I have come to prefer testing my infrastructure code on virtual machines. Part of this is that I have almost always had a ready supply of virtual machines to use for this testing.

Inspec is flexible. It has docker, virtual machine, and local functionality. It even has windows support. Inspec has a long list of native resources, enabling you to test high level features like 'postgres_config.' Because an inspec resource is only a status check, it is infinitely easier to write, debug and reason about.

The tone at ConfigMgmtCamp 2016 was that inspec is the future and will be replacing serverspec in most uses.

Further Work

I am interested to see what can be done with testing inspec in CI pipelines. The simple three command pattern of this blog could easily be encoded to a CI pipeline, if the CI system has sufficient resources. Inspec has a local application mode that could be used with puppet apply. I am interested to see if and when beaker integration will be attempted, as most of my infrastructure testing uses beaker at the moment.

Thanks to Bastelfreak and Bkero for editing this post prior to publication.


  1. Great description blindscientist. I'd liket to add that you do not need the `control` element for pure testing. This is only useful, if you use InSpec for compliance controls. In your case a simple:

    describe package('apache2') do
    it { should be_installed }

    describe port(80) do
    it { should be_listening }
    its('processes') {should eq ['apache2']}
    is enough.

    Regarding the port processes. `port(80)` may return multiple processes, since this applies to multiple IP addresses.

    1. Hi There,

      What a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this.
      The core API of Python provides some tools for the programmer to code reliable and more robust program. Python also has a build-in garbage collector which recycles all the unused memory.
      When an object is no longer referenced by the program, the heap space it occupies can be freed. The garbage collector determines objects which are no longer referenced by the program frees the occupied memory and make it available to the heap space.

      class Fighter:
      def __init__(self, name): = name = 100
      self.damage = 10
      def attact(self, other_guy): -= self.damage

      qazi = Fighter("Qazi")
      you = Fighter("Matt")



      Thanks a lot. This was a perfect step-by-step guide. Don’t think it could have been done better.

      Best Regards,

  2. The blog gave me idea to test ansible with inspec my sincere thanks for sharing this post Please continue to share this kind of post
    Devops Training in Bangalore

  3. Veru Nice Blog: on Testing Ansible with Inspec this is a very rare blog
    Devops Training in Bangalore

  4. Very helpful article it was a pleasure reading it.
    myTectra is the Marketing Leader In Banglore Which won Awards on 2015, 2016, 2017 for best training in Bangalore:
    python interview questions

    python online training

  5. Best Digital Marketing company Anantapur

    helpful information, thanks for writing and share this information

  6. Andhra Pradesh Engineering Agricultural and Medical Common Entrance Test AP EAMCET is a state level entrance examination conducted in the state of Andhra Pradesh for the selection of candidates into Engineering and Medicine and Agriculture courses. It is conducted by Jawaharlal Nehru technical Institute.

  7. Hi,
    Finding the time and actual effort to create a superb article like this is great thing. I’ll learn many new stuff right here! Good luck for the next post.... Click Here Know More About Ansible Training

  8. This above information really Good beginners are looking for these type of blogs, Thanks for sharing article on Devops Online Training

  9. Great write-up, I am a big believer in commenting on blogs to inform the blog writers know that they’ve added something worthwhile to the world wide web!..
    digital marketing agency in india

  10. This concept is a good way to enhance the knowledge.thanks for sharing. please keep it up salesforce Online Training Bangalore

  11. CrownQQ | Domino agent QQ | BandarQ | Domino99 Online Largest

    Who Is The Agent Bandarq, Domino 99, And The Trusted Online Poker City in Asia comes to all of you with exciting game games and exciting bonuses for all of you

    Bonus on CrownQQ:
    * Bonus rolling 0.5%, every week
    * Refferal Bonus 10% + 10%, lifetime
    * Bonus Jackpot, which you can get easily

    Featured Games CrownQQ:
    * Online Poker
    * BandarQ
    * Domino99
    * Bandar Sakong
    * Sakong
    * Bandar66
    * AduQ
    * Sakong

    More Info Visit:
    Website: AGEN BANDARQ CrownQQ
    BBM: 2B382398
    FB: AgentCrownqq
    Twitter: crown_qq