How to simulate an OpenStack Infra Slave

Situation: You’ve committed your code, you’ve submitted a patch, and yet for some reason, and regardless of the number of rechecks, your tests simply won’t pass the gate? How can you test the gate, locally, to triage what’s happening? By creating a local slave VM.


To complete this tutorial, you will need the following:

  • Vagrant
  • VirtualBox
  • A local clone of OpenStack’s system-config repository: git clone git://

Create a local.pp manifest.

A quick look at the .gitignore file at the root of the system-config project reveals that both ./manifests/local.pp and Vagrantfile are ignored. With that in mind, let us start by creating a simple local puppet manifest which describes our node:

# path: ./manifests/local.pp
# Any node with hostname "slave-.*" will match.
node /slave-.*/ {
  class { 'openstack_project::single_use_slave':
    sudo => true,
    thin => false,

The openstack_project::single_use_slave manifest is used by nodepool – or rather, by disk-image-builder on behalf of nodepool- to build the virtual machine image used in OpenStack’s gate. This happens once a day, so any changes made in system_config will require at least 24 hours to propagate to the build queue.

Create a Vagrantfile

Next, we create a Vagrantfile that invokes the above manifest. Note that I am explicitly setting hostname on each node – this allows us to choose specifically which manifest will be applied to our guest.

# path: ./Vagrantfile

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Create a new trusty slave: `vagrant up slave-trusty`
  config.vm.define "slave-trusty" do |trusty| = "ubuntu/trusty64" 'private_network', ip: ''
    trusty.vm.hostname = 'slave-trusty' # Use this to control local.pp

  # Create a new xenial slave: `vagrant up slave-xenial`
  # Will only work in vagrant > 1.8.1
  config.vm.define "slave-xenial" do |xenial| = "ubuntu/xenial64" 'private_network', ip: ''
    xenial.vm.hostname = 'slave-xenial' # Use this to control local.pp

  # Increase the memory for the VM. If you need to run devstack, this needs
  # to be at least 8192
  config.vm.provider "virtualbox" do |v|
    v.memory = 2048

  # Install infra's supported version of puppet.
  config.vm.provision "shell",
      inline: "if [ ! -f '/etc/apt/preferences.d/00-puppet.pref' ]; then /vagrant/; fi"

  # Install all puppet modules required by openstack_project
  config.vm.provision "shell",
      inline: "if [ ! -d '/etc/puppet/modules/stdlib' ]; then /vagrant/; fi"

  # Symlink the module in system_config into /etc/puppet/modules
  config.vm.provision "shell",
      inline: "if [ ! -d '/etc/puppet/modules/openstack_project' ]; then ln -s /vagrant/modules/openstack_project /etc/puppet/modules/openstack_project; fi"

  config.vm.provision :puppet do |puppet|
    puppet.manifest_file  = "local.pp"

IMPORTANT NOTE: As of Vagrant 1.8.3, the above declared slave-xenial will fail to boot properly. This is because at this time, the published ubuntu/xenial64 image does not contain the guest additions, which must be installed manually. For specifics on how to do this, please examine this launchpad issue.

Vagrant up!

Last step: Execute vagrant up slave-trusty. With luck, and a little patience, this will create a brand new, clean, running jenkins-slave for you to test your build in.

Where next?

From this point, you should take a look at the project-config repository and determine which additional VM configuration steps are being executed by your job, so you can create an environment specific to the problem you’re trying to triage. Alternatively, you can explore some of the other nodes in ./manifests/site.pp, and perhaps extend the Vagrantfile above to instantiate a VM for one of infra’s services, such as StoryBoard or Grafana. Using the above template, you should be able to construct test instances of any infra component.

Update (June 27th, 2016)

The above method may also be used to simulate a regular OpenStack Infra server, with a few modifications. For this example, we’ll try to simulate an OpenStack Mirror. Add the following to your local puppet manifest:

# path: ./manifests/local.pp
node mirror {
  # This module is included on all infra servers. It sets up accounts, public keys, and the like.
  class { 'openstack_project::server':
    iptables_public_tcp_ports => [22, 80],
    sysadmins                 => hiera('sysadmins', [])
  # This module includes functionality specific to this server.
  class { 'openstack_project::mirror':
    vhost_name => $::ipaddress,
    require    => Class['Openstack_project::Server'],

After doing so, add this node to your Vagrantfile:

# path: ./Vagrantfile

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Create a new mirror slave: `vagrant up mirror`
  config.vm.define "mirror" do |mirror| = "ubuntu/trusty64" 'private_network', ip: ''
    trusty.vm.hostname = 'mirror' # Use this to control local.pp

... # Continue from example above.

And done! Now you can invoke vagrant up mirror and watch as your openstack-infra mirror server is provisioned. There are a few caveats:

  1. If you want to add a new puppet module, you’ll want to add it to modules.env. Doing so will only trigger an automatic install if you’re starting from a fresh guest host, so you’ll either also have to install it manually, or recreate your guest.
  2. Some manifests require a hostname. In this case, I usually reference the hosts’ IP Address, as managing DNS is too much effort for most test scenarios. vhost_name => $::ipaddress

JavaScript on the Trailing Edge

The public opinion of the JavaScript community is that it’s fast. We break things, we’re hungry for the latest features, and none of us want to return to the days of slow innovation that ended with the death of IE6. This really isn’t true; there are several core JavaScript projects, such as Angular, JQuery, and React, which have solid governance, and measured release cycles, that would mesh well with OpenStack. It just happens that those projects are surrounded by thousands of smaller ones, run by handfuls of engineers who are volunteering their time.

However, the JavaScript community offers many benefits, from layout frameworks to new user interface paradigms, and OpenStack could easily take advantage of all these. As I’ve pointed out, the user interface needs of a cloud platform vary by user, not by deployment, and it is high time that OpenStack catered to more than just the Operator mind set. There remain some obstacles to this, however they are easily solved:

Backwards Compatibility

The first challenge we face is backwards compatibility. We must balance the rapid adoption of new developments like ES6, with downstream LTS support commitments that can last several years. We must do all this, while also not losing ourselves in a morass of hacks, special cases, shortcuts, and workarounds. This requires common dependency management for all our JavaScript libraries, and we can easily draw on the lessons learned in OpenStack’s requirements project to lead the way.


Furthermore, we face a social challenge, that of complacency. The counterargument I most frequently get is “Why not use Horizon”. As my previous post on composable cloud interfaces highlights, Horizon is too narrowly focused. While it does an admirable job at supporting the Operator use case, and provides many ways to extend itself, a brief survey I performed last year revealed that two thirds of downstream Horizon users either maintain full forks of horizon’s source, or are building entirely custom user interfaces. To me, this is stark evidence that horizon falls short of meeting the use cases of all of our OpenStack operators.


Lastly, we face the rather pedestrian challenge of funding. While I’ve come across broad support for a greater role of JavaScript in OpenStack’s UI development – to the level of squeefun bouncing in a hallway when I mentioned ES6 – it remains a fact of life that those corporate members with the most to gain by the strategic evolution of OpenStack are usually content to let ‘someone else’ do the work, while dedicating their own employees towards more immediate revenue sources.


It’s a Catch-22 situation: We cannot prove the viability of JavaScript thick-client UI’s without a functional alternative to horizon, but we cannot get to that alternative without engineers willing – and able – to contribute. Personally, I feel very privileged to be one of a very small number of fully dedicated upstream engineers. To the best of my knowledge, Elizabeth Elwell and I are the only two entirely dedicated towards strategically advancing User Interface development in OpenStack. We are making good progress, however we do not anticipate adoption in the next cycle.

With help, Newton will contain the last pieces we need.

Horizon Usage Survey

Over the past few weeks, I’ve run a survey that attempts to discover how people use OpenStack’s Horizon (aka openstack-dashboard), and I’d like to publish some preliminary results. I’ll be soliciting responses during the Vancouver Summit next week, so if you haven’t participated yet, you still have time to do so. The link to do so is here:


In two weeks, the survey gathered 36 responses. Due to the small sample size and the non-random selection of participants, this data should not be considered statistically significant — Self-selected populations rarely are — however it does provide us with a window into how Horizon is used in the real world.

OpenStack Deployment Statistics

The following are charts that address the scale of our users’ OpenStack deployments.

Deployment Size

This is an indication of how many bare-metal instances comprise our user’s clouds.

OpenStack Version

What versions are currently deployed by our users. Note that some deploy multiple clouds.

Cloud Type

The type of cloud gives us an indication of what use cases our users encounter.

Horizon Deployment

These charts represent information about Horizon usage.

What is your UI?

Whether our users use Horizon, a custom-build UI, or both.

Install Tools

What tools do our users use to install and maintain horizon.

Host Operating System

The operating system on which Horizon is installed.

Horizon Customization

Information about the tools that are used to customize horizon, what parts of horizon are customized, and where Horizon falls short.

How did you customize?

There are many ways to customize horizon: Plugins, the Customization Module, creating your own Django Application with horizon as a dependency, or to just maintain your own source fork.

What was changed?

Which parts of Horizon were customized: Templates, Behaviors, Workflows, or more?

Maintained Source

In the case of a Django application, Custom UI, or a Horizon Fork, our users must maintain their own source repository.

What is the one key feature missing from horizon?

This was a free-form question, so I’ve taken the liberty to group the responses into different categories.

Usability and simplified experience

These responses address simplicity and usability in horizon.

  • Customer Facing features that improve and simplify the experience.
  • Masking Networks that cannot be attached to an instance during the instance boot wizard.
  • Simple image panel that only shows latest images, instead of all images.
  • Improved access and usability of horizon’s metrics visualization.
  • Use-friendly instance creation.
Hosted Cloud Features

These seem to be feature requests focused around hosting a cloud provider and selling it as a self-service cloud platform.

  • Self-service project management (Project Admin/Owner, etc).
  • Billing & Pricing integration.
New Features

These appear to be features

  • Approval Automation for Quotas, Tenants, and allocations.
  • Cloud Federation.
    (note: one respondent indicated that they fielded their own user interface because horizon could not talk to other clouds)
Extensibility Improvements
  • Panel Extensions are difficult to manage.
  • No uniform way to import horizon extensions, too many options.

For the sake of completeness, I’ve added features here that are not easily categorized.

  • Invincibility
  • Too many to List