Angular > React

I’ve been overseeing UI projects for … oh, decades now. I’ve shipped production applications in backbone.js, SproutCore, Angular 1, Angular 2, React/Redux, even going back to the old days of the Adobe Flex ecosystem with Cairngorm, PureMVC, and Parsley. I’m old, I’m grouchy, I’ve been around the block a few times, I’ve made all the mistakes.

I still think Angular is better than React.

Now, don’t get me wrong – I understand the appeal of React, and why Angular’s a tough sell (even though they can work together rather well). React’s learning curve is lower, it’s not a full framework, you can add React to an already existing application easily, while Angular (to be effective) requires a full rewrite. It’s very easy to get started with React, while Angular has a famously steep learning curve. React doesn’t force much on you, Angular… really does. As a result of this, most ‘comparisons’ I’ve found out there are poorly informed, and rarely overcome the resentment of “why are you making me use ThingA rather than ThingB I’m used to?”

And, lastly, React is the “sexy JS library du jour”, hand in hand with Vue.js. That counts for a lot, even though this is the Bandwagon Fallacy.

The React Snowflake

The first reason I dislike React, is because it encourages snowflake projects; projects that are different enough in their other technical decisions as to be effectively unique. This is often put forward as one of the “nice” things about React, however I consider it its worst shortcoming.

To illustrate, let’s list out a few of the other decisions that go into a project:

  • Language: Typescript, Javascript, other.
  • Build system: Webpack, Gulp, Grunt, Bazel, Babel?
  • Data layer: RxJS, Redux, MobX, too many to list.
  • In-app navigation: HTML5, #! based, React-navigation, etc.
  • Linting rules: Single-line vs multi-line variables, 2 vs. 4 spaces, etc.

If the above seems mostly irrelevant from your point of view, then I hazard to guess that you are working on a small team, where the learning curve and arguments around these technical decisions is lost in the churn of on-boarding. Either that, or your organization has put a lot of work into enforcing consistency.

“There should be one– and preferably only one –obvious way to do it.”

Zen of Python (PEP-20)

In the Angular ecosystem, most of the above decisions have been made for you. As a result, if you switch between Angular applications, it’s very likely that are that you’ve seen it all before. While on the outside this seems authoritarian, in reality it is highly liberating – you are free to focus on interesting technical problems, instead of fighting the tabs-vs-space battle for the upteenth time in your career.

In short: React encourages divergence, while Angular encourages consistency.

Feature Reuse (it’s more than just code)

If your company is successful, it will grow. This usually means additional projects, such as admin dashboards, white labels, or even entirely new products. The way this usually happens is with a copy/paste of the existing project, deleting all the bits that aren’t needed for the new application, at which point the implementation details begin to fragment. This especially becomes a headache when trying to maintain a consistent user experience, as one button may evolve away from the other. 

This is where Angular really shines, though I’ll have to dig into technical details a little to make my point. In short, Angular is actually an Inversion of Control engine. You build a component or service, and then provide it to the Injector to manage. This can be done individually, or groups of components can be bundled together in a ‘Module‘. These modules are powerful – as an example, you could build a UserProfileModule that, when injected, takes over the /user/profile path in any application that it’s provided to (it can even be lazy-loaded). Other modules can provide ambient features, such as the popular Angulartics2 metrics library.

As you can see, this is much more powerful concept than simple ‘code reuse’, as entire features can be not only shipped, but trivially imported. It’s one of the advantages of the heavy guard-rails that come with Angular – if everyone agrees on how to build lego’s, then it’s easier to fit them together.

So now, let’s return to the above problem. You’re creating a brand new application, and rather than copy/pasting from existing codebases, you add a dependency on an angular library that provides you with modules that you want to use. If the library updates, you do too.

A more concrete example of this is VMware Clarity. One of the reasons we’ve been so successful at maintaining a consistent UI across all of VMware’s SaaS Products is because our UI components come in one easy-to-use module.

Technical choices are business choices

Are you in a startup, planning to be acquired? Are you in a large organization, with multiple projects that need to maintain a common look & feel? Are you about to face a major re-org? Is one product being moved to maintenance while another is being built?

These kinds of changes are common, as our industry is very dynamic. And speaking as someone who’s been involved in Technical Due Diligence, I’ve seen acquisition deals fall through because technical decisions made by the prospective organizations would be too difficult to reconcile.

In the end, every technical decision we make is a strategic one – which is one of the reasons I’m a huge fan of the Choose Boring Technology philosophy. With that in mind, recall that an Angular application can use React components, while a React application cannot easily use an Angular application. In other words, it’s easier for you to adopt another organization’s code if you start with Angular, even if they’re React-based.

Encrypting sensitive variables in terraform using GnuPG agent

This post will walk you through how to encrypt sensitive terraform variables in a way that still permits them to be committed to VCS, while also being reasonably easy to decrypt. Examples use bash, however are easily adapted to other environments. Special thanks to this post on encrypting the ansible vault password, as my examples draw heavily from that source.

This method is particularly awesome, because you can explicitly declare who is permitted to decrypt it. So, for instance, all of the engineers on your team could be unable to access its contents, while your CI/CD system (jenkins or whatnot) can use their own gpg identity to decrypt the data.

Step 1: Create a json file with your sensitive variables

echo "{ \"github_api_key\": \"secret\" }" >> variables.json

Step 2: Encrypt it using gpg

gpg --encrypt ./variables.json 

The above call will ask you to add a list of recipients. In general it’s a good idea to add yourself, so you can decrypt the file at a later point. You can also add other users, such as the aforementioned CI/CD engine, or specific members of your team that you trust.

Step 3: Create an external provider in terraform

Terraform permits you to read data from external programs, as long as they return JSON. This is easily done:

data "external" "gpg" {
  program  = [ "gpg", "--batch", "--use-agent", "--decrypt", "./variables.json.gpg"]

provider "github" {
  token    = "${data.external.gpg.result.github_api_key}"

Now, assuming that you have a running gpg-agent, you’ll only have to enter your gpg passkey as your own security policy requires.

The only major caveat with the above, is that you cannot use these variables to maintain the terraform state.

How to simulate an OpenStack Infra Slave

Situation: You’ve committed your code, you’ve submitted a patch, and yet for some reason, and regardless of the number of rechecks, your tests simply won’t pass the gate? How can you test the gate, locally, to triage what’s happening? By creating a local slave VM.


To complete this tutorial, you will need the following:

  • Vagrant
  • VirtualBox
  • A local clone of OpenStack’s system-config repository: git clone git://

Create a local.pp manifest.

A quick look at the .gitignore file at the root of the system-config project reveals that both ./manifests/local.pp and Vagrantfile are ignored. With that in mind, let us start by creating a simple local puppet manifest which describes our node:

# path: ./manifests/local.pp
# Any node with hostname "slave-.*" will match.
node /slave-.*/ {
  class { 'openstack_project::single_use_slave':
    sudo => true,
    thin => false,

The openstack_project::single_use_slave manifest is used by nodepool – or rather, by disk-image-builder on behalf of nodepool- to build the virtual machine image used in OpenStack’s gate. This happens once a day, so any changes made in system_config will require at least 24 hours to propagate to the build queue.

Create a Vagrantfile

Next, we create a Vagrantfile that invokes the above manifest. Note that I am explicitly setting hostname on each node – this allows us to choose specifically which manifest will be applied to our guest.

# path: ./Vagrantfile

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Create a new trusty slave: `vagrant up slave-trusty`
  config.vm.define "slave-trusty" do |trusty| = "ubuntu/trusty64" 'private_network', ip: ''
    trusty.vm.hostname = 'slave-trusty' # Use this to control local.pp

  # Create a new xenial slave: `vagrant up slave-xenial`
  # Will only work in vagrant > 1.8.1
  config.vm.define "slave-xenial" do |xenial| = "ubuntu/xenial64" 'private_network', ip: ''
    xenial.vm.hostname = 'slave-xenial' # Use this to control local.pp

  # Increase the memory for the VM. If you need to run devstack, this needs
  # to be at least 8192
  config.vm.provider "virtualbox" do |v|
    v.memory = 2048

  # Install infra's supported version of puppet.
  config.vm.provision "shell",
      inline: "if [ ! -f '/etc/apt/preferences.d/00-puppet.pref' ]; then /vagrant/; fi"

  # Install all puppet modules required by openstack_project
  config.vm.provision "shell",
      inline: "if [ ! -d '/etc/puppet/modules/stdlib' ]; then /vagrant/; fi"

  # Symlink the module in system_config into /etc/puppet/modules
  config.vm.provision "shell",
      inline: "if [ ! -d '/etc/puppet/modules/openstack_project' ]; then ln -s /vagrant/modules/openstack_project /etc/puppet/modules/openstack_project; fi"

  config.vm.provision :puppet do |puppet|
    puppet.manifest_file  = "local.pp"

IMPORTANT NOTE: As of Vagrant 1.8.3, the above declared slave-xenial will fail to boot properly. This is because at this time, the published ubuntu/xenial64 image does not contain the guest additions, which must be installed manually. For specifics on how to do this, please examine this launchpad issue.

Vagrant up!

Last step: Execute vagrant up slave-trusty. With luck, and a little patience, this will create a brand new, clean, running jenkins-slave for you to test your build in.

Where next?

From this point, you should take a look at the project-config repository and determine which additional VM configuration steps are being executed by your job, so you can create an environment specific to the problem you’re trying to triage. Alternatively, you can explore some of the other nodes in ./manifests/site.pp, and perhaps extend the Vagrantfile above to instantiate a VM for one of infra’s services, such as StoryBoard or Grafana. Using the above template, you should be able to construct test instances of any infra component.

Update (June 27th, 2016)

The above method may also be used to simulate a regular OpenStack Infra server, with a few modifications. For this example, we’ll try to simulate an OpenStack Mirror. Add the following to your local puppet manifest:

# path: ./manifests/local.pp
node mirror {
  # This module is included on all infra servers. It sets up accounts, public keys, and the like.
  class { 'openstack_project::server':
    iptables_public_tcp_ports => [22, 80],
    sysadmins                 => hiera('sysadmins', [])
  # This module includes functionality specific to this server.
  class { 'openstack_project::mirror':
    vhost_name => $::ipaddress,
    require    => Class['Openstack_project::Server'],

After doing so, add this node to your Vagrantfile:

# path: ./Vagrantfile

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Create a new mirror slave: `vagrant up mirror`
  config.vm.define "mirror" do |mirror| = "ubuntu/trusty64" 'private_network', ip: ''
    trusty.vm.hostname = 'mirror' # Use this to control local.pp

... # Continue from example above.

And done! Now you can invoke vagrant up mirror and watch as your openstack-infra mirror server is provisioned. There are a few caveats:

  1. If you want to add a new puppet module, you’ll want to add it to modules.env. Doing so will only trigger an automatic install if you’re starting from a fresh guest host, so you’ll either also have to install it manually, or recreate your guest.
  2. Some manifests require a hostname. In this case, I usually reference the hosts’ IP Address, as managing DNS is too much effort for most test scenarios. vhost_name => $::ipaddress

JavaScript RoadMap for OpenStack Newton

This post contains the current working draft of the OpenStack JavaScript roadmap. It’s a big list, and we need help to land it during the Newton cycle. Overall themes for this cycle are Consistency, Interoperability, and engaging with the JavaScript community at large, all topics which I’ve written about at length. Our end goal is to build the foundations of a JavaScript ecosystem, which permits the creation of entirely custom interfaces.

Note: We are not trying to replace Horizon, we are aiming to help those downstream who need something more than “Vanilla OpenStack”, and thus maintain their own code. The vast majority of modern UI and UX development happens in JavaScript, and there are many use cases that have not yet been met.

OpenStack Projects

These projects are part of the big tent, and will see significant development during the Newton Cycle.


The ironic-webclient will release its first version during the Newton cycle. We’re awfully close to having the basic set of features supported, and with some excellent feedback from the OpenStack UX team, will also have a sexy new user interface that’s currently in the review queue. Once this work is complete, we will begin extracting common components into a new project, named…


This new project will be incubated as a single, gate-tested JavaScript API client library for the OpenStack API’s. Its audience is software engineers who wish to build their own user interface using modern javascript tools. As we cannot predict downstream use cases, special care will be taken to ensure the project’s release artifacts can eventually support both browser and server based applications.

Philosophically, we will be taking a page from the python-openstackclient book, and avoid creating a new project for each of OpenStack’s services. We can make sure our release artifacts can be used piecemeal, however trying to maintain code consistency across multiple different projects is a hard lesson that others have already learned for us. Let’s not do that again.

Infrastructure Projects

These projects belong to OpenStack’s Infrastructure and/or QA team. They’re used to support the building of JavaScript projects in OpenStack.


Yeoman is JavaScript’s equivalent of cookiecutter, providing a scaffolding engine which can rapidly set up, and maintain, new projects. Creating and maintaining a yeoman generator will be a critical part of engaging with the JavaScript community, and can drive adoption and consistency across OpenStack as well. Furthermore, it is sophisticated enough that it could also support many things that exist in today’s Python toolchain, such as dependency management, and common tooling maintenance.

Development of the yeoman generator will draw in lessons learned from OpenStack’s current UI Projects, including Fuel, StoryBoard, Ironic, Horizon, Refstack, and Health Dashboard, and attempt to converge on common practices across projects.


This project aims to bridge the gap between our JavaScript projects, and Horizon’s measured migration to AngularJS. We don’t believe in duplicating work, so if it is feasible to publish our libraries in a way that Horizon may consume (via the existing xstatic toolchain), then we certainly should pursue that. The notable difference is that our own projects, such as js-openstacklib, don’t have to go through the repackaging step that our current xstatic packages do; thus, if it is possible for us to publish to npm and to xstatic/pypi at the same time, that would be best.

Xenial Build Nodes

As of two weeks ago, OpenStack’s Infrastructure is running a version of Node.js and npm more recent than what is available on Trusty LTS. Ultimately, we would like to converge this version on Node4 LTS, the release version maintained by the Node foundation. The easiest way to do this is to simply piggyback on Infra’s impending adoption of Xenial build nodes, though some work is required to ensure this transition goes smoothly.

Maintained Projects

The following projects are active and considered ‘complete’, though they will require continuous maintenance throughout the Newton cycle. I’ve included all the needed work that I am aware of, however if there’s something I’ve missed, please feel free to comment.


eslint has updated to version 2.x, and no more rule bugfixes are being landed in 1.x. eslint-config-openstack will follow in kind, updating itself to use eslint 2.x. We will releases this version as eslint-config-openstack v2.0.0, and continue to track the eslint version numbers from there. Downstream projects are encouraged to adopt this, as it is unlikely that automated dependency updates for JavaScript projects will land this cycle.

NPM Mirrors

We are currently synchronizing all npm packages to our AFS master disks, which should be the final step in getting functional npm mirrors. Some minor tweaking will be required to make them functional, and they will need to be maintained throughout the next cycle. Issues raised in the #openstack-infra channel will be promptly addressed.

This includes work on both the js-openstack-registry-hooks project and the js-afs-blob-store project, which are two custom components we use to drive our mirrors.


CORS landed in mitaka, and we will continue to maintain it going forward. In the Newton cycle, we have the following new features planned:

  • Automatic allowed_origin detection from Keystone (zero-config).
  • More consistent use of set_defaults.
  • Configuration maintenance as projects deprecate X-* headers in accordance with RFC 6648.

Stretch Projects

These are projects which we know need to be done, however we simply do not have enough contributors.


Documentation is important. Usable documentation is even more important. The tricky bit is that OpenStack’s documentation is all python/sphinx based, and we have not yet investigated whether it’s possible to bridge the two languages. If you have time to explore this intersection, we’d be happy to hear your findings.

That concludes it for the Newton Cycle. As you can see, there’s a lot of work to do. Can you help?

JavaScript on the Trailing Edge

The public opinion of the JavaScript community is that it’s fast. We break things, we’re hungry for the latest features, and none of us want to return to the days of slow innovation that ended with the death of IE6. This really isn’t true; there are several core JavaScript projects, such as Angular, JQuery, and React, which have solid governance, and measured release cycles, that would mesh well with OpenStack. It just happens that those projects are surrounded by thousands of smaller ones, run by handfuls of engineers who are volunteering their time.

However, the JavaScript community offers many benefits, from layout frameworks to new user interface paradigms, and OpenStack could easily take advantage of all these. As I’ve pointed out, the user interface needs of a cloud platform vary by user, not by deployment, and it is high time that OpenStack catered to more than just the Operator mind set. There remain some obstacles to this, however they are easily solved:

Backwards Compatibility

The first challenge we face is backwards compatibility. We must balance the rapid adoption of new developments like ES6, with downstream LTS support commitments that can last several years. We must do all this, while also not losing ourselves in a morass of hacks, special cases, shortcuts, and workarounds. This requires common dependency management for all our JavaScript libraries, and we can easily draw on the lessons learned in OpenStack’s requirements project to lead the way.


Furthermore, we face a social challenge, that of complacency. The counterargument I most frequently get is “Why not use Horizon”. As my previous post on composable cloud interfaces highlights, Horizon is too narrowly focused. While it does an admirable job at supporting the Operator use case, and provides many ways to extend itself, a brief survey I performed last year revealed that two thirds of downstream Horizon users either maintain full forks of horizon’s source, or are building entirely custom user interfaces. To me, this is stark evidence that horizon falls short of meeting the use cases of all of our OpenStack operators.


Lastly, we face the rather pedestrian challenge of funding. While I’ve come across broad support for a greater role of JavaScript in OpenStack’s UI development – to the level of squeefun bouncing in a hallway when I mentioned ES6 – it remains a fact of life that those corporate members with the most to gain by the strategic evolution of OpenStack are usually content to let ‘someone else’ do the work, while dedicating their own employees towards more immediate revenue sources.


It’s a Catch-22 situation: We cannot prove the viability of JavaScript thick-client UI’s without a functional alternative to horizon, but we cannot get to that alternative without engineers willing – and able – to contribute. Personally, I feel very privileged to be one of a very small number of fully dedicated upstream engineers. To the best of my knowledge, Elizabeth Elwell and I are the only two entirely dedicated towards strategically advancing User Interface development in OpenStack. We are making good progress, however we do not anticipate adoption in the next cycle.

With help, Newton will contain the last pieces we need.

OpenStack Infra now uses Node.js v4 and npm v2

OpenStack’s Infrastructure is now running all of its NPM and NodeJS-based test jobs using the newer NodeJS v4, and npm 2.15. That’s pretty awesome, given that previously we were running on 0.10.25. ES6, anyone?

LTS is important

Here in OpenStack we try to stick as closely as possible to LTS packages. We do this for multiple reasons, chief of which is that many of our customers have governance and compliance constraints that prevent them from installing arbitrary bits on their hardware. Furthermore, up until recently, the only feasible test platform was Ubuntu Trusty (CentOS7 and RHEL are recent additions), which forced us to rely on NodeJS v0.10.25 and npm v 1.3.10. Neither of these are supported by the Node Foundation or npm inc., and security backports are the responsibility of the distro packaging team.

Vulnerable to Upstream Changes

These out-of-date versions leave us vulnerable to upstream changes, and last week we encountered this issue: npm upgraded the registry to provide gzipped content, something which our older npm client did not understand. The fix was slow in coming (at no fault of npm inc’s engineers), and we were left unable to access registry packages in a reliable way, preventing us from releasing some components of Mitaka… during release week.

When npm breaks, what do we do?

We needed two things- a quick fix, and a future strategy. If possible, both at the same time. Sadly, we couldn’t just update npm; while there’s precedent (we run a non-distro version of pip), the older npm client could not reach the registry to download the packages necessary to update itself. A different approach was needed.

Last summer, the Node Foundation joined the Linux Foundation, and announced plans for an LTS release schedule which was realized in October. Shortly afterwards, linux distro packages began appearing for those versions, and they have since graduated in both Debian and Ubuntu. While neither of these are yet available to us (Xenial Xerus releases later this month), it nevertheless gave us a clear path forward:

1. Upgrade to Node4 via the NodeSource package repository

In order to unblock our builds, we upgraded all of our jenkins slaves to use the NodeSource repository’s version of NodeJS 4, which ships with npm 2.15. While not precisely the LTS versions, they were close enough to solve the issue we were encountering. This would give us a backwards-compatible solution for all of our trusty nodes, while paving the way to the adoption of xenial.

This work has been completed, and our gate is happy again. Woohoo!

2. Switch to Xenial as quickly as feasible

We’ve already been testing xenial nodes in OpenStack’s gate, and anticipate officially making those nodes available during the Newton cycle. Once those become available, we’ll start moving all of our javascript jobs to those nodes instead.

3. Build a static mirror on top of OpenStack’s unified mirror infrastructure

By hosting our own package mirror, we isolate ourselves from many problems from the upstream package registry. There have already been plans to build this, and the first major components of it (Unified Mirrors) have already landed. The only thing these recent problems (as well as similar problems like leftpad) have done, is raised the urgency.


Stay tuned for updates

I’ll be posting updates on this effort to this blog, which is syndicated to OpenStack planet. Stay tuned, and let me know if you want to help! My IRC handle is @krotscheck, and I’m always online on FreeNode’s #openstack-infra channel.

Alternative Interfaces for OpenStack

What might alternative user interfaces for OpenStack look like? If you’ve read my previous post on Composable OpenStack, you understand that there is more than one way to think about – and interact with – a cloud. Use cases – thus desired user interaction – vary even more than business models, and while Horizon’s OpenStack Dashboard admirably satisfies the Cloud Operator use case, there are many other use cases which it struggles to satisfy.

As a research exercise, I’ve collected and/or mocked user interfaces for the use cases I’ve outlined in my aforementioned blog post. These are intended to be illustrative, not proscriptive. Perhaps most importantly, these interfaces can – and should – be able to live side-by-side, with as little code duplication as possible.

Option 1: The multi-mega-cloud

I’m not certain anyone has ever tried to tackle the Mega-Ops user experience. Multiple clouds, dozens of global regions, thousands of pieces of hardware… it’s a daunting challenge to try to distill that into a single, usable, interface. I’ve taken a stab at it, however it’s probably far more useful to focus on specific users; A NOC’s security dashboard might be entirely separate from its capacity dashboard, and may even be managed in an entirely separate location.


Option 2: The single-cloud operator

This is the use case closest to Horizon’s OpenStack-Dashboard. Here, the customer knows they are interacting with a cloud – concepts such as instances, quotas, images, and network topologies are front and center, and both the cloud’s operator, and their customers, will find the concepts familiar and/or easy to grasp.


Option 3: Is there a cloud here somewhere?

I’ve taken a screenshot here of the signup page for, one of the keynote customers from the 2015 OpenStack Summit in Vancouver. Pantheon built their entire business on top of OpenStack and Containers, however this is entirely hidden from their customers; All they know is that they enter some parameters, ‘swipe’ their credit card, and have a wordpress blog ready to go.


Option 4: The Application Developer

This dashboard is designed for an application developer, someone who cares more about the components of their application, than about the underlying instances running these components. In fact, the developer implicitly trusts their cloud provider to configure these services in a way that will scale to meet their needs; After all, not every engineer can learn everything there is to know about every service in their application stack, as this breadth of knowledge comes at the cost of depth.

And yet, this is still a valid dashboard for OpenStack, especially its secondary services such as Trove and Sahara. In fact, this interface removes the source of computing power entirely; the resources could be provided via HyperVisors or via Containers, and the customer would never really know the difference.


Option 5: The Single-Service Deployer

The last use case is that of the single-service deployer, in which, for whatever reason, a single service such as Swift or Ironic are deployed entirely by themselves. Here, I’ve taken a screenshot of the Ironic Standalone UI, which attempts to meet that use case with no dependency on Horizon whatsoever.

Screen Shot 2016-04-06 at 2.25.37 PM

Securely publishing to NPM, the OpenStack way

The following article has been making the rounds, claiming a new worm exploit against npm. First of all, this is not a new exploit, nor is it in any way unique to npm – pip, gem, rpm, and deb have the same issue. Many may not even consider this an exploit at all – it’s a known feature, provided by package repositories, that permit compiling platform-specific bytecode. This is useful if, for instance, your package depends on a c-level library.

The exploit works something like this:

  1. If you are easily authenticated against a package repository, and…
  2. …you install a package which compiles and runs someone else’s code, then…
  3. …an attacker can execute malicious code which can publish itself to your packages…
  4. …which will then subsequently infect anyone who fuzzily matches against your packages’ versions.

This is not news. Mitigation approaches exist. Here’s how we do it in OpenStack:

Step 1: Do not use privileged build environments

Every test, package, or other build command runs on a throwaway jenkins slave that only exists for that test, after which it is deleted. While during test setup the jenkins user begins with passwordless sudo, that privileged is almost always revoked before the tests are run. In short, even if malicious code is downloaded during npm install, it is never executed in an environment that permits a privilege escalation attack.

This approach doesn’t have to be restricted to our Cloud VM’s either. You can do this with docker images, vagrant boxes, you name it.

Step 2: Build an intermediary tarball with `npm pack`

`npm pack` builds a release tarball from a given package, in our case the current project at its release version tag. We do this on the above-mentioned throwaway slave, so that any scripts executed during the construction process cannot access any credentials. After construction, this tarball is uploaded to, from which anyone can retrieve it.

Step 3: Publish without scripts

OpenStack’s infrastructure contains one jenkins slave that possesses credentials necessary to publish artifacts. Its sole purpose in life is to download a release tarball, and to push that tarball to a package repository. In npm’s case, we execute  `npm publish <tarball> --ignore-scripts`, to ensure that none of the packages’ lifecycle events are accidentally executed, further isolating us from unexpected attacks.

Other security measures

In addition to the above publishing flow, we also have several policies in place intended to ensure that our packages are trustworthy.

    • Only one user owns our npm packages. This prevents other owners from accidentally compromising the package.
    • Only verified, gpg-signed git tags using registered keys will trigger our publish jobs. To easily enable this, add git-sign-tag=true to your global or local .npmrc (Of course, you’ll need to be able to sign a tag).
    • We strongly prefer using strict version matching in our packages, which also has the benefit of making our builds deterministic. The fastest way for you to accomplish this yourself is to commit your shrinkwrap file to version control.
    • We don’t just publish our code via npm; If you’d prefer using a git dependency, or a tarball link, that option is open to you.

We need a consistent OpenStack

The following is a table of some basic implementation details in OpenStack’s Mitaka API projects. It isn’t intended to shame anyone; it is intended to highlight tool and framework fragmentation in OpenStack. Here’s the data, the article follows below.

Integrated Release APIs

ceilometer PasteDeploy pecan,wsme py27,py34 global-requirements oslo-generate-config
cinder PasteDeploy routes py27 global-requirements oslo-generate-config
glance PasteDeploy routes,wsme py27,py34 global-requirements oslo-generate-config
heat PasteDeploy routes py27,py34 global-requirements oslo-generate-config
ironic pecan,wsme py27,py34 global-requirements
keystone PasteDeploy routes py27,py34 global-requirements oslo-generate-config
neutron PasteDeploy pecan py27,py34 global-requirements oslo-generate-config
nova PasteDeploy routes py27,py34 global-requirements oslo-generate-config
sahara flask py27,py34 global-requirements oslo-generate-config
swift ? py27,py34
trove PasteDeploy routes py27,py34 global-requirements

Supporting API Projects

aodh PasteDeploy pecan,wsme py27,py34 oslo-generate-config
barbican PasteDeploy pecan py27,py34 global-requirements
cloudkitty PasteDeploy pecan,wsme py27,py34 oslo-generate-config
congress PasteDeploy routes py27,py34 global-requirements oslo-generate-config
cue pecan,wsme py27,py34 global-requirements oslo-generate-config
designate PasteDeploy flask,pecan py27,py34 global-requirements
freezer falcon py27,py34 global-requirements
fuel py27,py34
kite pecan,wsme py27,py34
magnum pecan,wsme py27,py34 global-requirements oslo-generate-config
manila PasteDeploy routes py27,py34 global-requirements oslo-generate-config
mistral pecan,wsme py27,py34 global-requirements oslo-generate-config
monasca-api falcon py27
monasca-log-api falcon py27
murano PasteDeploy routes py27 global-requirements oslo-generate-config
searchlight PasteDeploy routes,wsme py27,py34 global-requirements oslo-generate-config
senlin PasteDeploy routes py27,py34 global-requirements oslo-generate-config
solum pecan,wsme py27,py34 oslo-generate-config
tacker PasteDeploy routes py27,py34 global-requirements
zaqar falcon py27,py34 global-requirements oslo-generate-config

Just scratching the surface

The table above only scratches the surface of OpenStack’s tool fragmentation, as it only focuses on frameworks and configuration in API projects. It does not address other inconsistencies, such as supported image types, preferred messaging layers, testing harnesses, oslo library adoption, or a variety of other items.

I’ve already spoken about Cognitive Load and OpenStack, how the variability in our projects can trick your brain and make you less effective. Furthermore, we’ve seen a lot of discussion on how we should Choose Boring Technology, as well as hallway discussions about how OpenStack should be more opinionated in its deployments. In fact, things have gotten so bad that the shade project was created – a library whose express intent is to hide all the differences in deployed OpenStack clouds.

Variability is bad for OpenStack

The lack of consistency across OpenStack is harming us, in very specific ways.

Contributing to multiple projects is hard

Nobody wants to climb multiple learning curves. Knowledge from one project directly transfers to another if the frameworks are similar enough, reducing this learning curve. In short, differences in projects create barriers to cross-project fertilization and contribution, and one way to chip away at those differences is to keep them as similar as possible.

Supporting OpenStack’s dependencies is hard

As an open source community, there is a strong ethos of helping support any projects that we depend on. Yet, how do we pick which upstream project to help fix? If all projects were consistent in their use of, say, WSME, there would be a far larger pool of talent invested in success, and discussions like this one would not happen as frequently (Note: I’m not necessarily advocating WSME here – it merely provides a very useful example).

Maintaining feature consistency is hard

There are many features which our various projects should all support. Simple things, like consistent search query parameters, consistent API version negotiation, consistent custom HTTP Header names – basically anything cross-project or from the API working group.

I have personal experience with this: In Mitaka I was able to land CORS support in most of OpenStack’s API’s. Of the 23 projects that I contributed to, most required that I learn project-specific approaches to governance, launchpad usage, testing harnesses, commit-message constraints, folder structure, and more. The entire experience only taught me one thing: Trying to bring a common feature to OpenStack is something I never want to do again.

Deploying/Running OpenStack is hard

Without an opinionated OpenStack install (down to the supporting services), the chances that someone has run into the same problem as you, drops significantly. Features which rely on service abstraction (messaging for instance) depend on layers of driver abstractions which add more points of failure, and often have to provide workarounds for features supported in one service, but not in another (assuming they don’t give up on that feature entirely).

Portability between clouds is hard

To quote Monty Taylor: “The existence of shade is a bug”. It turns out that OpenStack’s implied portability promise is pretty much a lie, and you will spend a significant amount of effort figuring out how this OpenStack happens to differ from That OpenStack.

We need a consistent OpenStack

We have seen the consequences of inconsistency first hand. In some cases, a complete lack of developer mobility has resulted in echo-chambers, entire projects that are completely convinced that their approach is superior to others’. Other projects are effectively code deserts, unable to recruit contributors. Deployments are difficult, feature support is inconsistent, and rather than address the problem and simplify our projects, we’ve instead built layers of abstraction so we can avoid our most grievous mistakes.

We need a consistent OpenStack. If each project takes a slightly different approach to something, it makes subsequent management and support very difficult. To that end, all of our projects should use a consistent set of tools, frameworks, and libraries.

It is time to abandon server-rendered HTML

There are many, many benefits, to building your business on API’s. Easier B2B interoperability, separation of presentation and business logic, as well as the natural varying velocity of API vs. UI development, are all arguments cited as to why you should present some form of API for your business. API’s are a good – if not necessary – part of the competitive landscape.

Historically, however, most web application development still begins with server-rendered HTML. As evidenced by frameworks such as Express, Django, or WordPress, the ability to render data to browser-ready HTML remains a core part of our development process. This is not surprising – in the early days of the internet, most of the computing power was available only on servers, and implementing complex user interface logic onto a browser was both poorly supported and immature.

We have come a long way since then. Standards – be it RFC’s, WhatWG documents, or W3C standards, have evolved to chip away at the need for HTML to be provided by the server. In 2014, the CORS specification broke down the single-origin policy barrier. The maturing of HTML5, and its adoption in modern browsers, has filled in key feature gaps and converted the web browser into an application platform. And, finally, the rise of browser-based application frameworks such as AngularJS and React, provide a much needed toolkit for manipulating HTML directly in the browser.

There remains one argument in favor of server-rendered HTML: Search Engine Optimization (SEO). Web crawlers are notoriously bad at dynamic web pages, and thus a long-standing `best practice` in my industry has been that all public content must be crawlable. There were some bridging technologies – such as the ?_escaped_fragment_ contract recommended by Google, however even those pushed the onus of generating static content to the server.

And then, on October 14th, 2015, Google deprecated their AJAX crawling API. Quote:

“Today, as long as you’re not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.”

Let’s be honest: Google is the gorilla when it comes to web crawling, and it’s only a matter of time before other search engines will follow suit (if they haven’t already). Web apps, be they simple or complex, can now be fully understood by a crawler, including any content provided by an API.

So… why are we rendering HTML on the server? Static content- be it static HTML or a complex AngularJS app – requires no server resources. In fact, you can host your entire application in Amazon’s S3, point a CDN at it, and it’s available globally. The alternative – paying for regionally placed UI servers – is by comparison unjustifiably expensive.

It is time for this technical approach to be retired. Treat your web page like an API client, no different than your mobile or desktop app.