Tag: npm

OpenStack Infra now uses Node.js v4 and npm v2

OpenStack’s Infrastructure is now running all of its NPM and NodeJS-based test jobs using the newer NodeJS v4, and npm 2.15. That’s pretty awesome, given that previously we were running on 0.10.25. ES6, anyone?

LTS is important

Here in OpenStack we try to stick as closely as possible to LTS packages. We do this for multiple reasons, chief of which is that many of our customers have governance and compliance constraints that prevent them from installing arbitrary bits on their hardware. Furthermore, up until recently, the only feasible test platform was Ubuntu Trusty (CentOS7 and RHEL are recent additions), which forced us to rely on NodeJS v0.10.25 and npm v 1.3.10. Neither of these are supported by the Node Foundation or npm inc., and security backports are the responsibility of the distro packaging team.

Vulnerable to Upstream Changes

These out-of-date versions leave us vulnerable to upstream changes, and last week we encountered this issue: npm upgraded the registry to provide gzipped content, something which our older npm client did not understand. The fix was slow in coming (at no fault of npm inc’s engineers), and we were left unable to access registry packages in a reliable way, preventing us from releasing some components of Mitaka… during release week.

When npm breaks, what do we do?

We needed two things- a quick fix, and a future strategy. If possible, both at the same time. Sadly, we couldn’t just update npm; while there’s precedent (we run a non-distro version of pip), the older npm client could not reach the registry to download the packages necessary to update itself. A different approach was needed.

Last summer, the Node Foundation joined the Linux Foundation, and announced plans for an LTS release schedule which was realized in October. Shortly afterwards, linux distro packages began appearing for those versions, and they have since graduated in both Debian and Ubuntu. While neither of these are yet available to us (Xenial Xerus releases later this month), it nevertheless gave us a clear path forward:

1. Upgrade to Node4 via the NodeSource package repository

In order to unblock our builds, we upgraded all of our jenkins slaves to use the NodeSource repository’s version of NodeJS 4, which ships with npm 2.15. While not precisely the LTS versions, they were close enough to solve the issue we were encountering. This would give us a backwards-compatible solution for all of our trusty nodes, while paving the way to the adoption of xenial.

This work has been completed, and our gate is happy again. Woohoo!

2. Switch to Xenial as quickly as feasible

We’ve already been testing xenial nodes in OpenStack’s gate, and anticipate officially making those nodes available during the Newton cycle. Once those become available, we’ll start moving all of our javascript jobs to those nodes instead.

3. Build a static mirror on top of OpenStack’s unified mirror infrastructure

By hosting our own package mirror, we isolate ourselves from many problems from the upstream package registry. There have already been plans to build this, and the first major components of it (Unified Mirrors) have already landed. The only thing these recent problems (as well as similar problems like leftpad) have done, is raised the urgency.

 

Stay tuned for updates

I’ll be posting updates on this effort to this blog, which is syndicated to OpenStack planet. Stay tuned, and let me know if you want to help! My IRC handle is @krotscheck, and I’m always online on FreeNode’s #openstack-infra channel.

Securely publishing to NPM, the OpenStack way

The following article has been making the rounds, claiming a new worm exploit against npm. First of all, this is not a new exploit, nor is it in any way unique to npm – pip, gem, rpm, and deb have the same issue. Many may not even consider this an exploit at all – it’s a known feature, provided by package repositories, that permit compiling platform-specific bytecode. This is useful if, for instance, your package depends on a c-level library.

The exploit works something like this:

  1. If you are easily authenticated against a package repository, and…
  2. …you install a package which compiles and runs someone else’s code, then…
  3. …an attacker can execute malicious code which can publish itself to your packages…
  4. …which will then subsequently infect anyone who fuzzily matches against your packages’ versions.

This is not news. Mitigation approaches exist. Here’s how we do it in OpenStack:

Step 1: Do not use privileged build environments

Every test, package, or other build command runs on a throwaway jenkins slave that only exists for that test, after which it is deleted. While during test setup the jenkins user begins with passwordless sudo, that privileged is almost always revoked before the tests are run. In short, even if malicious code is downloaded during npm install, it is never executed in an environment that permits a privilege escalation attack.

This approach doesn’t have to be restricted to our Cloud VM’s either. You can do this with docker images, vagrant boxes, you name it.

Step 2: Build an intermediary tarball with `npm pack`

`npm pack` builds a release tarball from a given package, in our case the current project at its release version tag. We do this on the above-mentioned throwaway slave, so that any scripts executed during the construction process cannot access any credentials. After construction, this tarball is uploaded to tarballs.openstack.org, from which anyone can retrieve it.

Step 3: Publish without scripts

OpenStack’s infrastructure contains one jenkins slave that possesses credentials necessary to publish artifacts. Its sole purpose in life is to download a release tarball, and to push that tarball to a package repository. In npm’s case, we execute  `npm publish <tarball> --ignore-scripts`, to ensure that none of the packages’ lifecycle events are accidentally executed, further isolating us from unexpected attacks.

Other security measures

In addition to the above publishing flow, we also have several policies in place intended to ensure that our packages are trustworthy.

    • Only one user owns our npm packages. This prevents other owners from accidentally compromising the package.
    • Only verified, gpg-signed git tags using registered keys will trigger our publish jobs. To easily enable this, add git-sign-tag=true to your global or local .npmrc (Of course, you’ll need to be able to sign a tag).
    • We strongly prefer using strict version matching in our packages, which also has the benefit of making our builds deterministic. The fastest way for you to accomplish this yourself is to commit your shrinkwrap file to version control.
    • We don’t just publish our code via npm; If you’d prefer using a git dependency, or a tarball link, that option is open to you.