OpenStack’s Infrastructure is now running all of its NPM and NodeJS-based test jobs using the newer NodeJS v4, and npm 2.15. That’s pretty awesome, given that previously we were running on 0.10.25. ES6, anyone?
LTS is important
Here in OpenStack we try to stick as closely as possible to LTS packages. We do this for multiple reasons, chief of which is that many of our customers have governance and compliance constraints that prevent them from installing arbitrary bits on their hardware. Furthermore, up until recently, the only feasible test platform was Ubuntu Trusty (CentOS7 and RHEL are recent additions), which forced us to rely on NodeJS v0.10.25 and npm v 1.3.10. Neither of these are supported by the Node Foundation or npm inc., and security backports are the responsibility of the distro packaging team.
Vulnerable to Upstream Changes
These out-of-date versions leave us vulnerable to upstream changes, and last week we encountered this issue: npm upgraded the registry to provide gzipped content, something which our older npm client did not understand. The fix was slow in coming (at no fault of npm inc’s engineers), and we were left unable to access registry packages in a reliable way, preventing us from releasing some components of Mitaka… during release week.
When npm breaks, what do we do?
We needed two things- a quick fix, and a future strategy. If possible, both at the same time. Sadly, we couldn’t just update npm; while there’s precedent (we run a non-distro version of pip), the older npm client could not reach the registry to download the packages necessary to update itself. A different approach was needed.
Last summer, the Node Foundation joined the Linux Foundation, and announced plans for an LTS release schedule which was realized in October. Shortly afterwards, linux distro packages began appearing for those versions, and they have since graduated in both Debian and Ubuntu. While neither of these are yet available to us (Xenial Xerus releases later this month), it nevertheless gave us a clear path forward:
1. Upgrade to Node4 via the NodeSource package repository
In order to unblock our builds, we upgraded all of our jenkins slaves to use the NodeSource repository’s version of NodeJS 4, which ships with npm 2.15. While not precisely the LTS versions, they were close enough to solve the issue we were encountering. This would give us a backwards-compatible solution for all of our trusty nodes, while paving the way to the adoption of xenial.
This work has been completed, and our gate is happy again. Woohoo!
2. Switch to Xenial as quickly as feasible
We’ve already been testing xenial nodes in OpenStack’s gate, and anticipate officially making those nodes available during the Newton cycle. Once those become available, we’ll start moving all of our javascript jobs to those nodes instead.
3. Build a static mirror on top of OpenStack’s unified mirror infrastructure
By hosting our own package mirror, we isolate ourselves from many problems from the upstream package registry. There have already been plans to build this, and the first major components of it (Unified Mirrors) have already landed. The only thing these recent problems (as well as similar problems like leftpad) have done, is raised the urgency.
Stay tuned for updates
I’ll be posting updates on this effort to this blog, which is syndicated to OpenStack planet. Stay tuned, and let me know if you want to help! My IRC handle is @krotscheck, and I’m always online on FreeNode’s #openstack-infra channel.