JavaScript Dependency Management in OpenStack
A problem that I’ve been working on the last week has been JS dependency management – driven by npm and bower – inside of OpenStack. To be honest, this problem can be extended to JS dependency management in any application that wants to be packaged within a Linux distribution, as that is the ultimate bar that needs to be met. In this case, however, we’re just going to focus on OpenStack. And, to narrow things down even more, we’re only going to focus on front-end, bower-driven dependencies, used during runtime of the front-end.
To be clear: We are not talking about which tools to use. We are talking about making a javascript/html project’s source code both trustworthy enough for packagers, while providing access to the npm-driven toolchain preferred in this community.
Note: I anticipate updating this post to make more recommendations as I build out Ironic’s Webclient. Stay tuned.
Bower: Commit your dependencies
TL/DR: The ultimate recommendation to the OpenStack community is to use project-appropriate tools to resolve dependencies, but to ultimately commit them to source. For python, you might use something like bower.py. For NPM/Javascript projects, I personally recommend main-bower-files as demonstrated in this gulp file.
Requirement: Builds must be deterministic
Packagers’ customers are banks. Governments. Large corporations. Entities which need to ensure that the software they’re running can be signed and verified, and there are significant dollar values rolled up in SLA’s to ensure this. There’s lots of policies in place for this, some of which seem so draconic as to be beyond unreasonable. If you’re curious, I recommend reading up on PCI compliance. It all makes sense, once you realize that it’s possible to guess a password from the return speed of an error response.
In the world of packaging, this means that builds must be deterministic: If you run the build different times, the output must be exactly the same. If you can’t do that, you can’t md5 or sha1 sum the results for verification, and suddenly the packager is on the hook for the next big security breach.
Fact: Bower is not deterministic
Bower’s pretty neat. It is a registry, rather than a repository, so it only provides the address of where you can get a library, rather than providing the package itself. In the vast majority of cases, this means that bower will point you at a git repository, from which the command line client then extracts the tags as versions. This is pretty awesome, because it means that you can make github host your repository for you.
Yet…. git lets you go back in time and rewrite history. While awesome, this means that bower itself does not provide a deterministic way of resolving a dependency, and therefore cannot be used by packagers. Yes, you can cache bower and the git/svn repositories that it links to. In fact, I wrote a bower-mirror puppet module that will build a server for you that does just that. That does not solve the problem of git being non-deterministic though. As long as a library’s primary source is a git tag, you can’t trust it.
Solution: Use bower anyway
Wait, what? No, I’m serious. Fact is that bower is the de-facto dependency registry for front-end development. We should use it, because it’s an awesome tool. We should also ensure that our builds are deterministic, which means that bower should not be run as part of a build, and should only be used to assist in resolving and committing dependencies.
There is precedent: The NPM documentation itself recommends that you commit all your dependencies, a fact that came out during the SSL Debacle of 2014. Yet even without this recommendation from the JavaScript community itself, there is precedent in OpenStack via the oslo-incubator libraries. Since they are libraries in incubation, they are directly copied and committed into a target project, rather than using pip.
How do you do this? Well, that’s up to you. If you’re a mostly-python project that wants to use the bower registry but is allergic to node, then I’d suggest something like bower.py. If instead you’re using the NPM toolchain, something like the ‘update_dependencies’ target in this gulpfile should work for you.