JavaScript RoadMap for OpenStack Newton

This post contains the current working draft of the OpenStack JavaScript roadmap. It’s a big list, and we need help to land it during the Newton cycle. Overall themes for this cycle are Consistency, Interoperability, and engaging with the JavaScript community at large, all topics which I’ve written about at length. Our end goal is to build the foundations of a JavaScript ecosystem, which permits the creation of entirely custom interfaces.

Note: We are not trying to replace Horizon, we are aiming to help those downstream who need something more than “Vanilla OpenStack”, and thus maintain their own code. The vast majority of modern UI and UX development happens in JavaScript, and there are many use cases that have not yet been met.

OpenStack Projects

These projects are part of the big tent, and will see significant development during the Newton Cycle.

ironic-webclient

The ironic-webclient will release its first version during the Newton cycle. We’re awfully close to having the basic set of features supported, and with some excellent feedback from the OpenStack UX team, will also have a sexy new user interface that’s currently in the review queue. Once this work is complete, we will begin extracting common components into a new project, named…

js-openstacklib

This new project will be incubated as a single, gate-tested JavaScript API client library for the OpenStack API’s. Its audience is software engineers who wish to build their own user interface using modern javascript tools. As we cannot predict downstream use cases, special care will be taken to ensure the project’s release artifacts can eventually support both browser and server based applications.

Philosophically, we will be taking a page from the python-openstackclient book, and avoid creating a new project for each of OpenStack’s services. We can make sure our release artifacts can be used piecemeal, however trying to maintain code consistency across multiple different projects is a hard lesson that others have already learned for us. Let’s not do that again.

Infrastructure Projects

These projects belong to OpenStack’s Infrastructure and/or QA team. They’re used to support the building of JavaScript projects in OpenStack.

js-generator-openstack

Yeoman is JavaScript’s equivalent of cookiecutter, providing a scaffolding engine which can rapidly set up, and maintain, new projects. Creating and maintaining a yeoman generator will be a critical part of engaging with the JavaScript community, and can drive adoption and consistency across OpenStack as well. Furthermore, it is sophisticated enough that it could also support many things that exist in today’s Python toolchain, such as dependency management, and common tooling maintenance.

Development of the yeoman generator will draw in lessons learned from OpenStack’s current UI Projects, including Fuel, StoryBoard, Ironic, Horizon, Refstack, and Health Dashboard, and attempt to converge on common practices across projects.

js-npm-publish-xstatic

This project aims to bridge the gap between our JavaScript projects, and Horizon’s measured migration to AngularJS. We don’t believe in duplicating work, so if it is feasible to publish our libraries in a way that Horizon may consume (via the existing xstatic toolchain), then we certainly should pursue that. The notable difference is that our own projects, such as js-openstacklib, don’t have to go through the repackaging step that our current xstatic packages do; thus, if it is possible for us to publish to npm and to xstatic/pypi at the same time, that would be best.

Xenial Build Nodes

As of two weeks ago, OpenStack’s Infrastructure is running a version of Node.js and npm more recent than what is available on Trusty LTS. Ultimately, we would like to converge this version on Node4 LTS, the release version maintained by the Node foundation. The easiest way to do this is to simply piggyback on Infra’s impending adoption of Xenial build nodes, though some work is required to ensure this transition goes smoothly.

Maintained Projects

The following projects are active and considered ‘complete’, though they will require continuous maintenance throughout the Newton cycle. I’ve included all the needed work that I am aware of, however if there’s something I’ve missed, please feel free to comment.

eslint-config-openstack

eslint has updated to version 2.x, and no more rule bugfixes are being landed in 1.x. eslint-config-openstack will follow in kind, updating itself to use eslint 2.x. We will releases this version as eslint-config-openstack v2.0.0, and continue to track the eslint version numbers from there. Downstream projects are encouraged to adopt this, as it is unlikely that automated dependency updates for JavaScript projects will land this cycle.

NPM Mirrors

We are currently synchronizing all npm packages to our AFS master disks, which should be the final step in getting functional npm mirrors. Some minor tweaking will be required to make them functional, and they will need to be maintained throughout the next cycle. Issues raised in the #openstack-infra channel will be promptly addressed.

This includes work on both the js-openstack-registry-hooks project and the js-afs-blob-store project, which are two custom components we use to drive our mirrors.

oslo_middleware.cors

CORS landed in mitaka, and we will continue to maintain it going forward. In the Newton cycle, we have the following new features planned:

  • Automatic allowed_origin detection from Keystone (zero-config).
  • More consistent use of set_defaults.
  • Configuration maintenance as projects deprecate X-* headers in accordance with RFC 6648.

Stretch Projects

These are projects which we know need to be done, however we simply do not have enough contributors.

Docs

Documentation is important. Usable documentation is even more important. The tricky bit is that OpenStack’s documentation is all python/sphinx based, and we have not yet investigated whether it’s possible to bridge the two languages. If you have time to explore this intersection, we’d be happy to hear your findings.


That concludes it for the Newton Cycle. As you can see, there’s a lot of work to do. Can you help?

JavaScript on the Trailing Edge

The public opinion of the JavaScript community is that it’s fast. We break things, we’re hungry for the latest features, and none of us want to return to the days of slow innovation that ended with the death of IE6. This really isn’t true; there are several core JavaScript projects, such as Angular, JQuery, and React, which have solid governance, and measured release cycles, that would mesh well with OpenStack. It just happens that those projects are surrounded by thousands of smaller ones, run by handfuls of engineers who are volunteering their time.

However, the JavaScript community offers many benefits, from layout frameworks to new user interface paradigms, and OpenStack could easily take advantage of all these. As I’ve pointed out, the user interface needs of a cloud platform vary by user, not by deployment, and it is high time that OpenStack catered to more than just the Operator mind set. There remain some obstacles to this, however they are easily solved:

Backwards Compatibility

The first challenge we face is backwards compatibility. We must balance the rapid adoption of new developments like ES6, with downstream LTS support commitments that can last several years. We must do all this, while also not losing ourselves in a morass of hacks, special cases, shortcuts, and workarounds. This requires common dependency management for all our JavaScript libraries, and we can easily draw on the lessons learned in OpenStack’s requirements project to lead the way.

Complacency

Furthermore, we face a social challenge, that of complacency. The counterargument I most frequently get is “Why not use Horizon”. As my previous post on composable cloud interfaces highlights, Horizon is too narrowly focused. While it does an admirable job at supporting the Operator use case, and provides many ways to extend itself, a brief survey I performed last year revealed that two thirds of downstream Horizon users either maintain full forks of horizon’s source, or are building entirely custom user interfaces. To me, this is stark evidence that horizon falls short of meeting the use cases of all of our OpenStack operators.

Funding

Lastly, we face the rather pedestrian challenge of funding. While I’ve come across broad support for a greater role of JavaScript in OpenStack’s UI development – to the level of squeefun bouncing in a hallway when I mentioned ES6 – it remains a fact of life that those corporate members with the most to gain by the strategic evolution of OpenStack are usually content to let ‘someone else’ do the work, while dedicating their own employees towards more immediate revenue sources.

 

It’s a Catch-22 situation: We cannot prove the viability of JavaScript thick-client UI’s without a functional alternative to horizon, but we cannot get to that alternative without engineers willing – and able – to contribute. Personally, I feel very privileged to be one of a very small number of fully dedicated upstream engineers. To the best of my knowledge, Elizabeth Elwell and I are the only two entirely dedicated towards strategically advancing User Interface development in OpenStack. We are making good progress, however we do not anticipate adoption in the next cycle.

With help, Newton will contain the last pieces we need.

Securely publishing to NPM, the OpenStack way

The following article has been making the rounds, claiming a new worm exploit against npm. First of all, this is not a new exploit, nor is it in any way unique to npm – pip, gem, rpm, and deb have the same issue. Many may not even consider this an exploit at all – it’s a known feature, provided by package repositories, that permit compiling platform-specific bytecode. This is useful if, for instance, your package depends on a c-level library.

The exploit works something like this:

  1. If you are easily authenticated against a package repository, and…
  2. …you install a package which compiles and runs someone else’s code, then…
  3. …an attacker can execute malicious code which can publish itself to your packages…
  4. …which will then subsequently infect anyone who fuzzily matches against your packages’ versions.

This is not news. Mitigation approaches exist. Here’s how we do it in OpenStack:

Step 1: Do not use privileged build environments

Every test, package, or other build command runs on a throwaway jenkins slave that only exists for that test, after which it is deleted. While during test setup the jenkins user begins with passwordless sudo, that privileged is almost always revoked before the tests are run. In short, even if malicious code is downloaded during npm install, it is never executed in an environment that permits a privilege escalation attack.

This approach doesn’t have to be restricted to our Cloud VM’s either. You can do this with docker images, vagrant boxes, you name it.

Step 2: Build an intermediary tarball with `npm pack`

`npm pack` builds a release tarball from a given package, in our case the current project at its release version tag. We do this on the above-mentioned throwaway slave, so that any scripts executed during the construction process cannot access any credentials. After construction, this tarball is uploaded to tarballs.openstack.org, from which anyone can retrieve it.

Step 3: Publish without scripts

OpenStack’s infrastructure contains one jenkins slave that possesses credentials necessary to publish artifacts. Its sole purpose in life is to download a release tarball, and to push that tarball to a package repository. In npm’s case, we execute  `npm publish <tarball> --ignore-scripts`, to ensure that none of the packages’ lifecycle events are accidentally executed, further isolating us from unexpected attacks.

Other security measures

In addition to the above publishing flow, we also have several policies in place intended to ensure that our packages are trustworthy.

    • Only one user owns our npm packages. This prevents other owners from accidentally compromising the package.
    • Only verified, gpg-signed git tags using registered keys will trigger our publish jobs. To easily enable this, add git-sign-tag=true to your global or local .npmrc (Of course, you’ll need to be able to sign a tag).
    • We strongly prefer using strict version matching in our packages, which also has the benefit of making our builds deterministic. The fastest way for you to accomplish this yourself is to commit your shrinkwrap file to version control.
    • We don’t just publish our code via npm; If you’d prefer using a git dependency, or a tarball link, that option is open to you.

It is time to abandon server-rendered HTML

There are many, many benefits, to building your business on API’s. Easier B2B interoperability, separation of presentation and business logic, as well as the natural varying velocity of API vs. UI development, are all arguments cited as to why you should present some form of API for your business. API’s are a good – if not necessary – part of the competitive landscape.

Historically, however, most web application development still begins with server-rendered HTML. As evidenced by frameworks such as Express, Django, or WordPress, the ability to render data to browser-ready HTML remains a core part of our development process. This is not surprising – in the early days of the internet, most of the computing power was available only on servers, and implementing complex user interface logic onto a browser was both poorly supported and immature.

We have come a long way since then. Standards – be it RFC’s, WhatWG documents, or W3C standards, have evolved to chip away at the need for HTML to be provided by the server. In 2014, the CORS specification broke down the single-origin policy barrier. The maturing of HTML5, and its adoption in modern browsers, has filled in key feature gaps and converted the web browser into an application platform. And, finally, the rise of browser-based application frameworks such as AngularJS and React, provide a much needed toolkit for manipulating HTML directly in the browser.

There remains one argument in favor of server-rendered HTML: Search Engine Optimization (SEO). Web crawlers are notoriously bad at dynamic web pages, and thus a long-standing `best practice` in my industry has been that all public content must be crawlable. There were some bridging technologies – such as the ?_escaped_fragment_ contract recommended by Google, however even those pushed the onus of generating static content to the server.

And then, on October 14th, 2015, Google deprecated their AJAX crawling API. Quote:

“Today, as long as you’re not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.”

Let’s be honest: Google is the gorilla when it comes to web crawling, and it’s only a matter of time before other search engines will follow suit (if they haven’t already). Web apps, be they simple or complex, can now be fully understood by a crawler, including any content provided by an API.

So… why are we rendering HTML on the server? Static content- be it static HTML or a complex AngularJS app – requires no server resources. In fact, you can host your entire application in Amazon’s S3, point a CDN at it, and it’s available globally. The alternative – paying for regionally placed UI servers – is by comparison unjustifiably expensive.

It is time for this technical approach to be retired. Treat your web page like an API client, no different than your mobile or desktop app.

 

 

OpenStack JavaScript Mitaka Recap

There aren’t that may people working on the JavaScript ecosystem in OpenStack, however even with that in mind, we’ve made quite a bit of progress in Mitaka.

  1. CORS support in all OpenStack services
    While there are still some extant bug fixes that need to land, those appear to be on schedule for the M3 release. We still need help though, so if you have some time to assist, the relevant launchpad bug is here.
  2. A common JavaScript style guide for OpenStack
    The project is named ‘eslint-config-openstack‘, and I’d like to invite everyone to keep an eye on it. The initial flurry of rule proposals seems to have settled, and at this point I’d like to move to a once-per-cycle release cadence.
  3. Publishing to NPM
    OpenStack’s infrastructure is now capable of publishing packages to NPM.
  4. Mirror Foundations
    While we weren’t able to set up an NPM package mirror for OpenStack’s Infrastructure, we were able to migrate our build mirrors to AFS, while adjusting the folder structure to make more room for other language mirrors. A nice side effect: We removed 20 minutes from devstack gate runs by creating space for pypi wheel mirrors.
  5. Ironic-UI Horizon Plugin
    Version 1 of the ironic horizon plugin has been built, and while it doesn’t have the bells and whistles that you may be expecting, it’s nevertheless a major splash by Elizabeth Elwell as she enters the contributor community. Stay tuned for more updates on this.

Did I forget something? Often it feels like there’s actually been more that I worked on, however my memory isn’t the greatest. Leave an update in the comments, and I’ll add it!

 

JavaScript Dependency Management in OpenStack

A problem that I’ve been working on the last week has been JS dependency management – driven by npm and bower – inside of OpenStack. To be honest, this problem can be extended to JS dependency management in any application that wants to be packaged within a Linux distribution, as that is the ultimate bar that needs to be met. In this case, however, we’re just going to focus on OpenStack. And, to narrow things down even more, we’re only going to focus on front-end, bower-driven dependencies, used during runtime of the front-end.

To be clear: We are not talking about which tools to use. We are talking about making a javascript/html project’s source code both trustworthy enough for packagers, while providing access to the npm-driven toolchain preferred in this community.

Note: I anticipate updating this post to make more recommendations as I build out Ironic’s Webclient. Stay tuned.

Bower: Commit your dependencies

TL/DR: The ultimate recommendation to the OpenStack community is to use project-appropriate tools to resolve dependencies, but to ultimately commit them to source. For python, you might use something like bower.py. For NPM/Javascript projects, I personally recommend main-bower-files as demonstrated in this gulp file.

Requirement: Builds must be deterministic

Packagers’ customers are banks. Governments. Large corporations. Entities which need to ensure that the software they’re running can be signed and verified, and there are significant dollar values rolled up in SLA’s to ensure this. There’s lots of policies in place for this, some of which seem so draconic as to be beyond unreasonable. If you’re curious, I recommend reading up on PCI compliance. It all makes sense, once you realize that it’s possible to guess a password from the return speed of an error response.

In the world of packaging, this means that builds must be deterministic: If you run the build different times, the output must be exactly the same. If you can’t do that, you can’t md5 or sha1 sum the results for verification, and suddenly the packager is on the hook for the next big security breach.

Fact: Bower is not deterministic

Bower’s pretty neat. It is a registry, rather than a repository, so it only provides the address of where you can get a library, rather than providing the package itself. In the vast majority of cases, this means that bower will point you at a git repository, from which the command line client then extracts the tags as versions. This is pretty awesome, because it means that you can make github host your repository for you.

Yet…. git lets you go back in time and rewrite history. While awesome, this means that bower itself does not provide a deterministic way of resolving a dependency, and therefore cannot be used by packagers. Yes, you can cache bower and the git/svn repositories that it links to. In fact, I wrote a bower-mirror puppet module that will build a server for you that does just that. That does not solve the problem of git being non-deterministic though. As long as a library’s primary source is a git tag, you can’t trust it.

Solution: Use bower anyway

Wait, what? No, I’m serious. Fact is that bower is the de-facto dependency registry for front-end development. We should use it, because it’s an awesome tool. We should also ensure that our builds are deterministic, which means that bower should not be run as part of a build, and should only be used to assist in resolving and committing dependencies.

There is precedent: The NPM documentation itself recommends that you commit all your dependencies, a fact that came out during the SSL Debacle of 2014. Yet even without this recommendation from the JavaScript community itself, there is precedent in OpenStack via the oslo-incubator libraries. Since they are libraries in incubation, they are directly copied and committed into a target project, rather than using pip.

How do you do this? Well, that’s up to you. If you’re a mostly-python project that wants to use the bower registry but is allergic to node, then I’d suggest something like bower.py. If instead you’re using the NPM toolchain, something like the ‘update_dependencies’ target in this gulpfile should work for you.

JS-Maven Roundup and Roadmap

To those of you who’ve been paying attention to the original project on GitHub, you’ve probably noticed that there’ve been a lot of changes there recently without any subsequent updates here. I’d like to take a moment to summarize what we’ve been up to, and keep everyone in the loop on where we’re going.

Inevitably as you start into a project you realize the full scope of what you’re trying to do, and as you do the finish line moves farther and farther towards the horizon. Most of the changes made were in respect to that.

js-maven-plugin is now js-maven

I renamed and moved the project because it was rapidly growing beyond the scope of a single maven plugin to include core libraries, tooling, archetypes and reports. Rather than try to hammer all these into one single java project I felt that switching over to a multi-module layout would give me the room I needed to grow. So far this seems to have been a good decision, as I now have the sense that I can (mentally) stretch my arms a little.

No more specific version commitments.

I’m making a concession to my own prioritization style here, which is very much a “Hey, this sounds like an interesting problem to solve” method. Needless to say, this doesn’t really work well with preplanned version releases, as there are some projects that are more fun and some that are less fun, and it resulted in me skipping around a lot. Instead, until I’ve got a few more contributors on the project, I’m simply going to outline a feature set for the various point versions and go after features as they strike me.

I understand that for an enterprise this is not the most optimal approach, so I will make this suggestion: If a particular feature is business critical enough for you to write your own solution, why not contribute it while you’re at it?

Documentation!

I’ve finally gotten around to setting up all the maven:site documentation for the project, and it’s now hosted on github’s pages for the project http://krotscheck.github.com/js-maven/ . Similarly, the README.md on github has been reduced to a quick start (since we know we all prefer copy/paste), as I don’t feel that much more was necessary.

Version Dependencies

We now support Maven 3 and Java 1.7 only. Annoying for some of you, yes, however in keeping with my philosophy of using the latest-and-greatest whenever possible we’re just going to put a line in the sand. That’s not to say we won’t ever support other versions, however those ports will have to come from the community (such as it is).

Licensing

I’m still debating licenses. Right now the project is under the MIT license, though I might move it over to New BSD. This project will never be licensed under the GPL or any similar viral/copyleft license, as those tend to greatly inhibit adoption.

Roadmap Commitments for v0.2.0

If you’ll take a look at the roadmap, you’ll see the list of features which I want to have ready for the next minor version release. Notable additions were the reporting plugins for JSLint and JSDoc, as it didn’t make much sense to me to include those without making them accessible either. Here’s a list of where we stand right now, with some details:

Plugin Documentation [branch- jsdoc-improvments]

Now that I’ve got the maven site reports running, they decided to get all unruly and hurt my feelings by insulting my coding style. The inevitable cleanup phase is in progress, which also includes a bit of judicious refactoring as the actual usability of the plugin gets a little focus.

Cleanup

As this is my first large maven plugin project, I’m still learning my way around. Refactoring and cleanup is necessary – for example, sourceEncoding should be handled via the global parameter and not individually per Mojo.

Maven Reports: JSLint and JSDoc

These are two new features I realized would be necessary as I got the site up and running. I’m not at all familiar how plugins like this get wired into maven as a whole, so they might go fast or they might not.

Common JS Libraries in a repository [branch- jslib-redeploy-plugin]

Having a javascript maven framework isn’t much use if you don’t have any libraries that you can depend on, so I’m writing a plugin that’ll take a URI-styled token string and will redeploy any artifacts found into a maven repository. This is proving tricky as I cannot resolve their own version dependencies easily (jqueryUI -> jquery for instance), however it’s coming along nicely. Once this is complete I’ll be able to provide a few more archetypes for other projects. jQuery, JavascriptMVC, you name it.

Unit Testing

I’m still evaluating the best javascript unit testing framework out there. There are many solutions for “How to test a javascript library in different javascript engines and/or browsers, but nobody seems to have figured out the holy grail yet. This feature might get punted into its own release, if it turns out that the scope is too big.

Building a JSDoc Runner for Maven

The next step in this plugin is to ensure that we can generate documentation. While it may seem like a stretch to do this so early in the process, it forces us to build a few other pieces that we’ll need later.

Our documentation generator will be JSDoc, chosen because it’s fairly mature, has a strong following, and its markup is broadly supported across IDE’s. In fact JSDoc seems to be one of the few broadly adopted standards out there, though to speculate why is beyond the scope of this post. What matters is that it’s a documentation writer written in Javascript that is run via Mozilla’s Rhino runtime. To properly run it we’ll have to resolve the dependency, unpack it into a temporary directory, and then spin up Rhino to execute the code.

I encountered a problem in the way JSDoc 2.4.0 interacts with Rhino 1.7RC3, a bug introduced by two inline function declarations without semicolons in TokenParser.js. While on one side one might argue that this is a bug in Rhino, declaring functions inside of other functions is a big no-no in the javascript world. As a result I’ve had to generate and deploy my own patched version of JSDoc.

Resolving JSDoc

All of the plugin artifacts for a given mojo are easily available as a parameter, thus locating the proper version of JSDoc is fairly simple.

Full source available on github

/**
 * @parameter default-value="${plugin.artifacts}"
 */
private List<Artifact> pluginArtifacts;

public Artifact locate(String groupId, String artifactId) {

	getLog().info("Resolving " + groupId + ":" + artifactId);

	for (Iterator<Artifact> iterator = pluginArtifacts.iterator(); iterator.hasNext();) {
		Artifact artifact = (Artifact) iterator.next();

		if (artifact.getGroupId().equals(groupId) && artifact.getArtifactId().equals(artifactId)) {
			return artifact;
		}
	}

	getLog().error(String.format("Failed to locate %s:%s", groupId, artifactId));
	getLog().error("This probably means that you didn't specify it as a dependency in the pom.xml file");

	throw new RuntimeException(String.format("Failed to locate %s:%s", groupId, artifactId));
}

Unpacking JSDoc

As all jar files are zip files, we can simply write a zip extraction routine to unpack it.

Full source available on github

ZipFile zipFile = new ZipFile(artifact.getFile(), ZipFile.OPEN_READ);
Enumeration zipFileEntries = zipFile.entries();

while (zipFileEntries.hasMoreElements()) {
	ZipEntry entry = (ZipEntry) zipFileEntries.nextElement();

	// Construct the new name
	String currentName = entry.getName();
	File destFile = new File(destinationDirectory, currentName);

	if (entry.isDirectory()) {
		// If we are a directory, create it.
		destFile.mkdirs();
	} else {
		// If we're a file, unzip it.
		BufferedInputStream is = new BufferedInputStream(zipFile.getInputStream(entry));
		int currentByte;

		// establish buffer for writing file
		byte data[] = new byte[BUFFER];

		// write the current file to disk
		FileOutputStream fos = new FileOutputStream(destFile);
		BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);

		// read and write until last byte is encountered
		while ((currentByte = is.read(data, 0, BUFFER)) != -1) {
			dest.write(data, 0, currentByte);
		}
		dest.flush();
		dest.close();
		is.close();
	}
}

Configure JSDoc

At this point we’re back in the drudgery of maven configuration. Since we’d like to keep our configurations relatively separate (in a <jsdoc> sub tag in the mojo configuration), I’ve created a separate JSDocOptions class that can accept all of our options. I’m not going to bother copying it all here as there’s quite a bit of code, but you can see the full source code here.

Running JSDoc via Rhino

Lastly, we have to execute JSDoc, which is done via Rhino. This, again, is fairly straightforward.

Full source available on github

public void run(File inputDirectory, File outputDirectory, JSDocOptions options)
	throws JSDocRunnerException
{
	// Build the string arguments.
	String[] args = options.buildArguments(jsDocRoot, inputDirectory, outputDirectory);

	// Run the javascript engine.
	getLog().info("Building JSDoc...");

	// Create our logging streams
	MojoLoggerOutputStream errorStream = new MojoLoggerOutputStream(LogLevel.ERR);
	Main.setErr(new PrintStream(errorStream));

	MojoLoggerOutputStream outputStream = new MojoLoggerOutputStream(LogLevel.WARN);
	Main.setOut(new PrintStream(outputStream));

	Main.main(args);
	getLog().info("Complete");

}

Building a Javascript Dependency Resolver in Maven

Dependency resolution, to most javascript developers, means adding another <script> include in the header of their html page, and in most cases this is fine. In fact in 95% of any javascript applications you don’t even have to host this file- just include a link to JQuery on Google Libraries and you won’t need to worry.

In the enterprise, however, different concerns start to take over. The project you’re working on might be dependent on an internal library built by another team, which is subsequently dependent on yet another library, and so on and so forth. As those libraries are incrementally updated, managing dependencies becomes a headache. Furthermore, the true power of javascript compilers like Google Closure are realized when all of the code is available at compile time. In fact, both the Simple and Advanced optimization modes require it, as it teaches Closure which code isn’t used and may thus be discarded.

Maven already has a very mature dependency resolution mechanism built into it, which is backed by a large public repository of common libraries. Javascript is still underrepresented in this arena, true, yet hopefully this plugin will eventually remedy that situation.

Choosing Scope

The maven dependency system requires a scope for each individual dependency to determine when it is needed in the build cycle. This is quite handy- you don’t for instance, want to have all the JUnit libraries be part of your final project compile, but you do want them available during the testing phase.

Similarly, Google’s closure compiler can process an external file (dependency) in two ways: Directly, immediately compiled into the final delivery source, and externally, files which are assumed to be self-contained and included in the final product via a different script tag. These two types of dependencies map rather nicely to Maven’s “compile” and “runtime” scope, however compiling a javascript file into a library causes complexity: If two libraries have the same compiled-in dependency, with different versions, perhaps even with different optimization levels, I run the risk of version and feature conflicts.

As a result, all of our dependencies will be treated as externals.

Resolving a dependency graph

The first step to building dependency resolution into my plugin requires constructing the dependency graph. Maven makes this (relatively) easy.

Full source available on github: DependencyResolver.java

private DependencyNode getDependencyGraph(Artifact rootArtifact)
        throws DependencyResolverException {
        try {
                Dependency rootDependency = new Dependency(rootArtifact, "compile");
                CollectRequest collectRequest = new CollectRequest(rootDependency,projectRepos);

                // Collect all the nodes
                CollectResult collectResult = repoSystem.collectDependencies(repoSession, collectRequest);
                return collectResult.getRoot();
        } catch (DependencyCollectionException e) {
                throw new DependencyResolverException("Cannot build project dependency graph", e);
        }
}

Filtering dependencies by scope and extension

Now that I have a graph of dependency relationships, I should now visit each node and only collect those nodes that match our required scopes. Since I’ll also use this extension for resolving jsunit, envjs or others, it behooves me to abstract it.

Full source available on github: DependencyResolver.java

public List<Artifact> resolve(String scope, String extension)
                throws DependencyResolverException {

        getLog().debug("Resolving dependencies: [scope:" + scope + "] [extension:"  + extension + "]");
        // Construct the dependency tree of our project.
        Artifact rootArtifact = new DefaultArtifact(project.getGroupId(),
                project.getArtifactId(), project.getPackaging(),
                project.getVersion());
        DependencyNode rootNode = this.getDependencyGraph(rootArtifact);

        // Gather dependency nodes in post order
        PostorderNodeListGenerator postOrderVisitor = new PostorderNodeListGenerator();

        // Filter out the root node.
        DependencyVisitor visitor = new FilteringDependencyVisitor(
                postOrderVisitor, new PatternExclusionsDependencyFilter(
                        rootArtifact.toString()));

        // If a scope is configured, add a scope filter.
        if (null != scope) {
                visitor = new FilteringDependencyVisitor(visitor,
                                new ScopeDependencyFilter(scope));
        }

        // If a type is configured, add a pattern exclusion filter
        if (null != extension) {
                String patternFilter = "*:*:" + extension + ":*";
                visitor = new FilteringDependencyVisitor(visitor,
                                new PatternInclusionsDependencyFilter(patternFilter));
        }

        // Run the collection.
        rootNode.accept(visitor);
        return postOrderVisitor.getArtifacts(true);
}

Resolving dependencies

As it turns out, maven dependencies may, or may not, be resolved. In practice this means that the system won’t bother to download the file until it’s actually needed, so we have to do it manually. The new Aether Dependency API in Maven3 means that artifacts are immutable, so we have to make a copy of each.

Full source available on github: DependencyResolver.java

        List<Artifact> unresolvedArtifacts = postOrderVisitor.getArtifacts(true);
        List<Artifact> resolvedArtifacts = new ArrayList<Artifact>();

        for (Iterator<Artifact> iterator = unresolvedArtifacts.iterator(); iterator.hasNext();) {
                Artifact artifact = iterator.next();
                ArtifactRequest artifactRequest = new ArtifactRequest(artifact,projectRepos, null);
                ArtifactResult artifactResult = repoSystem.resolveArtifact(repoSession, artifactRequest);
                getLog().debug(" + " + artifactResult.getArtifact().toString());
                resolvedArtifacts.add(artifactResult.getArtifact());
        }
        return resolvedArtifacts;

Including dependencies in our compiler

With a list of resolved dependency artifacts in hand, I can now include them in the compile method from the last post.

Full source available on github: JSLibraryCompilerMojo.java

        // Generate any externals that we need.
        List<JSSourceFile> externs = new ArrayList<JSSourceFile>();
        getLog().info("Resolving External Includes...");
        externs.addAll(resolveDependencies("compile","js"));

        // Generate the list of source files to include.
        List<JSSourceFile> source = new ArrayList<JSSourceFile>();
        getLog().info("Resolving Compile Includes...");
        source.addAll(generateJavascriptSourceList());

And there we are! We are now resolving dependencies from other javascript libraries that live in your, or someone else’s, maven repository.

Designing a Javascript Maven Plugin

As mentioned in my previous post, I believe that the tooling for enterprise grade javascript applications is simply not there yet. What is needed is a set of tools, preferably open source, that are framework agnostic and that fit into established development practice. In this and the subsequent posts, I will walk you through the process of building just this toolset using the Maven as a framework to glue it all together.

Overall Philosophy

First and foremost, the toolset must be independent of the code that is written. We simply cannot have a compiler that will only work for JQuery, or a testing harness that will only work if the test artifacts are composited via Rake.

Furthermore, I would like to adhere to Java conventions as much as possible, by which I mean JSDoc, JUnit-style unit tests and so forth. Java is syntactically very close to Javascript and Java’s strong prevalence in the enterprise space will reduce engineer friction.

The project artifacts will be packaged in as close to a native file type as possible. This becomes tricky: For something like a Javascript Library, a simple minified .js file will do. For a full web application with images, media, html and more this will be trickier as the nature of the server will define one packaging type over another: Jetty/Tomcat will prefer a .jar file, Apache will serve up an uncompressed archive, and so forth.

When possible, we’re going to take advantage of the latest and greatest: CSS compilers, for instance.

Why choose maven?

Maven is an obvious choice for various reasons. Firstly, it enjoys broad IDE integration. Secondly, it has an extremely powerful plugin framework that allows a user to use entire groups of plugins or focus on only the ones they really need. Thirdly, its dependency resolution mechanism is very powerful and very portable, which makes your builds extremely easy to bring to other platforms. Fourthly, it is open source and well supported by the community. Lastly, it enjoys broad support among continuous integration frameworks. All of these together make it a no-brainer to use as the core engine for our javascript toolset.

What workflow steps are needed?

Since maven defines a specific artifact lifecycle, we should start there and decide what we need our plugin to do. After starting to write this for a little bit however I realized that trying to declare it all up front is a significant undertaking, and should be split up by broader concepts:

  1. Compilation
  2. Optimization
  3. Unit Testing
  4. Functional Testing
  5. Packaging and Publishing
  6. Dependency Resolution
  7. Debug Runtime
  8. Headless Runtime
  9. PMD/Linting
  10. Documentation
  11. Reporting

Should we create an Archetype?

The short answer: Yes. The long answer: Archetypes are absolutely critical to drive adoption, examples and acceptance, and are a very useful speed-of-workflow tool. Given the many different potential framework projects out there however, we’re going to have to create many of these; One for JQuery, one for Mootools, one for Backbone… you get the idea. So how about we worry about that once we get more pieces of the compiler in place.

Where will the source live?

I’m glad you asked! I’ll be hosting the project on GitHub. Please feel free to contribute!

Where will the maven repository be?

This is as yet undecided. For now I recommend you clone the project and install it locally. Once there’s enough functionality on the project to warrant a full point release, we’ll revisit this.