We need a consistent OpenStack

The following is a table of some basic implementation details in OpenStack’s Mitaka API projects. It isn’t intended to shame anyone; it is intended to highlight tool and framework fragmentation in OpenStack. Here’s the data, the article follows below.

Integrated Release APIs

Project
ceilometer PasteDeploy pecan,wsme py27,py34 global-requirements oslo-generate-config
cinder PasteDeploy routes py27 global-requirements oslo-generate-config
glance PasteDeploy routes,wsme py27,py34 global-requirements oslo-generate-config
heat PasteDeploy routes py27,py34 global-requirements oslo-generate-config
ironic pecan,wsme py27,py34 global-requirements
keystone PasteDeploy routes py27,py34 global-requirements oslo-generate-config
neutron PasteDeploy pecan py27,py34 global-requirements oslo-generate-config
nova PasteDeploy routes py27,py34 global-requirements oslo-generate-config
sahara flask py27,py34 global-requirements oslo-generate-config
swift ? py27,py34
trove PasteDeploy routes py27,py34 global-requirements

Supporting API Projects

Project
aodh PasteDeploy pecan,wsme py27,py34 oslo-generate-config
barbican PasteDeploy pecan py27,py34 global-requirements
cloudkitty PasteDeploy pecan,wsme py27,py34 oslo-generate-config
congress PasteDeploy routes py27,py34 global-requirements oslo-generate-config
cue pecan,wsme py27,py34 global-requirements oslo-generate-config
designate PasteDeploy flask,pecan py27,py34 global-requirements
freezer falcon py27,py34 global-requirements
fuel web.py py27,py34
kite pecan,wsme py27,py34
magnum pecan,wsme py27,py34 global-requirements oslo-generate-config
manila PasteDeploy routes py27,py34 global-requirements oslo-generate-config
mistral pecan,wsme py27,py34 global-requirements oslo-generate-config
monasca-api falcon py27
monasca-log-api falcon py27
murano PasteDeploy routes py27 global-requirements oslo-generate-config
searchlight PasteDeploy routes,wsme py27,py34 global-requirements oslo-generate-config
senlin PasteDeploy routes py27,py34 global-requirements oslo-generate-config
solum pecan,wsme py27,py34 oslo-generate-config
tacker PasteDeploy routes py27,py34 global-requirements
zaqar falcon py27,py34 global-requirements oslo-generate-config

Just scratching the surface

The table above only scratches the surface of OpenStack’s tool fragmentation, as it only focuses on frameworks and configuration in API projects. It does not address other inconsistencies, such as supported image types, preferred messaging layers, testing harnesses, oslo library adoption, or a variety of other items.

I’ve already spoken about Cognitive Load and OpenStack, how the variability in our projects can trick your brain and make you less effective. Furthermore, we’ve seen a lot of discussion on how we should Choose Boring Technology, as well as hallway discussions about how OpenStack should be more opinionated in its deployments. In fact, things have gotten so bad that the shade project was created – a library whose express intent is to hide all the differences in deployed OpenStack clouds.

Variability is bad for OpenStack

The lack of consistency across OpenStack is harming us, in very specific ways.

Contributing to multiple projects is hard

Nobody wants to climb multiple learning curves. Knowledge from one project directly transfers to another if the frameworks are similar enough, reducing this learning curve. In short, differences in projects create barriers to cross-project fertilization and contribution, and one way to chip away at those differences is to keep them as similar as possible.

Supporting OpenStack’s dependencies is hard

As an open source community, there is a strong ethos of helping support any projects that we depend on. Yet, how do we pick which upstream project to help fix? If all projects were consistent in their use of, say, WSME, there would be a far larger pool of talent invested in success, and discussions like this one would not happen as frequently (Note: I’m not necessarily advocating WSME here – it merely provides a very useful example).

Maintaining feature consistency is hard

There are many features which our various projects should all support. Simple things, like consistent search query parameters, consistent API version negotiation, consistent custom HTTP Header names – basically anything cross-project or from the API working group.

I have personal experience with this: In Mitaka I was able to land CORS support in most of OpenStack’s API’s. Of the 23 projects that I contributed to, most required that I learn project-specific approaches to governance, launchpad usage, testing harnesses, commit-message constraints, folder structure, and more. The entire experience only taught me one thing: Trying to bring a common feature to OpenStack is something I never want to do again.

Deploying/Running OpenStack is hard

Without an opinionated OpenStack install (down to the supporting services), the chances that someone has run into the same problem as you, drops significantly. Features which rely on service abstraction (messaging for instance) depend on layers of driver abstractions which add more points of failure, and often have to provide workarounds for features supported in one service, but not in another (assuming they don’t give up on that feature entirely).

Portability between clouds is hard

To quote Monty Taylor: “The existence of shade is a bug”. It turns out that OpenStack’s implied portability promise is pretty much a lie, and you will spend a significant amount of effort figuring out how this OpenStack happens to differ from That OpenStack.

We need a consistent OpenStack

We have seen the consequences of inconsistency first hand. In some cases, a complete lack of developer mobility has resulted in echo-chambers, entire projects that are completely convinced that their approach is superior to others’. Other projects are effectively code deserts, unable to recruit contributors. Deployments are difficult, feature support is inconsistent, and rather than address the problem and simplify our projects, we’ve instead built layers of abstraction so we can avoid our most grievous mistakes.

We need a consistent OpenStack. If each project takes a slightly different approach to something, it makes subsequent management and support very difficult. To that end, all of our projects should use a consistent set of tools, frameworks, and libraries.

Breaking down a Client Application

With so little time to blog over the past few months, I took a few moments to really think about why I’d fallen out of the habit. I believe (personally) that much of this is because my writing time now goes into guiding and educating my own development team, and once I realized that I saw no reason to exclude all of you from any of these emails either. So yes, you should expect the content to start picking up again, but the topics will be focused around applied software development rather than the marketing and retail topics I spoke about while I was still at Resource.

This particular email was written to demystify the difference (as far as I’m concerned) of all the different parts of a client application. However rather than a heavy discussion on methods and algorithms, what I present here are actually the higher level concepts that govern the structure of an application. I tried to order them in a fairly logical fashion, starting at the “front” of the application (what you see) and moving towards the “back” (the bits that the user should never see).

But first, I feel it is important to understand the difference between business logic and view logic:

  • Business logic manages data and enforces business rules. It cleans, saves, reads, compares, and makes sure that the data models are properly updated. It is also important to make the distinction between server-side business logic and client business logic. That which lives on the server is usually focused more on business rules, while that which lives in the client usually cares more about server interaction and data manipulation.
  • View logic manages how data is displayed. It manages state, formatters, strings and view indexes, and doesn’t really care about how that data got to where it is, it only cares about how it should be displayed within this particular view.

Having said that, here are the major components of a client application. Caveat, this might not mean much to you Javascript guys.

Skin

A skin acts to describe the containers, colors and graphics that a control or view use. It can change its appearance based on a skin state (not to be confused with view state), and can refer to styles set by the component which it is skinning. It should not have any business or data logic whatsoever.

Control

A Control is the lowest level of functional components which comprise an application. These are our buttons, labels, data grids, view containers and so forth, and they are characterized by the fact that the data they display is not specialized, nor do they contain any real internal business logic at all. They know how to interact with a generic type and dispatch events when a particular action occurs.

View

A view is a collection of controls, containers, formatters, skins and subviews, all connected via a presentation model. It acts to collect and arrange elements rather than figuring out what each piece is supposed to do. As such, Views should contain little to no code at all, and should be managed by states and properties.

Item Renderer

An Item renderer is a very specialized kind of view, which deals with the display of a single piece of data in a collection. As a result, it is the only view which should be dependent on an external piece of data – one that is injected into a property of that ItemRenderer from a parent (the data property). Item renderers may contain Presentation MOdels to help them manage user gestures, but rarely make use of a data model as this is already provided.

Presentation Model

It is the job of the presentation model to act as a mediator between the view and the rest of the application. This does not mean that it should manage data, nor necessarily contain it. Instead, its job is to expose all the methods and data which are being used by the View, which includes models (bindable), states, actions which describe user methods and (in some cases) validators.

Data Model

The data model is a dumb data container. It should contain little-to-no logic on its own, and instead allow the tasks to clean, sort and manipulate anything that is assigned to the model. Thinking of it differently: The Data Model is the authoritative source of any data used in the application. All data starts here and ends here. Views get their data from the Data Model via binding. Tasks retrieve model instances. Presentation models make data available by exposing a public instance of a data model.

Task

Tasks are where our “business” logic resides, in the most generic and granular way possible. As an example, the act of loading a user might require reading the user object, retrieving their preferences, and initializing the application locale. Rather than try to handle this all in one task, it is better to use a SequenceTask or ParalellTask to collect the three different actions. Why? Because your preferences screen might need to reload user preferences after they’ve been saved, but you don’t want to refresh the entire user.

Tasks, much like presentation models, are able to collect any data and utilities they need to achieve what they need to achieve, however when they are done they need to let go of those items and dispose of themselves properly. The important thing to note here is that any business logic that can be put into the service layer should live in the service layer; We don’t want to expose the guts of our business rules to the world, and servers are much easier to protect from questionable data manipulation.

Delegate

A delegate is our service proxy, and acts as a channel back and forth from the database. It allows us access to the business logic that lives on the server in a reliable and consistent way. In most cases, each delegate method will have one task that wraps it and makes sure that the data is properly cleaned and presented.

Utility

Utilities are classes that provide common data manipulation routines that are global to all aspects of the application. A good utility, for instance, would be a method that handles date conversions from GMT to Local and back. A less desirable utility method would be one that formats a piece of data for a specific view, as this should be handled within the view itself.

Libraries

Libraries are the catchall for everything that was left out, and it includes pieces of highly specialized logic that don’t really fit neatly elsewhere. Good examples of this are video encoding libraries, classes that manage encryption, filesystem manipulation and other tasks like that.

VO

A VO, or Value Object (also known as DO/Data Object) is the raw data which we manipulate. It describes a particular Domain Object and acts as a package that we pass from method to model to view. They only contain data related to that same object, and should not contain any formatting logic and only limited validation logic (is-not-zero checks and so forth). Formatting should be handled by the view, validation should be handled by tasks or utilities.