Catch-Up

It’s been an inexcusably long time since my last post, so I thought I would write about some of what I have been doing instead of blogging.

Reading and writing about Backbone.js

I have been asked to be a tech reviewer for Addy Osmani‘s new book “Developing Backbone.js Applications“. Its a great introduction to Backbone and also covers a lot of detail about using it for real-world applications. Addy has pulled together a great little team of volunteers to review and tweak the book and it really shows. It’s Creative Commons so you can read the book for free – or indeed contribute – by heading over to Github.

Getting paid for Open Source

I was checking out an open issue on Twitter’s Bower package manager (mentioned in my last post) and was playing around with a patch for it. Bower is a command-line Node application and uses EventEmitters out the wazoo with no Promises. I was a little out of my comfort zone.

I was ready to let someone who knew what they were doing fix it when I saw a post saying that there was a bounty on the issue. I’d love to believe that my open source work is entirely altruistic but the thought of getting paid made the learning curve look a bit less daunting. What can I say, I’m a mercenary!

Anyway, I submitted a PR which was accepted after just a couple of tweaks and BountySource released the money within a few days. It was very painless and I was $180 richer (~10% taken in PayPal fees and the BountySource cut). I probably made about $10 an hour on it – more than US minimum wage – but it was a very large bounty compared to most of the others currently posted. I don’t think anyone will be making a living off Bounty Source for a while yet.

Getting involved with JavaScript development tools

2013 will be the year of JavaScript tooling. Grunt, Bower and Yeoman are together looking to raise the baseline for front-end development and no doubt there are others out there too.

Before jQuery became popular people were writing their own XHR wrappers and DOM manipulation code. It was tricky but that was just the way it was. Now no one in their right mind starts a basic web project without including jQuery or something like it. I think the same thing will happen around tooling. I believe that in a year or two it will be almost unthinkable not to have a build step in your process, or to start a simple project from scratch instead of using a bootstrapping tool.

With that in mind I have been digging deep into how Grunt, Bower and Yeoman work so that I can start to contribute. Expect to hear more about these projects from me in the future.

The State of Javascript Package Management

One notable thing about JavaScript in the browser compared to other popular languages is the almost lack of a standard library. Even some of the most common tasks can require a fair chunk of code. You can write it all yourself, but crafting cross-browser compatible code remains a minefield. If you don’t want to do it yourself, what can you do?

The “traditional” way in JavaScript was to copy dubious fragments of code from forums and Q and A sites into your project and hope for the best. Then, later, came the big libraries. Now if you want a cross-browser method of changing the colour of all of your links you can download an almost 10,000 line library file that included, somewhere in the middle, the functionality that you wanted. And you could manually download a fresh copy of a probably slightly different version each time you started a new project.

Clearly, this situation is not ideal, which is why we have seen several package managers spring up.

What is a package manager anyway?

I guess before we go any further we should talk about what a package manager is meant to achieve in the general case. Most package management systems are trying to solve a set of related problems.

The first, and most obvious, is to provide a reliable, standard mechanism for installing software created by third-parties. By specifying a convention (or possibly some very strict rules) about the structure of the packages the end-user only has to learn how the package manager works, not the specific installation procedure of the package they are trying to install. Things like where the software is installed, how it is built or compiled, or how it integrates with the rest of the system are all taken care of by the package manager.

Secondly, package managers provide dependency management. If package foo requires libraries bar and baz in order to function then the package manager will know to install bar and baz as well when asked to install foo. Most  can also deal with specific version requirements for dependencies, attempting to satisfy the tree of dependencies in such a way that compatible versions of packages are installed together.

Another common feature is to allow you to keep up to date with new versions of packages as they are released. In some cases you don’t want to upgrade – you don’t need those new features, so why risk breaking your project? Other times you want the latest and greatest so that you have all of the exciting features or essential bug fixes. A good package manager allows you to choose your own path.

A package manager should also allow you to discover new packages. If you want to solve a problem that you are pretty sure has been solved before, you can jump into your package manager’s search and see a list of possible solutions to check out.

The last features we’ll talk about here are for the people who create the packages in the first place. Some, but certainly not all, package managers will come with tools to help you create the package ready for distribution, and all of these systems need a way for the package author to put their work somewhere it can be discovered and used by the community.

A JavaScript package management tour

There are currently six contenders in the JavaScript package management arena – npm, bower, Ender, volo, component and jam. npm comes bundled with recent versions of Node.js, while the others are installable via npm with npm install <package> -g where the package name is bower, ender, volo, component or jamjs.

npm

According to the FAQ, npm does not stand for Node Package Manager, though it is certainly a good description. The npm repository contains many JavaScript libraries and quite a few command line tools. There are packages available through npm to be used with browser based projects but most are specific to Node on the server. npm is important for browser development anyway, though, as all of the other browser-oriented tools mentioned below are themselves npm packages.

There are two ways to install packages with npm – global and local. Usually you would install tools you want to use from the command line globally and libraries you want to require() locally. Packages are installed with the npm install command. When you locally install a package npm will place the package files in a node_modules/<package> folder in your project route. Packages installed this way will be accessible in your node programs with require(). Globally installed command line tools will usually be put somewhere in your PATH (like /usr/local/bin on Linux).

The npm repository contains almost 20,000 packages which can be searched either on the command line with npm search or via the website. An npm module is defined by a file named package.json in the project root. This file has details like the name, description and version, which file is loaded when you require() the package, and dependencies. Dependencies are split into “dependencies” and “devDependencies” so that things that are only needed to build or test your package when developing it wont be downloaded by your end users.

When your package is ready for public consumption you can use npm publish to push it to the repository. Names are taken on a first-come, first-served basis. Only someone registered as an owner of a package can upload new versions but this is easily managed with the command line tools if you wish to add someone or transfer ownership. You need to have an npm account to publish packages.

bower

bower was created by Twitter for use on their internal web projects. A bower package can contain any assets a website might need, such as CSS, JavaScript or images.

This is the simplest of the tools discussed here as it provides a method for downloading packages and their dependencies and no more. This also makes it the most liberal, as any existing package can be put into the bower repository without modification and without adding constraints on the user.

The bower repository has 698 components which can be searched (by name only) using bower search or on a third-party search site. With bower a ‘package’ is actually a git repository, so the bower repository is only a list mapping names to public git endpoint urls. bower install <name> fetches the git repository into a components/<name> directory within the project root. bower has zero security or identity management. Anyone can publish a package on bower with bower register <name> <git endpoint>. Repository management is manual, which is to say that you have to message one of the admins if you want to remove, rename or re-register a component.

The package itself is defined by a file called component.json in the root of the git repository. Only the name, version and dependencies fields actually do anything. When retrieving packages from git endpoints bower uses tag names in the repository as valid version identifiers, so you must tag at least one revision before your package can be consumed.

Ender

Not just a package manager, Ender actually builds you a customised library to use from small, modular packages. The philosophy behind Ender is to give you only the code that you actually need for your project, and to do so in such a way that a module could be replaced by something roughly equivalent with the minimum of fuss.

To start a project using Ender you use ender build <name>[, <name>, …] to create a custom library using the given packages. The created library exposes a global variable $ with properties defined by the component packages. Once your library has been started you use ender add and ender remove to manage the components that it provides. The library is written into ender.js and ender.min.js.

Ender piggybacks off of the npm registry, with each Ender package being an npm package. To signify that an npm module is compatible you add the ‘ender’ keyword to package.json. ender search will search npm, giving higher weight to packages with the keyword. You can see all packages marked for Ender using the npm keyword search on the website. There are currently 212 registered.

As well as adding ‘ender’ to your keyword list you can also optionally add an ender property to your package.json file that specifies a ‘bridge’ file which allows you to tightly integrate with the Ender API. Otherwise, creating and registering packages is the same as for npm.

volo

volo can be used much like bower to simply install the contents of a package, but it also includes a project automation/build system and can automatically turn installed packages into AMD modules, amongst other things.

You can bootstrap a project by using volo create <project name> which will create a directory called ‘<project name>’ containing the default template, though you can specify another template to use. You don’t have to use a template to use volo, however. If you use volo add <name> to add a package to your project then volo will use a set of rules to determine where to extract the package.

volo actually uses Github as its backend – a package is a Github repository and volo add will use the Github search API to find packages if you only provide a name or keyword. You can specify a full repository name with volo add <username>/<project name>, e.g. volo add jquery/jquery.

There is no special file for defining a package for volo, and publishing is simply a matter of creating the repository on Github. There is, however, a set of guidelines that will make your project easier to find and consume. Because every browser-oriented JavaScript repository on Github is technically a volo package, volo almost certainly has the highest available package count though exactly how many is unknown.

component

component is based on CommonJS style modules (the module style that Node uses) so like with Ender you must commit your whole project to this style. You start a project with component create <project name>  which will ask you several questions before creating a new directory under the current path with the given name. From inside the project root you can use component install <name> to install dependencies. You then need to use component build to create the file that you will load in the browser.

Like volo, component packages are Github repositories so dependency names are in the format <username>/<project name>. You can “publish” your package by adding the details of your repository to the wiki. You specify the details of your package in a component.json which is not compatible with the bower config file with the same name.

You can search the available components with component search or view the list on the wiki. The list currently contains 431 entries.

Jam

Jam is a tool for managing AMD packages only, and comes with a modified version of require.js, including a “compiler” based on the r.js tool.

Packages are installed with jam install <name> and get put in jam/<name> by default. To actually consume the packages, though, you use the modified require.js that comes with Jam. This file is updated when you add and remove packages to your application.

Jam has its own repository and you need an account to manage packages. The repository contains 279 packages which can be searched with the web interface.or with jam search. The sort order of the results is a little strange though; when searching for ‘jquery’, jQuery itself is the very last result.

Jam uses a package.json file which is compatible with npm, with some jam specific options in the jam property. You can then share your package by typing jam publish from the root of your project.

Enough facts, here are some opinions

So, six solutions to the same problem. One for node modules and command line tools, and five for the browser. Its definitely a great thing that these tools exist, and I highly recommend using one of them, even if you only use them as a convenient way of downloading jQuery when you start a project.

On the other hand, I can’t imagine that five different browser script managers will survive for long if they continue to overlap so much – both with each other and with other available tools. The command line interfaces are all very similar, as are the configuration options. If you want project automation then grunt has a much bigger community around it than Ender, Jam or volo, and Yeoman looks like it is going to clean up in the bootstrapping race. The repositories contain mostly the same packages and repository management is not likely to be the killer feature.

In an ideal world there would be two or maybe three options but they would be providing very different things. Instead, Ender and component are solving the same problem – small, decoupled components to replace the monolithic libraries. And grunt plus bower can do everything that Jam and volo do.

Package authors either have to put in the extra effort to maintain compatibility with all of the different systems or you end up with no system having access to every package. If a package author decides to create a component module, well then tough. You can’t use that with the other systems; especially not with bower as the component.json files conflict with each other.

Of course, it’s not up to me to declare a winner, and there isn’t an obvious candidate. I don’t get to say “you should all just use bower” or whatever, so it will probably take a while to shake out. But if I may make a prediction for 2013 it is that a lot of time will be wasted making each tool a little bit more like the others, rather than trying to differentiate themselves.

Promises, promises

For my inaugural post I’m going to talk a bit about promises in javascript. Promises are away to help you restructure asynchronous code so that program flow is more obvious. The usual way to write asynchronously in Javascript is to use callbacks, but that can lead to “callback hell” and the “pyramid of doom” – the ever rightward drift of your code due to nested callbacks

function loginFormHandler(username, password)
{
  login(username, password, function()
  {
    getUserDetails(username, function(userDetails)
    {
      getProfileImage(username, function(profileImage)
      {
        getPosts(username, function(posts)
        {
          showUserPage(userDetails, profileImage, posts);
        }, errorHandler);
      }, errorHandler);
    }, errorHandler);
  }, errorHandler);
}

You can alleviate this slightly by making all of your callbacks named functions, but then you lose the visual flow of the operations. With promises, instead of requiring a callback, you return a promise object. Each promise object has a method .then() that accepts success and error callbacks as parameters to be called when the asynchronous code finishes or fails.

But, and this is the important part, the .then() method returns another promise that will be fulfilled once the callback from the original promise have executed. This means you can chain promises to make asynchronous code that still shows the order that things will happen.

function loginFormHandler(username, password)
{
  login(username, password).
    then(getUserDetails).
    then(getProfileImage).
    then(getPosts).
    then(showUserPage, errorHandler);
}

When your asynchronous function has finished doing whatever it is doing you call promise.resolve(yourResult), while if it fails you can call promise.reject(anErrorObject). The appropriate callback is invoked with the argument passed to resolve() or reject(). The exception is when your callback itself returns another promise. Then the next .then() method along then chain will actually act on this new promise.

function getProfileImage(username)
{
  var promise = new Promise(); // The way of creating a new promise is library specific
  var image = new Image();
  image.addEventListener("load", function(event)
  {
    promise.resolve(this);
  });
  image.addEventListener("error", function(event)
  {
    promise.reject(new Error("Failed to load image"));
  });

  return promise;
}

Different promises libraries offer slightly different things, but the one that I have been playing with, called Q, lets errors propagate along the chain, so that if your first promise is rejected it will follow the .then() chain until it finds an error handler (or doesn’t find one at all which means it fails silently…)

Another nice feature is being able to group up an array of promises and wait for them all to complete, which in our example would allow us to fire off all four our sub functions in parallel since they don’t rely on each other, which in most cases would give you a speed boost.

Thoughts

Having used them in a real world project, I do have a few things I haven’t properly sorted out. I’ve found it hard to keep straight in my head what happens when return a promise from a callback. What happens if you resolve a promise with another promise? These aren’t difficult questions to get the answer to but they do mean stopping and thinking about it which can break your flow.

Whenever ES6 (the next version of the standard that Javascript is based on) is published, promises could become much cooler. There are examples written showing how, by using generators and yielding promises rather than returning them you can actually write code that looks very similar to synchronous code indeed. This isn’t going to be something you can use in cross-browser code for a long time though.

While a lot of my code got neater, some got uglier or more complicated in the promises version. I also tried explaining promises to a co-worker and not only failed to make it sound like an interesting solution to a problem, I actually made it sound like a massive over-complication. In part these things are probably due to only just getting to grips with the concepts – certainly the code got cleaner and cleaner with refactoring. Still, I’m keeping an open mind for the moment.

Hello world!

Over the last 6 months or so I’ve been doing a lot of work in Javascript, particularly with fancy HTML5 features like <canvas>, the FileSystem API and XHR2.

Because these things are all new and in flux, there aren’t a lot of great resources out there for everything. When there are good resources they tend to be written by groups working for browser developers (MDN, HTML5Rocks, MSDN, dev.opera.com) and so are not reliably browser neutral. Worse than that, the official HTML5 spec documents many wonderful features that no one has implemented yet. I’ve been bitten by that disappointment more than once!

And the world of Javascript apps is still very young. 10 years ago Javascript was pretty much only for form validation. Now you can write full-blown desktop applications and games. The limits to what you can achieve in the browser are receding fast, which brings new challenges. Keeping development sane at 1M+ lines of browser Javascript is not a solved problem so new techniques are popping up all over the place.

So I’ve decided that I am going to put up a quick post every week or so about what I’ve been hacking on or reading about so that I have a record of my discoveries and hopefully some small working examples of various techniques to look back at. On the off chance that any one else is interested, that’s a bonus!