The State of Javascript Package Management

One notable thing about JavaScript in the browser compared to other popular languages is the almost lack of a standard library. Even some of the most common tasks can require a fair chunk of code. You can write it all yourself, but crafting cross-browser compatible code remains a minefield. If you don’t want to do it yourself, what can you do?

The “traditional” way in JavaScript was to copy dubious fragments of code from forums and Q and A sites into your project and hope for the best. Then, later, came the big libraries. Now if you want a cross-browser method of changing the colour of all of your links you can download an almost 10,000 line library file that included, somewhere in the middle, the functionality that you wanted. And you could manually download a fresh copy of a probably slightly different version each time you started a new project.

Clearly, this situation is not ideal, which is why we have seen several package managers spring up.

What is a package manager anyway?

I guess before we go any further we should talk about what a package manager is meant to achieve in the general case. Most package management systems are trying to solve a set of related problems.

The first, and most obvious, is to provide a reliable, standard mechanism for installing software created by third-parties. By specifying a convention (or possibly some very strict rules) about the structure of the packages the end-user only has to learn how the package manager works, not the specific installation procedure of the package they are trying to install. Things like where the software is installed, how it is built or compiled, or how it integrates with the rest of the system are all taken care of by the package manager.

Secondly, package managers provide dependency management. If package foo requires libraries bar and baz in order to function then the package manager will know to install bar and baz as well when asked to install foo. Most  can also deal with specific version requirements for dependencies, attempting to satisfy the tree of dependencies in such a way that compatible versions of packages are installed together.

Another common feature is to allow you to keep up to date with new versions of packages as they are released. In some cases you don’t want to upgrade – you don’t need those new features, so why risk breaking your project? Other times you want the latest and greatest so that you have all of the exciting features or essential bug fixes. A good package manager allows you to choose your own path.

A package manager should also allow you to discover new packages. If you want to solve a problem that you are pretty sure has been solved before, you can jump into your package manager’s search and see a list of possible solutions to check out.

The last features we’ll talk about here are for the people who create the packages in the first place. Some, but certainly not all, package managers will come with tools to help you create the package ready for distribution, and all of these systems need a way for the package author to put their work somewhere it can be discovered and used by the community.

A JavaScript package management tour

There are currently six contenders in the JavaScript package management arena – npmbowerEndervolo, component and jam. npm comes bundled with recent versions of Node.js, while the others are installable via npm with npm install <package> -g where the package name is bowerendervolo, component or jamjs.

npm

According to the FAQ, npm does not stand for Node Package Manager, though it is certainly a good description. The npm repository contains many JavaScript libraries and quite a few command line tools. There are packages available through npm to be used with browser based projects but most are specific to Node on the server. npm is important for browser development anyway, though, as all of the other browser-oriented tools mentioned below are themselves npm packages.

There are two ways to install packages with npm – global and local. Usually you would install tools you want to use from the command line globally and libraries you want to require() locally. Packages are installed with the npm install command. When you locally install a package npm will place the package files in a node_modules/<package> folder in your project route. Packages installed this way will be accessible in your node programs with require(). Globally installed command line tools will usually be put somewhere in your PATH (like /usr/local/bin on Linux).

The npm repository contains almost 20,000 packages which can be searched either on the command line with npm search or via the website. An npm module is defined by a file named package.json in the project root. This file has details like the name, description and version, which file is loaded when you require() the package, and dependencies. Dependencies are split into “dependencies” and “devDependencies” so that things that are only needed to build or test your package when developing it wont be downloaded by your end users.

When your package is ready for public consumption you can use npm publish to push it to the repository. Names are taken on a first-come, first-served basis. Only someone registered as an owner of a package can upload new versions but this is easily managed with the command line tools if you wish to add someone or transfer ownership. You need to have an npm account to publish packages.

bower

bower was created by Twitter for use on their internal web projects. A bower package can contain any assets a website might need, such as CSS, JavaScript or images.

This is the simplest of the tools discussed here as it provides a method for downloading packages and their dependencies and no more. This also makes it the most liberal, as any existing package can be put into the bower repository without modification and without adding constraints on the user.

The bower repository has 698 components which can be searched (by name only) using bower search or on a third-party search site. With bower a ‘package’ is actually a git repository, so the bower repository is only a list mapping names to public git endpoint urls. bower install <name> fetches the git repository into a components/<name> directory within the project root. bower has zero security or identity management. Anyone can publish a package on bower with bower register <name> <git endpoint>. Repository management is manual, which is to say that you have to message one of the admins if you want to remove, rename or re-register a component.

The package itself is defined by a file called component.json in the root of the git repository. Only the name, version and dependencies fields actually do anything. When retrieving packages from git endpoints bower uses tag names in the repository as valid version identifiers, so you must tag at least one revision before your package can be consumed.

Ender

Not just a package manager, Ender actually builds you a customised library to use from small, modular packages. The philosophy behind Ender is to give you only the code that you actually need for your project, and to do so in such a way that a module could be replaced by something roughly equivalent with the minimum of fuss.

To start a project using Ender you use ender build <name>[, <name>, ...] to create a custom library using the given packages. The created library exposes a global variable $ with properties defined by the component packages. Once your library has been started you use ender add and ender remove to manage the components that it provides. The library is written into ender.js and ender.min.js.

Ender piggybacks off of the npm registry, with each Ender package being an npm package. To signify that an npm module is compatible you add the ‘ender’ keyword to package.json. ender search will search npm, giving higher weight to packages with the keyword. You can see all packages marked for Ender using the npm keyword search on the website. There are currently 212 registered.

As well as adding ‘ender’ to your keyword list you can also optionally add an ender property to your package.json file that specifies a ‘bridge’ file which allows you to tightly integrate with the Ender API. Otherwise, creating and registering packages is the same as for npm.

volo

volo can be used much like bower to simply install the contents of a package, but it also includes a project automation/build system and can automatically turn installed packages into AMD modules, amongst other things.

You can bootstrap a project by using volo create <project name> which will create a directory called ‘<project name>’ containing the default template, though you can specify another template to use. You don’t have to use a template to use volo, however. If you use volo add <name> to add a package to your project then volo will use a set of rules to determine where to extract the package.

volo actually uses Github as its backend – a package is a Github repository and volo add will use the Github search API to find packages if you only provide a name or keyword. You can specify a full repository name with volo add <username>/<project name>, e.g. volo add jquery/jquery.

There is no special file for defining a package for volo, and publishing is simply a matter of creating the repository on Github. There is, however, a set of guidelines that will make your project easier to find and consume. Because every browser-oriented JavaScript repository on Github is technically a volo package, volo almost certainly has the highest available package count though exactly how many is unknown.

component

component is based on CommonJS style modules (the module style that Node uses) so like with Ender you must commit your whole project to this style. You start a project with component create <project name>  which will ask you several questions before creating a new directory under the current path with the given name. From inside the project root you can use component install <name> to install dependencies. You then need to use component build to create the file that you will load in the browser.

Like volo, component packages are Github repositories so dependency names are in the format <username>/<project name>. You can “publish” your package by adding the details of your repository to the wiki. You specify the details of your package in a component.json which is not compatible with the bower config file with the same name.

You can search the available components with component search or view the list on the wiki. The list currently contains 431 entries.

Jam

Jam is a tool for managing AMD packages only, and comes with a modified version of require.js, including a “compiler” based on the r.js tool.

Packages are installed with jam install <name> and get put in jam/<name> by default. To actually consume the packages, though, you use the modified require.js that comes with Jam. This file is updated when you add and remove packages to your application.

Jam has its own repository and you need an account to manage packages. The repository contains 279 packages which can be searched with the web interface.or with jam search. The sort order of the results is a little strange though; when searching for ‘jquery’, jQuery itself is the very last result.

Jam uses a package.json file which is compatible with npm, with some jam specific options in the jam property. You can then share your package by typing jam publish from the root of your project.

Enough facts, here are some opinions

So, six solutions to the same problem. One for node modules and command line tools, and five for the browser. Its definitely a great thing that these tools exist, and I highly recommend using one of them, even if you only use them as a convenient way of downloading jQuery when you start a project.

On the other hand, I can’t imagine that five different browser script managers will survive for long if they continue to overlap so much – both with each other and with other available tools. The command line interfaces are all very similar, as are the configuration options. If you want project automation then grunt has a much bigger community around it than Ender, Jam or volo, and Yeoman looks like it is going to clean up in the bootstrapping race. The repositories contain mostly the same packages and repository management is not likely to be the killer feature.

In an ideal world there would be two or maybe three options but they would be providing very different things. Instead, Ender and component are solving the same problem – small, decoupled components to replace the monolithic libraries. And grunt plus bower can do everything that Jam and volo do.

Package authors either have to put in the extra effort to maintain compatibility with all of the different systems or you end up with no system having access to every package. If a package author decides to create a component module, well then tough. You can’t use that with the other systems; especially not with bower as the component.json files conflict with each other.

Of course, it’s not up to me to declare a winner, and there isn’t an obvious candidate. I don’t get to say “you should all just use bower” or whatever, so it will probably take a while to shake out. But if I may make a prediction for 2013 it is that a lot of time will be wasted making each tool a little bit more like the others, rather than trying to differentiate themselves.

About these ads

20 thoughts on “The State of Javascript Package Management

  1. Johnny K

    More than just Yeoman, I’d recommend checking out Brunch, which isn’t based on grunt. Bower support is coming in the next version, and its skeletons and watcher are absolutely amazing.

    Reply
  2. Mariusz Nowak (@medikoo)

    If we’re after *one* package manager that serves all environments (server, browsers, etc.) then NPM is the only choice, that allows to write modules without unnecessary boilerplate. It’s worth mentioning tools like Webmake – https://github.com/medikoo/modules-webmake or Browserify – http://browserify.org/ with them you can easily pack NPM modules for browser.
    I’m currently building quite complex applications, for browsers that also run some code on a server-side and CommonJS modules style provided through NPM packages works great. Technically I see no point in other managers.

    Reply
  3. Pingback: Javascript Package Management ← Tech Notes

  4. José F. Romaniello (@jfroma)

    Even if I like the CommonJS Module Pattern the big problem with the browser is that you get a gigantic file with your code and the code of your dependencies, which is a mess to debug. So, in my opinion at least for now AMD modules are better for browser

    This leaves me with two choices bower or jamjs and I prefer bower because I have tried jam before and it seems very invasive with the require.js
    things .

    There is a way you can convert harmony like module pattern to AMD today with: http://caolanmcmahon.com/posts/try_harmony_modules_today/

    Reply
  5. ionrock

    Seeing as I rarely do much client side development, I might be misunderstanding package management in the browser. It seems like browser based javascript is really a build problem more so than a package management issue. I could see npm locally for getting an environment configured, but I’d imagine a build step that actually creates what is deployed. At that point it seems like the page should download what is necessary. Am I missing something?

    Reply
    1. Mat Post author

      At runtime the page downloads what it needs, as you said, but where from? A package manager allows you to fetch the correct versions of things you need into your own project so that they can be hosted with the rest of your site, ready for the browser to download them.

      Taking bower as an example, if I want to use jquery in my site I can use “bower install jquery” and it will add a “./components/jquery/” directory to my project. Then in the HTML I can add .

      Why is this better than just downloading the file yourself? For me the killer feature is upgrading. Using bower again, “bower update” from within your project will find all of the packages that you are using and fetch the latest compatible versions of them. If you have set up your dependencies correctly then semantic versioning (http://semver.org/) should mean you have all of the latest bug fixes and security patches without breaking anything.

      There are plenty of other benefits of each of these tools, though what they are varies slightly between them.

      Reply
  6. Pingback: The State of Javascript Package Management | Father Milk

  7. Pingback: YUI Weekly for January 11th, 2013 - YUI Blog

  8. Pingback: Friday Links #237 | Blue Onion Software *

  9. Konstantin Raev

    I tried all of them a bit, here is my opinion.

    1. All front end javascript community is on GitHub and maintaining packages in more than one place is a hassle. I stopped using Jam after I realized that I am adding fifth library to the Jam repository just copying them from github.

    And who is the package maintainer: the one who added a package or the one who forked it on GitHub or the original creator? The lag between a package update and the repository push can be quite big.
    This left me to choose between volo, bower and component.

    2. AMD was appealing at first with a great website and community around it.
    But they keep on insisting that the killer feature is not having to compile your javascript into one is so awesome.
    I don’t agree, it is nice not to compile all the time during development but I will have to do it before production anyway and will have to test it compiled again. With Source Maps feature in latest browsers debugging compiled javascript is as easy as AMD.

    Also I hate all the boilerplate code of AMD that is required to make it run.
    And I don’t see it in the future when JS native modules come.

    3. This left me to choose between bower and component.
    And I chose component.
    It was the simplest one to get started with.
    And it is blazing fast with downloading and building modules, that is because the design makes you create very tiny and small focused components and it uses only the required files, it does not download all the extra trash like compiled sources, documentation and committed dependencies from repository.
    With source maps and automatic build it takes milliseconds to see changes on the screen and debug them.

    Reply
  10. Peter StJ

    The main problem seem to be compatibility. So what is logical in this case is to include a transformation script with the package manager. Using esprima with additional tools should be possible to convert at least one or two other module types into something that can be consumed as dependency. I believe the idea to use git repos as namespaces to avoid collision is great, (better than npm’s names). That, combined with pre-processing of the consumed package to make it compatible sounds like a good step forward, instead of forcing the authors to make commitments to a package manager. What do you think of this?
    
    Closure tools offer package management as well. Closure is nice, but it does not do much if you need lots of third party code and closure requires more effort from the developer, it is hard to get started with and is limiting of what you can do with the language. On the other hand it is an example of persisting API – the modularized APIs has hardly changed in the years, if an api needs severe change it is introduced as a new one.

    Using browserify tends to include one and same package with different versions. The problem is caused again by the lack of discipline. Brilliant people come up with a brilliant idea of a module and it is great, lots of people use it etc. But in 6 months another brilliant idea strikes the author and instead of applying discipline and work around the existing API or introduce a facade to mitigate migration issues the authors go straight to introducing a breaking API and they do not care, they simply bump up the version. JQuery has done similar thing (removing different ways of binding and leaving only one – look all the protesting they have got) and they are not applauded for it.

    I spend my days buried in enormous code base. Until recently the only sane way to manage several repositories with tons of code in each was closure. I have tried AMD (requireJS) and it sucks, Every time a path changes I need to go over and write the path anew – this completely kills productivity. You also need that extra code and dedicate your modular code to the AMD pattern, which means it will hardly be seen as usable by another community.

    Recently I have looked at TJ’s component, I think it is a great idea to flatten out the dependencies and to use namespaces to prevent collision, however the tool is very primitive and lacks extensibility by design (TJ says it is not possible to attach hooks as plugins, I have to fork), so to make it work with your tools and fit your requirement it will need forking. Really not ideal. Also it can do little for third party code as well. So basically you are again stuck with the community.

    I think the solution proposed at the top indeed is the only logical solution at this stage. It will allow communities to mix code, which is what we all want.

    Reply
    1. Vasja

      That’s exactly what I’m worried about. There is no way to validate unknown package quality except for sitting there and waste time scanning through the code.

      Reply
  11. Pingback: Catch-Up | Wibblymat's code blog

  12. Anonymous

    Please try to put a date in your post :)
    In a quickly changing field as html5 and Js, it’s a usefull information to know how old are the info we’re reading.
    thank you for the good article though ;-)

    Reply
  13. Pingback: Kari's World » Blog Archive » Javascript dependency hell

  14. Pingback: Checking in front-end dependencies

  15. Pingback: Javascript Package Management – NPM – Bower – Grunt | shreelimbkar

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s