June 8, 2015

GSoC: Week 2 - ClojureScript's JS dependency index

I have been accepted to this years Google's Summer of Code to work on ClojureScript. The goal of my project is to improve the integration of ClojureScript with the existing JavaScript ecosystem. I will post weekly updates about the progress of the project here. For more details about the project have a look at the ClojureScript GitHub wiki page.

During week 2 we continued to focus on adding support for CommonJS libraries to ClojureScript. An integral part of the CommonJS specification is the require function. The require function is used to import functionality that has been exported by other modules. In the following I will try to describe what we need to consider when adding support for CommonJS modules which depend on other modules and how a possible solution could look like.

Let's assume we have the following two CommonJS modules:

// libs/german.js
exports.hello = function() {
    return "Hallo";
};

// libs/greeting.js
var german = require("german");

exports.hello = function(name) {
    return german.hello() + ", " + name;
};
In the example above, we are importing the functionality that is provided by the german module into the greeting module. This means, if we would like to use the greeting module in our ClojureScript project we will need to add both modules to the ClojureScript compiler options, so that the compiler can find the sources.
(require 'cljs.build.api)

(cljs.build.api/build "src"
  {:main 'hello-world.core
   :output-to "out/main.js"
   :foreign-libs
     [{:file "libs/greeting.js"
       :provides ["greeting"]
       :module-type :commonjs}
      {:file "libs/german.js"
       :provides ["german"]
       :module-type :commonjs}]})

In general, to be able to resolve and check JavaScript dependencies for a ClojureScript namespace, the ClojureScript compiler first constructs a JavaScript dependency index using the compiler options. This index not only contains the dependencies that are passed via the compiler options :foreign-libs and :libs, but also contains entries for the modules that are listed in goog/deps.js. The dependency index is indexed by namespaces and each entry contains a map with more information about that namespace, e.g. the location of the file or which other modules this namespace requires. For example, for our example above we would like to have entries in the dependency index similar to the following:

{...
 greeting
   {:file libs/greeting.js
    :provides [greeting]
    :foreign true
    :url #object[java.net.URL 0x523424b5 file:/Users/marianeise/Documents/development/hello_world/libs/greeting.js]
    :module-type :commonjs
    :requires [german]}
 german
   {:file libs/german.js
    :provides [german]
    :foreign true
    :url #object[java.net.URL 0x523424b5 file:/Users/marianeise/Documents/development/hello_world/libs/german.js]
    :module-type :commonjs
    :requires []}
...}

In the example above, we can see that the map for the greeting module contains an entry under the :requires key which holds a vector with all the namespaces that are required. To create this entry, we would need to be able to parse calls to the require function from CommonJS modules and later also for AMD and ECMAScript 6 modules, which both use a different syntax to import modules. Instead of writing parsers for three different module specifications, we've decided to go with a different approach for now that uses existing functionality. The ClojureScript compiler already includes the functionality to parse the provide and require statements from Google Closure-compatible JavaScript files which are included into the project using the :libs option. As mentioned in my previous post, we are using the Google Closure compiler to convert JavaScript modules to Google Closure modules to be able to include them into a ClojureScript project. This means, before creating the dependency index we convert the CommonJS modules into Google Closure modules and write them to disk.

// out/german.js
goog.provide("module$libs$german");
var module$libs$german = {};
module$libs$german.hello = function() {
  return "Hallo";
};

// out/greeting.js
goog.provide("module$libs$greeting");
var module$libs$greeting = {};
goog.require("module$libs$german");
var german$$module$libs$greeting = module$libs$german;
module$libs$greeting.hello = function(name) {
  return german$$module$libs$greeting.hello() + ", " + name;
};

We now have Google Closure-compatible JavaScript modules and can update the compiler options to look similar to the following:

{:output-to "out/main.js"
 :foreign-libs []
 :libs ["out/greeting.js" "out/german.js"]
 :main 'hello-world.core}

Instead of passing the modules as foreign libraries we now pass them via the :libs compiler option to build the dependency index. Note that also the path to the files has changed to point to the converted version of the modules. This means, the modules will be treated as "normal" Google Closure-compatible JavaScript files by the compiler and the imports for our greeting module will be extracted successfully. The entries in the JavaScript dependency index will look similar to the following:

{...
 module$libs$greeting
   {:requires [module$libs$german]
    :provides [module$libs$greeting]
    :url #object[java.net.URL 0x4a067c25 file:/Users/marianeise/Documents/development/hello_world/out/greeting.js]
    :closure-lib true
    :lib-path out/greeting.js}
 module$libs$german
   {:requires []
    :provides [module$libs$german]
    :url #object[java.net.URL 0xa1217f9 file:/Users/marianeise/Documents/development/hello_world/out/german.js]
    :closure-lib true
    :lib-path out/german.js}
...}

To learn more about the dependency index, have a look at js_deps.clj in the ClojureScript project. To see the current progress for this project, have a look at the GitHub branch.

Tags: cljs cljs internals js modules commonjs gsoc