Общо показвания

април 28, 2012

The JavaScript ecosystem problem

I tend to follow the news in the JavaScript field because it is related to my work. What I notice lately is that every developer out there is making attempts to popularize his/her favorite framework and/or tool.

Here is my objection to all this: Currently the situation is this - the ecosystem of javascript tools and libraries is fragmented. The pieces are so small and so incompatible that it makes it impossible to build a large web application without actually spend months in development time trying to make the pieces work with each other.

There is jQuery, which unfortunately is now much more than a tool to hide the browser quirks. It now provides pub/sub, promises and what not. There is also a whole lot of 'plug-ins' for jQuery. But those require different version of jQuery. So you might need more than one jQuery.

Then there is 'modular' code, AMD. One being the new dojo (1.7), which is a pretty big modular library, but it is not really compatible with other AD loaders. There is RequireJS, which is also popular because of its 'optimizer', but it is only that - loader, and does not provide any library for browser quirks hiding, so the developer is left with no library at all. jQuery is popular choice in this case.

There is closure library and closure compiler, however those are completely unusable with code that was not specially designed to work with them.

There is prototype, there is backbone there is underscore... every single one of those has its fan base and none of theme follows a code pattern that will allow a developer to just use the needed functionality without any additional hassle.

This of course leads to the decision enforcement: the developer has to make decision which technology to use (some 'ninjas' even go as far as creating their own set of tools/mini libraries and use those, but this is time consuming and error prone and the library takes additional work to be supported and renewed). It does not matter what technology is selected, every single one that is popular has its benefits. What is more worrisome is the following: How much tie will it take for the used tool/library to become incompatible/obsolete?

Lets say that currently I find the closure tools to bring the best combination of browser quirks hiding, performance and code optimization quality. As every single tool out there it imposes a set of rules I need to follow in order to make my code work with it. Notice that __ALL__ tools do this. For example jQuery also requires the whole library to be loaded before I load the  plugin. It also requires a global variable to be exposed ($ or jquery). Most libraries count on a global variable to be available with a certain name. Closure makes exception in this, but imposes much more rules. So let's say that I create a super useful UI component, that is based on closure library. If you want to use it you have two choices: a) use closure library/tools or b) rewrite the functionality with your fave library. The last one is the more popular choice. One and same widget/component is currently being re-written over and over again with little augmentation to the actual functionality. LightWindow anyone? It is available as stand alone, jquery plugin, mootools plugin, prototype plugin, probably as closure library component as well. Can you imagine how many useful development hours have been wasted for this single component to be ported to so many tools/libraries?

Lets continue our in-mind exploration! I finish my component, I write tests for it and and I am happy with it, it goes to production. Then one day, lets say 2 years later I need it in another project. Unfortunately this new project does not use closure tools, because, let's say requirejs optimizer has been rewritten to use a new AST parser that is able to rewrite and minify javascript more effectively and faster than closure compiler and does not impose such strict rules on the code style. And (I will imagine this one) there is great browser abstraction library that is AMD and require js compatible, and maybe even IE is now standards compliant and I don't really need that much abstraction. So I don't really need all the closure library code. I can only hope that someone has already rewritten my code as an AMD module. But because it was closure compiled most probably no one did that ( here is a -1 for closure compiled code, you might think that you protect your code but what you are doing is actually making more work for yourself for later times, just because you did not allow someone else to do it for you), I need to do it manually. How possible is that scenario. Very much possible.

The problem this small example demonstrates is the lack of stability in the defined interfaces. Yesterday everything was jquery. Today people tend to look at other direction, very soon something else might be the big thing. Today people are talking about large code bases and large web applications. AMD and Closure tools are the most talked in those, while I think extJS is also a great example.

Looking at other languages the interfaces remain much longer untouched. In JavaScript code written 2 years ago is now deprecated because the engines change and optimization are found.. code written 10 years ago is actually not running anymore in the new browsers. I have a large code base of windowing system in the browser (similar to extJS) at my hand and honestly I do not know what to do with it...

What should we do?

I don't know.  My best bet right now would be AMD. But as we all know there should be modules in next version of the javascript language. Which means that the code will still be unusable few years from now. Or it will require legacy tooling, which is even worse than rewriting it. Imagine a situation where the quirks are actually in your code instead of in the host environment and you need shims for the code itself to assure its proper working.

I think this was on the Dart team's mind when the new language was born. And yet I am not sure that it is the answer. Maybe time will show.

But in the mean time I think the developers should work toward more compatible and replaceable code. Code where you do not require a module, but instead you require a functionality. Or an interface. Lets say I want to use the query selector interface for the DOM. Most modern browsers support it natively, but some that are still popular and significant percentage of people use do not. In that case the current solution is to use a library, that has selector engine implementation. jQuery has one, mootools also has one, dojo also has its own. Unfortunately not two of those provide the same interface. Mootools separated the selector engine and it can be used stand alone, closure library use dojo's implementation as a namespace, but yet those are not hot swappable. For example I cannot just require the functionality? A robust solution would be such as: the developer requires the functionality and the tool, based on the environment it runs in provides either a complete selector engine (like Slick for example) or a thin wrapper around the native implementation, should the host supports it. This is possible in the current state of affairs, but what about all the differences in the browsers?

This problem causes lots of time being spend in evaluating the endless possible solution to one and same problem. Which I find insane! And that is why I think I will have to move on on another field. Because this one, the web development, has taken too much talking and thinking and not so many action....

април 08, 2012

One more thing on javascript minification

There seem to be some very troubling misunderstanding that arises when one go on the hunt for compression technology for javascript: most comparison is made on two false assumptions.
  1. Closure compiler should be used in simple mode
  2. Code size measurement should be done when using either libraries not written in the required for advanced optimization style or using own application code.
Here is why those two are fallacies!

I will start with number two - unused code from libraries and utilities.
In advanced mode the compiler makes something no other compression tool out there is able to do (including uglifyjs, as confirmed by Mihai Bazon this is even not in their scope): the compiler is able to determine the types as if it was run-time and thus strip code paths that were never utilized.

Most code is written to be application agnostic and thus contains lots and lots of methods and functions and variables that are never used in an application. All those contribute to slow execution because they a) need to be parsed and then b) evaluated

Here is an example code
Car = function(color) {
  this.color_ = color;
  this.mileage_ = 0;
};

Car.prototype.drive = function( direction ) {
  this.mileage_ ++;
};

var n = new Car('black');
console.log(n.color_);
UglifyJS currently is completely unable to identify the fact that, should this be considered the whole project code, the drive method and the mileage_ property are never used. Even more, when posted this on the discussion group I гot pointed out that considering the following additional code:
function driveCar(c) { c.drive() }
uglify is not able to determine the type of the argument for the driveCar function, thus, is unable to know that the drive method is used (or unused in case c is not actually instance of Car) and thus it is unable to determine if the drive method should be removed or not.

I was also pointed out that this kind of code modifications are outside the scope of the project. I can understand and accept that. But consider how much one could save from a DOM access library if all code related to ajax is stripped, should ajax is not required, or promise related, or pub/sub related code can be removed from the resulting build should it not be used.I would argue that while this might not seem like a lot, getting built code that contains only the instruction that you actually use is much better than containing all possible instructions in all used pieces of code.

Now for number one: Closure is compared to UglifyJS in simple mode.
Well, of course it is slower, it is written in java! And of course closure cannot compile all your libraries, it is written to compile applications, not libraries for third party use ( however at google they use it for this too!). Using the compiler in advanced mode will most often break your code, this is why there are some code style rules to be followed should the compiler is to be used in that mode. The gains are significant, but the friction is so much, most developers just do not bother. For one, you cannot just stop any library in your project path and expect it to work. This is a big show stopper for most developers. Then you also need to write your code in google's funny way of name spaces. You are forced to use only a few subset of JS patterns ( no matter how memory and efficiency optimized those are - limits are limits).

So next time you evaluate the two options, please make note of the difference between minification and 'compilation'. While it is not a real compilation, the code changes made are much greater than method and property renaming.


април 07, 2012

JavaScript optimizations

I have been working the last year and a half on large in-browser applications and the last few months I have been looking into automated method of code optimization. The two most significant options out there currently are Closure compiler and Uglify-js.

I have used both and the following text is about my experience.

I will start with the few per-conditions: The code is modularized, i.e. different components are written as modules in separate files already and not all components should be loaded for the application to run. The application should be able to load additional dependencies/components as reaction to user actions or other events. The code should be minified  if possible and concatenated into small number of files, the lower the number of requests - the better.

Starting with code base of 1.8MB the task at hand was to squeeze it to something that can be loaded via the Internet on a low performance STB device. The application is the main device interface, so it should indicate activity to the UI asap.

The code was initially developer with requirejs so making a build of it with node requirejs module was not very difficult. The code needed some tweaking but in one day work the whole project compiled to 279KB. The resulting javascript file is only one (i.e. monolithic build) and once downloaded was able to start the UI in 2.79 seconds (as opposite to 9 seconds when un-compiled, due to more file loading mostly). This however, combined with the network lag of downloading 280KB  resulted in a black screen for almost 6 seconds and was not fast enough for a home entertainment equipment.  What we needed was 'load indication' module, that loads first and then indicate the load progress for the next few seconds.

Next we decided to try and use closure compiler. The compiler however forces the developer to use limited set of javascript patterns and it took a few days to stip the incompatible code or replace it with one that is. Some modules were not ported, for example the settings module. We used the closure-script to ease the development cycle. The result was impressive. The compiler is able to inline lot of the code ( for example property getters/setters are inlined, which lowers the scope creation load) and the tool we used is able to include the needed files per modules and modules are defined with namespaces. One caveat is that all modules need additional code for initialization, which we needed to write in addition to the logic code we already had. Another caveat (or one can think of it as advantage) is the renaming of method names, which means that you get much smaller size of the final built but you loose the names. The structure of your code might change beyond recognition, but the payoff often worth it. However the additional code that need to be written and the required changes should your code base is written without the compiler in mind can make the process long and tedious.

Finally we decided to use the requirejs whole project build capabilities. Once again it required some getting used to it and a bit testing, but we managed to separate the monolithic build to 4 modules. While testing the closure compiler we really liked the defines ( i.e. variables that can be defined at compile time and allow the compiler to strip code that is branched out based on the value of the defines) and we wanted to used it in the project. It was a nice surprise for us the fact that uglify-js supports defines as well. Both uglify and closure compiler require special syntax for this to work but the overhead is very small compared to the benefits.  We use it now to strip the debug modules and debug statements from the build. As requirejs works in node we were able to produce a small development server that can run the build process based on the query string submitted with the index request in less than 15 minutes. This frees us from the java virtual machine (as opposed to the closure tools). Uglify also makes its magic faster compared to closure compiler. Uglify however does not do function inlining for methods (getters/setters) nor does overwrite method names ( as this is not safe transformation in the context of non-restrictive javascript style ), it also does not make namespace squashing, which means that you need to make some extra effort to make sure your code does not use excessive scope creation/nesting.

The final thought on this project are as follow:
  1. You need to first decide which compiler you would you as both require different code style to work properly. 
  2. You need to go an extra step to make sure the code is structured suitably for modularization in both compilers.
  3. Do not forget that it is simply a tool, there are more than one ways to get to the destination, do not get obsessed!
As a last point I would suggest closure tools for projects that want to work across all browsers and have the latest abstractions (for example they already have abstraction for the transitionend event as it differs in firefox and chrome) and you do not want to collect the abstractions yourself. Closure library can save you lot of time for this.

On the other hand if you are targeting only newer browsers you can go with the faster and easier to use uglify-js/requirejs.