Peterbe.com

One of the many great things about working for Mozilla is that everything we do is Open Source. Even our wiki is open (however we have an internal wiki for corporation boring stuff such as meeting rooms, HR etc.)

Last week I wrote an internal application for Mozilla's build engineers. Essentially it's a roster that lists one user per day and it's helped by being visualized as a calendar and as a vCal export. It's very unlikely that anybody outside Mozilla will find this particularly useful. But who knows, perhaps other companies have needs to take turns to sheriff build machines.

Anyway, the project was easy to write because we have something called Playdoh. It's a set of nifty and useful settings and a folder structure and it comes with a submodule called "playdoh-lib" which is stuffed with lots of useful packages that you'll most likely want to use. If you browse Playdoh on Github it might look like a lot of stuff but after a second look you'll see that there's actually almost no code. So don't you dare to play the "bloat card"! :)

What this app uses is TastyPie for the REST API which was awesome by the way.

For the authentication I used django-auth-ldap and some custom classes because at Mozilla we use email addresses instead of usernames.

To make the vCal export I use VObject which was easy to work with but has some usual syntax in places.

Jinja was used for the template rendering and it meant I had to do some tricks to use the django.contrib.auth.views.login view but with my templates. Might be worth looking into if people are interested.

The code has 98% test coverage but I had to upgrade to the latest nose to be able to run test coverage on app modules that have similar names to modules in the standard lib.

On Kwissle I'm using Google Closure Compiler to minify all Javascript files. It's fine but because it's java and because I'm running this on a struggling EC2 micro instance the CPU goes up to 99% for about 10 seconds when it does the closure compilation. Sucks!

So, I threw UglifyJS into the mix and instead of replacing the Closure compiler I added it so it runs alongside but I obviously only keep one of the outputs.

Here is the log output when I run it on my MacbookPro:

MAKING ./static/js/account.js
UglifyJS took 0.0866 seconds to compress 3066 bytes into 1304 (42.5%)
Closure took 1.2365 seconds to compress 3066 bytes into 1225 (40.0%)
MAKING ./static/js/ext/jquery.cookie.js
UglifyJS took 0.0843 seconds to compress 3655 bytes into 3009 (82.3%)
Closure took 1.3472 seconds to compress 3655 bytes into 4086 (111.8%)
MAKING ./static/js/ext/jquery.tipsy.js
UglifyJS took 0.1029 seconds to compress 7527 bytes into 3581 (47.6%)
Closure took 1.3062 seconds to compress 7527 bytes into 3425 (45.5%)
MAKING ./static/js/maxlength_countdown.js
UglifyJS took 0.082 seconds to compress 1502 bytes into 1033 (68.8%)
Closure took 1.2159 seconds to compress 1502 bytes into 853 (56.8%)
MAKING ./static/js/ext/socket.io-0.6.3.js
UglifyJS took 0.299 seconds to compress 76870 bytes into 30787 (40.1%)
Closure took 2.4817 seconds to compress 76870 bytes into 30628 (39.8%)
MAKING ./static/js/scoreboard.js
UglifyJS took 0.084 seconds to compress 2768 bytes into 1239 (44.8%)
Closure took 1.2512 seconds to compress 2768 bytes into 1167 (42.2%)
MAKING ./static/js/rumbler.js
UglifyJS took 0.0872 seconds to compress 3087 bytes into 1384 (44.8%)
Closure took 1.2587 seconds to compress 3087 bytes into 1235 (40.0%)
MAKING ./static/js/ext/shortcut.js
UglifyJS took 0.0987 seconds to compress 5796 bytes into 2537 (43.8%)
Closure took 1.3231 seconds to compress 5796 bytes into 2410 (41.6%)
MAKING ./static/js/play.js
UglifyJS took 0.1483 seconds to compress 18473 bytes into 10592 (57.3%)
Closure took 1.4497 seconds to compress 18473 bytes into 10703 (57.9%)
MAKING ./static/js/playsound.js
UglifyJS took 0.0824 seconds to compress 1205 bytes into 869 (72.1%)
Closure took 1.2335 seconds to compress 1205 bytes into 873 (72.4%)  

(Note here that for the file ./static/js/ext/jquery.cookie.js Closure failed and when it fails it leaves the code as is and prepends it with a copy about the error from the stdout. that's why it's greater than 100% on that file)

Here are the averages of those numbers:

AVERAGE TIME: (lower is better)
  * UglifyJS: 0.11554 seconds
  * Closure: 1.41037 seconds

AVERAGE REDUCTION: (higher is better)
  * UglifyJS: 45.6% 
  * Closure: 51.5%

(I'm skipping the file that Closure failed to minify)

So, what does that mean in bytes? These are the source Javascript files for two pages but the total is 123949.0 bytes. With Closure that saves me 63833.7 bytes (62 kbytes) whereas UglifyJS only saves me 57760.2 bytes (56 kbytes) of bandwidth.

Discussion...

The fact that Closure fails on one file is a real bummer. I'm not even using the advanced options here.

UglifyJS doesn't save as many bytes as Closure does. This is potentially important because after all, minification process happens only once per revision of the original file but might be served hundreds or millions of times.

Because I run my minifications on-the-fly it does matter to me that UglifyJS is 1220% faster.

It's rare but I have observed twice that the minified Javascript from Closure has actually broken my code. I default to suspect that's my fault for not making the code "minifyable" enough (e.g. too many global variables).

I've just noticed that I'm relying on files that almost never change (e.g. jquery.tipsy.js). I might as well create a ...min.js versions of them and add them to the repository.

In conclusion...

Because of the convenience of UglifyJS being so much faster and that it doesn't choke on that jquery.cookie.js file I'm going switch to UglifyJS for the moment. The remaining bytes that I don't save become insignificant if you add the gzip effect and compared to images the bandwidth total is quite insignificant.

UPDATE (JAN 2016)

I wrote a follow-up post comparing UglifyJS2 with Closure Compiler with the advanced option.

Here are the slides from yesterday's London Python Dojo event.

I presented and demo'ed "Kwissle" to my fellow Python London friends and focused a lot on the technology but also tried to plug the game a bit.

Having seen that there's a lot of interest in "socket" related web applications about I thought this was a good chance to say that you don't need NodeJS and that tornadio is a great framework for that.

My good friend Chris who runs the Min River Tea Farm has this week launched a new product: Sampler Pack Chinese tea

This is brilliant because if you, like me, love some good green teas but don't want to buy a whole bag yet, then get a cute little sampler and decide which of his teas you like the most. Or, for £9 you can buy a couple of these cute little tubs and give away as gifts. That's what I'm going to do.

At the moment you have to be in Europe to be able to order these (delivery done by Amazon UK) but if you're outside of Europe ping them and ask if something can be arrange.

Here's a little interesting story about using MongoKit to quickly draw items from a MongoDB

So I had a piece of code that was doing a big batch update. It was slow. It took about 0.5 seconds per user and I sometimes had a lot of users to run it for.

The code looked something like this:

 for play in db.PlayedQuestion.find({'user.$id': user._id}):
    if play.winner == user:
         bla()
    elif play.draw:
         ble()
    else:
         blu()

Because the model PlayedQuestion contains DBRefs MongoKit will automatically look them up for every iteration in the main loop. Individually very fast (thank you indexes) but because of the number of operations very slow in total. Here's how to make it much faster:

   for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):

The problem with this is that you get dict instances for each which is more awkward to work with. I.e. instead of `play.winner` you have use `play['winner'].id`. Here's my solution that makes this a lot easier:

class dict_plus(dict):

  def __init__(self, *args, **kwargs):
       if 'collection' in kwargs:  # excess we don't need
           kwargs.pop('collection')
       dict.__init__(self, *args, **kwargs)
       self._wrap_internal_dicts()

   def _wrap_internal_dicts(self):
       for key, value in self.items():
           if isinstance(value, dict):
               self[key] = dict_plus(value)

   def __getattr__(self, key):
       if key.startswith('__'):
           raise AttributeError(key)
       return self[key]

  ...

 for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):
    play = dict_plus(play)
    if play.winner.id == user._id:
         bla()
    elif play.draw:
         ble()
    else:
         blu()

Now, the whole thing takes 0.01 seconds instead of 0.5. 50 times faster!!

For the past couple of months I've been working on a little fun side-project that I've decided to call: "Kwissle"

It's an online game where you're paired up randomly with someone else who's online on the site at the same time and you enter a battle of general knowledge questions.

The rules are simple:

• each battle has 10 questions
• you get 15 seconds per question
• if you type in the right answer (spelling mistakes are often allowed) you get 3 points
• if you load the alternatives and get it right you get 1 point
• you have to be fast or else your opponent might steal the question
• whoever has the most points win!

The app has been quite a technical challenge because it's real-time. Like a chat. To accomplish this, I've decided to use the socket.io library with Tornado which means that if your browser supports it, it's using WebSockets or else if falls back on Flash sockets.

Because there is no easy way to source questions, I've instead opted to do it by crowd sourcing. What that simply means is that YOU help out adding the questions. You can also help out by reviewing other peoples questions before I eventually publish them.

Bear with me though; this is an early alpha release and it just about works. There are a tonne of features and improvements I want to add to make it more fun and solid but, like I said, this is a side-project so the only time I get to work on it is late evenings and weekends. The source code is not open but I'm going to try to be entirely open with what's going on on a tumblr blog

Since Google doesn't really have a bug tracker where I can report bugs I blog about it instead.

Here's a typical example that Google's strife to get rid of duplicate content still needs some work. About 10 different mailing list archive sites have indexed the same email thread. Not very helpful.

At Mozilla we use jingo-minify to bundle static resources such as .js and .css files. It's not a perfect solution but it's got some great benefits. One of them is that you need to know exactly which static resources you need in a template and because things are bundled you don't need to care too much about what files it originally consisted of. For example "jquery-1.6.2.js" + "common.js" + "jquery.cookies.js" can become "bundles/core.js"

A drawback of this is if you forget to compress and prepare all assets (using the compress_assets management command in jingo-minify) is that you break your site with missing static resources. So how to test for this?

If you look at html5boilerplate.com they never use any type on their <script> tags. Hmm... but is there more to it?

If you don't specify a type, the default becomes "text/javascript" but according to RFC4329 the "text/javascript" MIME type is obsolete in favor of "application/javascript".

If the default MIME type for a <script> tag thus becomes either "text/javascript" or "application/javascript" is there any sensible browser on this green earth that would not translate a piece of inline Javascript code as, exactly that; Javascript? Probably not.

What about <script> tags with a src attribute? Does the type matter?

I read the spec a couple of times and it feels like reading legalese but it ultimately says: the value of the type tag must be that of the body of the script tag.

So, what happens if you embed a javascript file with a mismatching type? Let's see:

If people find this useful I might turn it into a proper jQuery plugin and publish it.

Without further ado; here's the demo

What it does is that for all input fields that have a maxlength="nnn" it shows a counter to the right that increases in opacity as it reaches the maximum. You can generally start it like this:

$('input[maxlength]').maxlength_countdown();

Since the plugin "hard codes" the count down expression in English you can override it easily like this:

$('input[name="message"]').maxlength_countdown(function (count, max) {
   return count + " (max: " + max + ")";
});

What do you think? Is it useful? Does it make sense?