Goodies from tornado-utils - part 2: tornado_static

September 22, 2011
0 comments Tornado

This is Part 2 in a series of blogs about various bits and pieces in the tornado-utils package. Part 1 is here

tornado_static

Code is here

This code takes care of two things: 1) optimizing your static resources and 2) bundling and serving them with unique names so you can cache aggressively.

The trick is to make your development environment such that there's no need to do anything when in "debug mode" but when in "production mode" it needs to be perfect. Which files (e.g. jquery.js or style.css) you use and bundle is up to you and it's something you control from the templates in your Tornado app. Not a config setting because, almost always, which resources (aka. assets) you need is known and relevant only to the templates where you're working.

Using UI modules in Tornado requires a bit of Tornado-fu but here's one example and here is another (untested) example:


# app.py

import tornado.web
from tornado_utils.tornado_static import (
  StaticURL, Static, PlainStaticURL, PlainStatic) 

class Application(tornado.web.Application):
   def __init__(self):
       ui_modules = {}
       if options.debug:
           ui_modules['Static'] = PlainStatic
           ui_modules['StaticURL'] = PlainStaticURL
       else:
           ui_modules['Static'] = Static
           ui_modules['StaticURL'] = StaticURL

       app_settings = dict(
           template_path="templates",
           static_path="static",
           ui_modules=ui_modules,
           debug=options.debug,
           UGLIFYJS_LOCATION='~/bin/uglifyjs',
           CLOSURE_LOCATION="static/compiler.jar",
           YUI_LOCATION="static/yuicompressor-2.4.2.jar",
           cdn_prefix="cdn.mycloud.com",
       )

       handlers = [
         (r"/", HomeHandler),
         (r"/entry/([0-9]+)", EntryHandler),
       ] 
       super(Application, self).__init__(handlers, **app_settings)

def main(): # pragma: no cover
   tornado.options.parse_command_line()
   http_server = tornado.httpserver.HTTPServer(Application())
   http_server.listen(options.port)
   tornado.ioloop.IOLoop.instance().start()

if __name__ == "__main__":
   main()

Note! If you're looking to optimize your static resources in a Tornado app you probably already have a "framework" for setting up UI modules into your app. The above code is just to wet your appetite and to show how easy it is to set up. The real magic starts to happen in the template code. Here's a sample implementation:


<!doctype html>
<html>
   <head>
       <title>{% block title %}{{ page_title }}{% end %}</title>
       <meta charset="utf-8">
       {% module Static("css/ext/jquery.gritter.css", "css/style.css") %}
   </head>
   <body>
      <header>...</header>
   {% block content %}
   {% end %}
   {% module Static("js/ext/head.load.min.js") %}
   <script>
   var POST_LOAD = '{% module StaticURL("js/dojo.js", "js/dojo.async.js") %}';
   </script>
   </body>
</html>

What you get when run is a template that looks like this:


<!doctype html>
<html>
   <head>
       <title>My title</title>
       <meta charset="utf-8">
<link rel="stylesheet" type="text/css" href="//cdn.mycloud.com/combined/jquery.gritter.css.style.1313206609.css">
   </head>
   ...

(Note that this will create a whitespace optimized filed called "jquery.gritter.css.style.1313206609.css" in "/tmp/combined")

Have a play and see if it makes sense in your app. I do believe this can do with some Open Source love but so far it works great for me on Kwissle, DoneCal and TooCoolFor.Me

Goodies from tornado-utils - part 1: TestClient

September 20, 2011
0 comments Tornado

This is Part 1 in a series of blogs about various bits and pieces in the tornado-utils package.

tornado-utils is the result of me working on numerous web apps in Tornado that after too much copy-and-paste from project to project eventually became a repo standing on its own two legs.

TestClient

Code is here

This makes it possible to write integration tests that executes GET and POST requests on your app similarly to how Django does it. Example implementation:


# tests/base.py

from tornado_utils.http_test_client import TestClient, HTTPClientMixin
from tornado.testing import LogTrapTestCase, AsyncHTTPTestCase

class BaseAsyncTestCase(AsyncHTTPTestCase, LogTrapTestCase):
   pass 

class BaseHTTPTestCase(BaseAsyncTestCase, HTTPClientMixin):

   def setUp(self):
       super(BaseHTTPTestCase, self).setUp()
       self.client = TestClient(self)

# tests/test_handlers.py

from .base import BaseHTTPTestCase

class HandlersTestCase(BaseHTTPTestCase):

   def setUp(self):
       super(HandlersTestCase, self).setUp()
       self._create_user('peterbe', 'secret')  # use your imagination

   def test_homepage(self):
       response = self.client.get('/')
       self.assertEqual(response.code, 200)
       self.assertTrue('stranger' in response.body)

       data = {'username': 'peterbe', 'password': 'secret'}
       response = self.client.post('/login/', data)
       self.assertEqual(response.code, 302)

       response = self.client.get('/')
       self.assertEqual(response.code, 200)
       self.assertTrue('stranger' not in response.body)
       self.assertTrue('hi peterbe' in response.body)

You can see a sample implementation of this here

Note that this was one of the first pieces of test code I wrote in my very first Tornado app and it's not unlikely that some assumptions and approaches are naive or even wrong but what we have here works and it makes the test code very easy to read. All it basically does is wraps the http_client.fetch(...) call and also maintains a bucket of cookies

I hope it can be useful to someone new to writing tests in Tornado.

Reciprocal lesson about gender perspectives

September 2, 2011
0 comments Wondering

Picked this up on a food blog in one of the comments; a female commenter wrote:

"Guys went for me all the time in my late teens and early twenties. I’m no great beauty, but they did anyway. I was slender.
I did not remain slender. No one looks at me now.
I'm the same person."

Err! Wrong! You're not the same person! If we assume that this "slender" versus "not slender" is the differentiating factor and ignore age for now then with beauty changes the person.

It's such a common misconception sometimes heard from women that the personality is one factor of attraction and that external beauty is another. In the opposite, women do play down the external beauty factor more when considering men's total attraction value. Take heed women, your beauty factor matters more than men's. No news there. The mistake here is to think the opposite sex thinks like you do.

So what's the reciprocal lesson in this? For a man. Quite simply: don't think that women think like men. Or in plain English, just because you're a stud, don't think that you can be an ass.

Title - a javascript snippet to control the document title

August 22, 2011
0 comments JavaScript

This is a piece of Javascript code I use on "Kwissle" to make the document title change temporarily. Other people might find it useful too.

Code looks like this:


var Title = (function() {
 var current_title = document.title
   , timer;

 return {
    showTemporarily: function (msg, msec) {
      msec = typeof(msec) !== 'undefined' ? msec : 3000;
      if (msec < 100) msec *= 1000;
      if (timer) {
        clearTimeout(timer);
      }
      document.title = msg;
      timer = setTimeout(function() {
        document.title = current_title;
      }, msec);
    }
 }
})();

Demo here

EmailInput HTML5 friendly for Django

August 2, 2011
6 comments Django

Suppose you have a Django app with a login where people can only log in with their email address. Then use this widget on your login form:


## The input widget class
class EmailInput(forms.widgets.Input):
   input_type = 'email'

   def render(self, name, value, attrs=None):
       if attrs is None:
           attrs = {}
       attrs.update(dict(autocorrect='off',
                         autocapitalize='off',
                         spellcheck='false'))
       return super(EmailInput, self).render(name, value, attrs=attrs)

## Example usage
class AuthenticationForm(django.contrib.auth.forms.AuthenticationForm):
   """override the authentication form because we use the email address as the
   key to authentication."""
   # allows for using email to log in
   username = forms.CharField(label="Username", max_length=75,
                              widget=EmailInput())
   rememberme = forms.BooleanField(label="Remember me", required=False)

EmailInput HTML5 friendly for Django This input field does some cool stuff in the browser such as automatic validation in the browser as seen in this screenshot here.

More importantly it fixes a very annoying problem when surfing on a smartphone or a tablet like the iPad. As I'm about to type "someusername@mozilla.com" it first wants to start capitalized and which might fail the login. Also if the email address contains a word that it wants to correct like ("mozilla" -> "Mozilla") you have to click the little correct tooltip to tell the input is correct in verbatim.

Note to Djangonauts who want to use this and have a dual authentication backend that takes both usernames and email addresses, this form will make it impossible to log in as something called "admin" for example.

A blog comment spam solution: Retalition!

July 31, 2011
0 comments Wondering

You know all those blog comments that slip through our blog commenting filters. The ones written by humans who pass the captcha tests. The ones who pretend to say something generic but really just want to put a link in to their viagra|laptops|watches|meds selling sites. Yes, those assholes.

What we do is to take their links and stick them on the worlds most spam full website which one of us set up. Access to edit it can be controlled by a ring of trust amongst web developers and tech savvy blog owners. Because being featured on a site like that is going to be noticed by Google as a terrible spam site and therefore flag those URLs as spam thus blocking them from searches and thus ruining it for their owners and thus them giving up on spamming our blogs.

Perhaps someone at Google like Matt Cutts could weigh in and tell us if it's feasible. Perhaps Google already have some tool where individuals can flag individual URLs but that wouldn't have the group-power of something like my idea.

For those familiar with how the Linux Kernal is developed with a distributed source code tool, this could work the same with github patches and layered layers of trust all the way up to Linus Torvalds.

Is it a completely bonkers idea?

A taste of the Django on inside Mozilla, Sheriffs Duty

July 22, 2011
0 comments Django

A taste of the Django on inside Mozilla, Sheriffs Duty One of the many great things about working for Mozilla is that everything we do is Open Source. Even our wiki is open (however we have an internal wiki for corporation boring stuff such as meeting rooms, HR etc.)

Last week I wrote an internal application for Mozilla's build engineers. Essentially it's a roster that lists one user per day and it's helped by being visualized as a calendar and as a vCal export. It's very unlikely that anybody outside Mozilla will find this particularly useful. But who knows, perhaps other companies have needs to take turns to sheriff build machines.

Anyway, the project was easy to write because we have something called Playdoh. It's a set of nifty and useful settings and a folder structure and it comes with a submodule called "playdoh-lib" which is stuffed with lots of useful packages that you'll most likely want to use. If you browse Playdoh on Github it might look like a lot of stuff but after a second look you'll see that there's actually almost no code. So don't you dare to play the "bloat card"! :)

What this app uses is TastyPie for the REST API which was awesome by the way.

For the authentication I used django-auth-ldap and some custom classes because at Mozilla we use email addresses instead of usernames.

To make the vCal export I use VObject which was easy to work with but has some usual syntax in places.

Jinja was used for the template rendering and it meant I had to do some tricks to use the django.contrib.auth.views.login view but with my templates. Might be worth looking into if people are interested.

The code has 98% test coverage but I had to upgrade to the latest nose to be able to run test coverage on app modules that have similar names to modules in the standard lib.

Comparing Google Closure with UglifyJS

July 10, 2011
17 comments JavaScript

On Kwissle I'm using Google Closure Compiler to minify all Javascript files. It's fine but because it's java and because I'm running this on a struggling EC2 micro instance the CPU goes up to 99% for about 10 seconds when it does the closure compilation. Sucks!

So, I threw UglifyJS into the mix and instead of replacing the Closure compiler I added it so it runs alongside but I obviously only keep one of the outputs.

Here is the log output when I run it on my MacbookPro:


MAKING ./static/js/account.js
UglifyJS took 0.0866 seconds to compress 3066 bytes into 1304 (42.5%)
Closure took 1.2365 seconds to compress 3066 bytes into 1225 (40.0%)
MAKING ./static/js/ext/jquery.cookie.js
UglifyJS took 0.0843 seconds to compress 3655 bytes into 3009 (82.3%)
Closure took 1.3472 seconds to compress 3655 bytes into 4086 (111.8%)
MAKING ./static/js/ext/jquery.tipsy.js
UglifyJS took 0.1029 seconds to compress 7527 bytes into 3581 (47.6%)
Closure took 1.3062 seconds to compress 7527 bytes into 3425 (45.5%)
MAKING ./static/js/maxlength_countdown.js
UglifyJS took 0.082 seconds to compress 1502 bytes into 1033 (68.8%)
Closure took 1.2159 seconds to compress 1502 bytes into 853 (56.8%)
MAKING ./static/js/ext/socket.io-0.6.3.js
UglifyJS took 0.299 seconds to compress 76870 bytes into 30787 (40.1%)
Closure took 2.4817 seconds to compress 76870 bytes into 30628 (39.8%)
MAKING ./static/js/scoreboard.js
UglifyJS took 0.084 seconds to compress 2768 bytes into 1239 (44.8%)
Closure took 1.2512 seconds to compress 2768 bytes into 1167 (42.2%)
MAKING ./static/js/rumbler.js
UglifyJS took 0.0872 seconds to compress 3087 bytes into 1384 (44.8%)
Closure took 1.2587 seconds to compress 3087 bytes into 1235 (40.0%)
MAKING ./static/js/ext/shortcut.js
UglifyJS took 0.0987 seconds to compress 5796 bytes into 2537 (43.8%)
Closure took 1.3231 seconds to compress 5796 bytes into 2410 (41.6%)
MAKING ./static/js/play.js
UglifyJS took 0.1483 seconds to compress 18473 bytes into 10592 (57.3%)
Closure took 1.4497 seconds to compress 18473 bytes into 10703 (57.9%)
MAKING ./static/js/playsound.js
UglifyJS took 0.0824 seconds to compress 1205 bytes into 869 (72.1%)
Closure took 1.2335 seconds to compress 1205 bytes into 873 (72.4%)  

(Note here that for the file ./static/js/ext/jquery.cookie.js Closure failed and when it fails it leaves the code as is and prepends it with a copy about the error from the stdout. that's why it's greater than 100% on that file)

Here are the averages of those numbers:


AVERAGE TIME: (lower is better)
  * UglifyJS: 0.11554 seconds
  * Closure: 1.41037 seconds

AVERAGE REDUCTION: (higher is better)
  * UglifyJS: 45.6% 
  * Closure: 51.5%

(I'm skipping the file that Closure failed to minify)

So, what does that mean in bytes? These are the source Javascript files for two pages but the total is 123949.0 bytes. With Closure that saves me 63833.7 bytes (62 kbytes) whereas UglifyJS only saves me 57760.2 bytes (56 kbytes) of bandwidth.

Discussion...

The fact that Closure fails on one file is a real bummer. I'm not even using the advanced options here.

UglifyJS doesn't save as many bytes as Closure does. This is potentially important because after all, minification process happens only once per revision of the original file but might be served hundreds or millions of times.

Because I run my minifications on-the-fly it does matter to me that UglifyJS is 1220% faster.

It's rare but I have observed twice that the minified Javascript from Closure has actually broken my code. I default to suspect that's my fault for not making the code "minifyable" enough (e.g. too many global variables).

I've just noticed that I'm relying on files that almost never change (e.g. jquery.tipsy.js). I might as well create a ...min.js versions of them and add them to the repository.

In conclusion...

Because of the convenience of UglifyJS being so much faster and that it doesn't choke on that jquery.cookie.js file I'm going switch to UglifyJS for the moment. The remaining bytes that I don't save become insignificant if you add the gzip effect and compared to images the bandwidth total is quite insignificant.

UPDATE (JAN 2016)

I wrote a follow-up post comparing UglifyJS2 with Closure Compiler with the advanced option.

Slides about Kwissle from yesterdays London Python Dojo

July 8, 2011
0 comments Python

Since this was published, I've abandoned the kwissle.com domain name. So no link no more.

Here are the slides from yesterday's London Python Dojo event.

I presented and demo'ed "Kwissle" to my fellow Python London friends and focused a lot on the technology but also tried to plug the game a bit.

Having seen that there's a lot of interest in "socket" related web applications about I thought this was a good chance to say that you don't need NodeJS and that tornadio is a great framework for that.

Chinese tea sampler pack now on sale

June 16, 2011
0 comments Misc. links

Chinese tea sampler pack now on sale My good friend Chris who runs the Min River Tea Farm has this week launched a new product: Sampler Pack Chinese tea

This is brilliant because if you, like me, love some good green teas but don't want to buy a whole bag yet, then get a cute little sampler and decide which of his teas you like the most. Or, for £9 you can buy a couple of these cute little tubs and give away as gifts. That's what I'm going to do.

At the moment you have to be in Europe to be able to order these (delivery done by Amazon UK) but if you're outside of Europe ping them and ask if something can be arrange.