Peterbe.com

In yesterdays DjangoCon BDFL Keynote Adrian Holovaty called out that Django needs a Real-Time story. Well, here's a response to that: django-sockjs-tornado

Immediately after the keynote I went and found a comfortable chair and wrote this app. It's basically a django app that allows you to run a socketserver with manage.py like this:

python manage.py socketserver

Now, you can use all of SockJS to write some really flashy socket apps. In Django! Using Django models and stuff. The example included shows how to write a really simple chat application using Django models. check out the whole demo here

If you're curious about SockJS read the README and here's one of many good threads about the difference between SockJS and socket.io.

The reason I could write this app so quickly was because I have already written a production app using sockjs-tornado so the concepts were familiar. However, this app has (at the time of writing) not been used in any production. So mind you it might still need some more love before you show your mom your django app with WebSockets.

I've finally had time to sort out the django-mongokit code so it's now fully compatible with Django 1.4.

Because I don't personally use the project I've sort of got myself lost in various patches from some awesome contributors who keep it in check.

Also, thanks to Marc Abramowitz who added a tox.ini which I've now updated to also test python 2.7 and Django 1.4

Go forth and build awesome Django apps with MongoKit which is still think is the best wrapper available on PyMongo out there.

Last month I built a very basic mobile app using jQuery Mobile. It's still available here.

One flaw I found with jQuery Mobile is that it's a bit slow. Scrolling up and down feels sluggish. It looks pretty though.
This time around I opted instead to use Twitter Bootstrap instead. (inspired by some other blog post with a similar experience)

Also, with this version I've set up a domain name: uslicensespotter.com
The source code is available here and there are some interesting pieces of this that I'd like to delve into deeper.

Optimization

This new app feels faster and scrolling is smoother. There is still some parts where it doesn't feel snappy. The primary part is when you click one of the buttons. It works something like this:

// binding (simplified)
  $('form a.btn').click(function(event) {
    var $el = $(this);
    var id = $el.attr('id');
    if ($el.hasClass('btn-success')) {
      switch_off($el);
      State.remove(id);
    } else {
      switch_on($el, new Date());
      State.add(id);
    }
    return false;
  });


// the switch_on function
  function switch_on($el, timestamp) {
    $el.addClass('btn-success');
    $('i', $el).addClass('icon-white').removeClass('icon-remove').addClass('icon-check');
    $('span', $el).text(timeSince(timestamp));
  }

There are two potentials for making this faster.

1. Instead of using element.click(...) I can use touch events which supposedly fire better thus starting the actual animation earlier.
2. Instead of modifying the element (adding a class, adding a class, removing a class, adding a class, changing the text) what I could do is to remove the element from the DOM, manipulate it and then re-insert it back into the DOM. That way the (mobile) browser doesn't have to re-render after each manipulation.

My next task is to apply both of these ideas and try to somehow keep and eye on which makes the biggest impact.

Deployment

One neat new feature with this new code is the deployment script, build.py. The actual code might be a bit of a mess but the way it works for me is great.

What I do is that I work on the app by editing dev.html and when I deploy I run python build.py and it opens dev.html and concatenates and minifies all CSS and Javascript ready for ideal caching. This reduces the static assets down to just three files weighing only 78Kb altogether.

(Note: even more can be done such as switching to ZeptoJS and removing bootstrap stuff I don't need with a custom download)

What the script does is that it parses the dev.html template with lxml and applies whatever transformations it needs to do such as generating the Apple touch images eg. this one

I think a lot of mobile web developers can benefit from this script. Maybe you don't want to copy it verbatim but you might want to copy it and do the transformations you want to do.

(This post is a response to Richard Patchet's request for tips on how to actually use premailer)

First of all, premailer is a Python library that converts a document of HTML and tranforms its <style> tags into inline style attributes on the HTML itself. This comes very handy when you need to take a nicely formatted HTML newletter template and prepare it before sending because when you send HTML emails you can't reference an external .css file.

So, here's how to turn it into a command line script.

First, install, then write the script:

$ pip install premailer
$ touch ~/bin/run-premailer.py
$ chmod +x ~/bin/run-premailer.py

Now, you might want to do this differently but this should get you places:

#!/usr/bin/env python

from premailer import transform

def run(files):
    try:
        base_url = [x for x in files if x.count('://')][0]
        files.remove(base_url)
    except IndexError:
        base_url = None

    for file_ in files:
        html = open(file_).read()
        print transform(html, base_url=base_url)

if __name__ == '__main__':
    import sys
    run(sys.argv[1:])

To test it, I've made a sample HTML page that looks like this:

<html>
    <head>
        <title>Test</title>
        <style>
        h1, h2 { color:red; }
        strong {
          text-decoration:none
          }
        p { font-size:2px }
        p.footer { font-size: 1px}
        p a:link { color: blue; }
        </style>
    </head>
    <body>
        <h1>Hi!</h1>
        <p><strong>Yes!</strong></p>
        <p class="footer" style="color:red">Feetnuts</p>
        <p><a href="page2/">Go to page 2</a></p>
    </body>
</html>

Cool. So let's run it: $ run-premailer.py test.html

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
        <h1 style="color:red">Hi!</h1>
        <p style="font-size:2px"><strong style="text-decoration:none">Yes!</strong></p>
        <p style="{color:red; font-size:1px} :link{color:red}">Feetnuts</p>
    <p style="font-size:2px"><a href="page2/" style=":link{color:blue}">Go to page 2</a></p>
    </body>
</html>

Note that premailer supports converting relative URLs, so let's actually using that:
$ run-premailer.py test.html https://www.peterbe.com

<html>
  <head>
    <title>Test</title>
  </head>
  <body>
        <h1 style="color:red">Hi!</h1>
        <p style="font-size:2px"><strong style="text-decoration:none">Yes!</strong></p>
        <p style="{color:red; font-size:1px} :link{color:red}">Feetnuts</p>
    <p style="font-size:2px"><a href="https://www.peterbe.com/page2/" 
     style=":link{color:blue}">Go to page 2</a></p>
    </body>
</html>

I'm sure you can think of many many ways to improve that. Mayhaps use argparse or something fancy to allow for more options. Mayhaps make it so that you can supply named .css files on the command line that get automagically inserted on the fly.

This is part 1 in, hopefully, a series of blog articles about developing mobile apps with Javascript.

My app that I'm going to build is called "US License Plate Spotter". A dead-simple app where you tick off each US state once you see it.

In case you didn't know, in the USA, you mostly see "local" license plates because if you for example buy a car in Micigan but move to California, after about 3 months you have to re-license it with plates from the state you're living in. So here where I live, in California, you mostly see "California plates" but every now and then you see other plates such as Nevada, Washington or Floria. The further away, the less likely to be spotted.

Anyway, the first version is available right here: https://www.peterbe.com/uslicenseplates/index.html Code is on Github
It works best in smartphones like an Android or iOS app since it's built with jQuery Mobile

This is about 2 hours of work which is pretty quick but it was easy because I've used jQuery Mobile a lot in the past and this was more or less just getting familiar with the recent changes.
The app works fine and I even used it this last weekend to keep track of new license plates since Friday.

Next steps:

1. Polish it a bit more with an icon, an about page and maybe add the date it was spotted
2. Add "cloud storage" (at the moment it uses localStorage) using the Facebook API
3. Compile an Android and iOS version with PhoneGap and see if I can launch it in some app stores.

UPDATE

Here's part 2 in the series.

The @staticmethod decorator is nothing new. In fact, it was added in version 2.2. However, it's not till now in 2012 that I have genuinely fallen in love with it.

First a quick recap to remind you how @staticmethod works.

class Printer(object):

    def __init__(self, text):
        self.text = text

    @staticmethod
    def newlines(s):
        return s.replace('\n','\r')

    def printer(self):
        return self.newlines(self.text)

p = Printer('\n\r')
assert p.printer() == '\r\r'

So, it's a function that has nothing to do with the instance but still belongs to the class. It belongs to the class from an structural point of view of the observer. Like, clearly the newlines function is related to the Printer class. The alternative is:

def newlines(s):
    return s.replace('\n','\r')

class Printer(object):

    def __init__(self, text):
        self.text = text

    def printer(self):
        return newlines(self.text)

p = Printer('\n\r')
assert p.printer() == '\r\r'

It's the exact same thing and one could argue that the function has nothing to do with the Printer class. But ask yourself (by looking at your code); how many times do you have classes with methods on them that take self as a parameter but never actually use it?

So, now for the trump card that makes it worth the effort of making it a staticmethod: object orientation. How would you do this neatly without OO?

class UNIXPrinter(Printer):

    @staticmethod
    def newlines(s):
        return s.replace('\n\r', '\n')

p = UNIXPrinter('\n\r')
assert p.printer() == '\n'  

Can you see it? It's ideal for little functions that should be domesticated by the class but have nothing to do with the instance (e.g. self). I used to think it looked like it's making a pure looking thing like something more complex that it needs to be. But now, I think it looks great!

I learned something today thanks to my colleague Axel Hecht; the difference between $element.data('foo') and $element.attr('data-foo').

Basically, the .data() getter/setter is more powerful since it can do more things. For example:

<img id="image" data-number="42">

// numbers are turned to integers
assert($('#image').data('number') + 1 == 43);
// the more rudimentary way
assert($('#image').attr('data-number') + 1 == '421');

Integers is just one thing the .data() getter is able to parse. It can do other cool things too like booleans and JSON. Check out its docs

So, why would you NOT use .data()?

One reason is that with .data(name, value) the original DOM element is not actually modified. This can cause trouble if other pieces of Javascript depend on the value of a data- attribute further along in the page rendering process.

To see it in action check out: test.html

In conclusion: just be aware of it. Feel free to use the .data() getter/setter because it's way better but be aware of the potential risks.

First of all, this technique is only really applicable to apps where there's only one big HTML template which is then shuffles, part hidden and part visible thanks to lots of Javascript. Those familiar with jQuery Mobile will have seen this.

On Around The World there are a lot of images. Majority of them you don't need to see immediately because only one screen is loaded at the time. The page structure looks like this:

<div class="section" id="page1">
  <h2>Page 1</h2>
  <img src="section-icon1.png">
</div>
<div class="section" id="page2" style="display:none">
  <h2>Page 2</h2>
  <img src="section-icon2.png">
</div>
<div class="section" id="page3" style="display:none">
  <h2>Page 3</h2>
  <img src="section-icon3.png">
</div>

So, if you load that you'll notice that your browser will download "section-icon1.png", "section-icon2.png" and "section-icon3.png" even though two of the images are not going to be displayed. Good for pre-loading the images when they're later needed but bad for the user experience since the browser will be busy downloading images rather than displaying the first visible section.

This is how I solve this; first I change the HTML to be this:

<div class="section" id="page1">
  <h2>Page 1</h2>
  <img src="." data-src="section-icon1.png" class="deferred">
</div>
<div class="section" id="page2" style="display:none">
  <h2>Page 2</h2>
  <img src="." data-src="section-icon2.png" class="deferred">
</div>
<div class="section" id="page3" style="display:none">
  <h2>Page 3</h2>
  <img src="." data-src="section-icon3.png" class="deferred">
</div>

And now for the magic that turns these img tags into real normal img tags. The truth is that the Javascript about loading individual sections is a bit more complicated but in its inner core it looks something like this:

// variable 'hash' is something like 'page2'
if ($(hash + '.section').size()) {
  $('.section:visible').hide();
  $(hash + '.section').show();
  $('img.deferred', hash).each(function() {
    var el = $(this);
    el.attr('src', el.data('src'));
    el.removeClass('deferred');
  });
  ...

It makes the HTML slightly more complicated but the end result is great. It's not just useful for the first-time load but also applicable every time someone reloads the page.

I've blogged before about how this site can easily push out over 2,000 requests/second using only 6 WSGI workers excluding latency. The reason that's possible is because the whole page(s) can be cached server-side. What actually happens is that the whole rendered HTML blob is stored in the cache server (Redis in my case) so that no database queries are needed at all.

I wanted my site to still "feel" dynamic in the sense that once you post a comment (and it's published), the page automatically invalidates the cache and thus, the user doesn't have to refresh his browser when he knows it should have changed. To accomplish this I used a hacked cache_page decorator that makes the cache key depend on the content it depends on. Here's the code I actually use today for the home page:

def _home_key_prefixer(request):
    if request.method != 'GET':
        return None
    prefix = urllib.urlencode(request.GET)
    cache_key = 'latest_comment_add_date'
    latest_date = cache.get(cache_key)
    if latest_date is None:
        # when a blog comment is posted, the blog modify_date is incremented
        latest, = (BlogItem.objects
                   .order_by('-modify_date')
                   .values('modify_date')[:1])
        latest_date = latest['modify_date'].strftime('%f')
        cache.set(cache_key, latest_date, 60 * 60)
    prefix += str(latest_date)

    try:
        redis_increment('homepage:hits', request)
    except Exception:
        logging.error('Unable to redis.zincrby', exc_info=True)

    return prefix


@cache_page_with_prefix(60 * 60, _home_key_prefixer)
def home(request, oc=None):
    ...
    try:
        redis_increment('homepage:misses', request)
    except Exception:
        logging.error('Unable to redis.zincrby', exc_info=True)
    ...

And in the models I then have this:

@receiver(post_save, sender=BlogComment)
@receiver(post_save, sender=BlogItem)
def invalidate_latest_comment_add_dates(sender, instance, **kwargs):
    cache_key = 'latest_comment_add_date'
    cache.delete(cache_key)

So this means:

• whole pages are cached for long time for fast access
• updates immediately invalidates the cache for best user experience
• no need to mess with ANY SQL caching

So, the next question is, if posting a comment means that the cache is invalidated and needs to be populated, what's the ratio of hits versus hits where the cache is cleared? Glad you asked. That's why I made this page:

www.peterbe.com/stats/

It allows me to monitor how often a new blog comment or general time-out means poor django needs to re-create the HTML using SQL.

At the time of writing, one in every 25 hits to the homepage requires the server to re-generate the page. And still the content is always fresh and relevant.

The next level of optimization would be to figure out whether a particular page update (e.g. a blog comment posting on a page that isn't featured on the home page) should or should not invalidate the home page. esp

The advantage with WebSockets (over AJAX) is basically that there's less HTTP overhead. Once the connection has been established, all future message passing is over a socket rather than new HTTP request/response calls. So, you'd assume that WebSockets can send and receive much more messages per unit time. Turns out that that's true. But there's a very bitter reality once you add latency into the mix.

So, I created a simple app that uses SockJS and an app that uses jQuery AJAX to see how they would perform under stress. Code is here. All it does is basically, send a simple data structure to the server which echos it back. As soon as the response comes back, it starts over. Over and over till it's done X number of iterations.

Here's the output when I ran this on localhost here on my laptop:

# /ajaxtest (localhost)
start!
Finished
10 iterations in 0.128 seconds meaning 78.125 messages/second
start!
Finished
100 iterations in 0.335 seconds meaning 298.507 messages/second
start!
Finished
1000 iterations in 2.934 seconds meaning 340.832 messages/second

# /socktest (localhost)
Finished
10 iterations in 0.071 seconds meaning 140.845 messages/second
start!
Finished
100 iterations in 0.071 seconds meaning 1408.451 messages/second
start!
Finished
1000 iterations in 0.466 seconds meaning 2145.923 messages/second

Wow! It's so fast that the rate doesn't even settle down. Back-of-an-envelope calculation tells me the WebSocket version is 5 times faster roughly. Again; wow!

Now reality kicks in! It's obviously unrealistic to test against localhost because it doesn't take latency into account. I.e. it doesn't take into account the long distance the data has to travel from the client to the server.

So, I deployed this test application on my server in London, England and hit it from my Firefox here in California, USA. Same number of iterations and I ran it a number of times to make sure I don't get hit by sporadic hickups on the line. Here are the results:

# /ajaxtest (sockshootout.peterbe.com)
start!
Finished
10 iterations in 2.241 seconds meaning 4.462 messages/second
start!
Finished
100 iterations in 28.006 seconds meaning 3.571 messages/second
start!
Finished
1000 iterations in 263.785 seconds meaning 3.791 messages/second

# /socktest (sockshootout.peterbe.com) 
start!
Finished
10 iterations in 5.705 seconds meaning 1.752 messages/second
start!
Finished
100 iterations in 23.283 seconds meaning 4.295 messages/second
start!
Finished
1000 iterations in 227.728 seconds meaning 4.391 messages/second

Hmm... Not so cool. WebSockets are still slightly faster but the difference is negligable. WebSockets are roughly 10-20% faster than AJAX. With that small a difference I'm sure the benchmark is going to vastly effected by other factors that make it unfair for one or the the other such as quirks in my particular browser or the slightest hickup on the line.

What can we learn from this? Well, latency kills all the fun. Also, it means that you don't necessarily need to re-write your already working AJAX heavy app just to gain speed because even though it's ever so slightly faster, the switch from AJAX to WebSocket comes with other risks and challenges such as authentication cookies, having to deal with channel concurrency, load balancing on the server etc.

Before you say it, yes I'm aware than WebSocket web apps comes with other advantages such as being able to hold on to sockets and push data at will from the server. Those are juicy benefits but massive performance boosts ain't one.

Also, I bet that writing this means that peeps will come along and punch hole in my code and my argument. Something I welcome with open arms!