hastebinit - quickly paste snippets into hastebin.com

October 11, 2012
9 comments Python, Linux

I'm quite fond of hastebin.com. It's fast. It's reliable. And it's got nice keyboard shortcuts that work for my taste.

So, I created a little program to quickly throw things into hastebin. You can have one too:

First create ~/bin/hastebinit and paste in:


#!/usr/bin/python

import urllib2
import os
import json

URL = 'http://hastebin.com/documents'

def run(*args):
    if args:
        content = [open(x).read() for x in args]
        extensions = [os.path.splitext(x)[1] for x in args]
    else:
        content = [sys.stdin.read()]
        extensions = [None]

    for i, each in enumerate(content):
        req = urllib2.Request(URL, each)
        response = urllib2.urlopen(req)
        the_page = response.read()
        key = json.loads(the_page)['key']
        url = "http://hastebin.com/%s" % key
        if extensions[i]:
            url += extensions[i]
        print url


if __name__ == '__main__':
    import sys
    sys.exit(run(*sys.argv[1:]))

Then run: chmod +x ~/bin/hastebinit

Now you can do things like:

$ cat ~/myfile | hastebinit
$ hastebinit < ~/myfile
$ hastebinit ~/myfile myotherfile

Hopefully it'll one day help at least one more soul out there!

How I stopped worrying about IO blocking Tornado

September 18, 2012
5 comments Tornado

So, the cool thing about Tornado the Python web framework is that it's based on a single thread IO loop. Aka Eventloop. This means that you can handle high concurrency with optimal performance. However, it means that can't do things that take a long time because then you're blocking all other users.

The solution to the blocking problem is to then switch to asynchronous callbacks which means a task can churn away in the background whilst your web server can crack on with other requests. That's great but it's actually not that easy. Writing callback code in Tornado is much more pleasant than say, Node, where you actually have to "fork" off in different functions with different scope. For example, here's what it might look like:



class MyHandler(tornado.web.RequestHandler):
    @asynchronous
    @gen.engine
    def get(self):
        http_client = AsyncHTTPClient()
        response = yield gen.Task(http_client.fetch, "http://example.com")
        stuff = do_something_with_response(response)
        self.render("template.html", **stuff)

It's pretty neat but it's still work. And sometimes you just don't know if something is going to be slow or not. If it's not going to be slow (e.g. fetching a simple value from a fast local database) you don't want to do it async anyway.

Truncated! Read the rest by clicking the link below.

Introducing: League of Friends on Around The World

September 15, 2012
0 comments Web development

League of Friends
After about a month of weekend development the League of Friends is finally finished.

Usually on games like this, if it has a highscore list you might find yourself at number 3,405,912 and the people at the top of the highscore list are people you've never heard of so what's the point of comparing yourself with them?

Inviting someone by email
On Around The World, you select your own friends for your league. Everyone you invite get an email asking if they want to accept it mutually. If you want to invite someone who isn't already on Around The World, you can type in their email address and complete an email that gets sent to that friend on your behalf from Around The World.

About Peter
Also with this, you can click on any of your travelling friends and get lots more details about their progress. It doesn't reveal anything about how smart or not smart that friend is so you never have to worry about looking stupid because it never reveals with easy questions you accidentally got wrong.

Real-timify Django with SockJS

September 6, 2012
4 comments Django, JavaScript, Tornado

In yesterdays DjangoCon BDFL Keynote Adrian Holovaty called out that Django needs a Real-Time story. Well, here's a response to that: django-sockjs-tornado

Immediately after the keynote I went and found a comfortable chair and wrote this app. It's basically a django app that allows you to run a socketserver with manage.py like this:

python manage.py socketserver

Chat Demo screenshot
Now, you can use all of SockJS to write some really flashy socket apps. In Django! Using Django models and stuff. The example included shows how to write a really simple chat application using Django models. check out the whole demo here

If you're curious about SockJS read the README and here's one of many good threads about the difference between SockJS and socket.io.

The reason I could write this app so quickly was because I have already written a production app using sockjs-tornado so the concepts were familiar. However, this app has (at the time of writing) not been used in any production. So mind you it might still need some more love before you show your mom your django app with WebSockets.

django-mongokit now compatible with Django 1.4

August 11, 2012
0 comments Python

I've finally had time to sort out the django-mongokit code so it's now fully compatible with Django 1.4.

Because I don't personally use the project I've sort of got myself lost in various patches from some awesome contributors who keep it in check.

Also, thanks to Marc Abramowitz who added a tox.ini which I've now updated to also test python 2.7 and Django 1.4

Go forth and build awesome Django apps with MongoKit which is still think is the best wrapper available on PyMongo out there.

US License Plate Spotter (part 2)

August 8, 2012
0 comments JavaScript

screenshot
Last month I built a very basic mobile app using jQuery Mobile. It's still available here.

One flaw I found with jQuery Mobile is that it's a bit slow. Scrolling up and down feels sluggish. It looks pretty though.
This time around I opted instead to use Twitter Bootstrap instead. (inspired by some other blog post with a similar experience)

Also, with this version I've set up a domain name: uslicensespotter.com
The source code is available here and there are some interesting pieces of this that I'd like to delve into deeper.

Optimization

This new app feels faster and scrolling is smoother. There is still some parts where it doesn't feel snappy. The primary part is when you click one of the buttons. It works something like this:



// binding (simplified)
  $('form a.btn').click(function(event) {
    var $el = $(this);
    var id = $el.attr('id');
    if ($el.hasClass('btn-success')) {
      switch_off($el);
      State.remove(id);
    } else {
      switch_on($el, new Date());
      State.add(id);
    }
    return false;
  });


// the switch_on function
  function switch_on($el, timestamp) {
    $el.addClass('btn-success');
    $('i', $el).addClass('icon-white').removeClass('icon-remove').addClass('icon-check');
    $('span', $el).text(timeSince(timestamp));
  }

There are two potentials for making this faster.

  1. Instead of using element.click(...) I can use touch events which supposedly fire better thus starting the actual animation earlier.
  2. Instead of modifying the element (adding a class, adding a class, removing a class, adding a class, changing the text) what I could do is to remove the element from the DOM, manipulate it and then re-insert it back into the DOM. That way the (mobile) browser doesn't have to re-render after each manipulation.

My next task is to apply both of these ideas and try to somehow keep and eye on which makes the biggest impact.

Deployment

One neat new feature with this new code is the deployment script, build.py. The actual code might be a bit of a mess but the way it works for me is great.

What I do is that I work on the app by editing dev.html and when I deploy I run python build.py and it opens dev.html and concatenates and minifies all CSS and Javascript ready for ideal caching. This reduces the static assets down to just three files weighing only 78Kb altogether.

(Note: even more can be done such as switching to ZeptoJS and removing bootstrap stuff I don't need with a custom download)

What the script does is that it parses the dev.html template with lxml and applies whatever transformations it needs to do such as generating the Apple touch images eg. this one

I think a lot of mobile web developers can benefit from this script. Maybe you don't want to copy it verbatim but you might want to copy it and do the transformations you want to do.

Beach volleyball bums

August 2, 2012
2 comments Misc. links

Bums
My good friend @jonanmary brought this very amusing tweet to my attention:

I know you've all seen this, but it's awesome anyway: Photographing other sports like beach volleyball metro.us/newyork/sports... #olympics

FIVB

That's brilliant! The thing is, if you didn't already know it, beach volleyball gals are mandated to wear those speedos looking things they wear. What's even more funny is something another good friend, @trollkip on twitter point:

@jonanmary @peterbe out of interest, earlier, I looked up who makes these rules. Governing body: fivb.org/EN/FIVB/Board_ Yeah.

UPDATE

Apparently, the rule about what beach volleyball players have to wear changed recently
I love me some tanned sexy lady-skin but don't be an asshole about it. Let her choose.

Is Nginx obsolete now that we have Amazon CloudFront?

July 28, 2012
24 comments Web development

About 5 years ago I switched from Apache to Nginx. And with that switch I could practically stop stabbing my feet with HTTP accelerators like Squid and Varnish because Nginx serves files from the filesystem both faster and more efficient than the accelerators. And, it's one less moving part that can go wrong.

Then in late 2010 Amazon introduced Custom Origins on their Amazon CloudFront CDN service. Compared to other competing CDNs I guess CloudFront loses some benchmarks and win some others. Nevertheless, network latency is the speed-freaks biggest enemy and CDNs are awesome.

With Custom Origin all you do is tell CloudFront to act as a "proxy". It takes and URL and replaces the domain name to go and fetch the original from your own server. For example...

  1. You prepare http://mydomain.com/static/foo.css
  2. You configure your CloudFront get your new domain (aka. "Distribution")
  3. You request http://efac1bef32rf3c.cloudfront.net/static/foo.css
  4. CloudFront fetches the resource from http://mydomain.com/static/foo.css and saves a copy
  5. CloudFront observes which cache headers were used and repeat that. Forever.

So, if I make my Nginx server serve /static/foo.css with:

Expires: Thu, 31 Dec 2037 23:55:55 GMT
Cache-Control: max-age=315360000
Cache-Control: public

Then CloudFront will do the same and it means it will never come back to your Nginx again. In other words, your Nginx server serves the cacheable static assets once and all other requests are just the usual HTML and JSON and whatever your backend web server spits out.

So, what does this mean? It means that we can significantly re-think they way we write code that prepares and builds static assets. Instead of a complex build or a run-time process that ultimately writes files to the filesystem we can basically do it all in run-time and not worry about speed. E.g. something like this::


# urls.py
  url(r'/static/(.*\.css)', views.serve_css)

# views.py
def serve_css(request, filename):
    response = http.HttpResponse(mimetype="text/css")
    response.setHeader('Cache-Control': 'public, max-age:315360000')
    content = open(filename).read()
    content = cssmin.cssmin(content)
    content = '/* copyright: you */\n%s' % content
    response.write(content)
    return response

That's untested code that can be vastly improved but I hope you get the idea. Obviously there are lots more things you can and should do such concatenating files.

So, what does this also mean? You don't need Nginx. At least not for serving static files faster. I've shown before that something like Nginx + uWSGI is "better" (faster and less memory) than something like Apache + mod_wsgi but oftentimes the difference is negligable.

I for one am not going to re-write all my various code I have to prepare for optimal static assets hosting but I'll definietly keep this stuff in mind. After all, there are other nifty things Nginx can do too.

By the way, here's a really good diagram that explains CloudFront

UPDATE

Want to read this in Serbian? Thank you Anja Skrba for the translation!

How to use premailer as a command line script

July 13, 2012
5 comments Python

(This post is a response to Richard Patchet's request for tips on how to actually use premailer)

First of all, premailer is a Python library that converts a document of HTML and tranforms its <style> tags into inline style attributes on the HTML itself. This comes very handy when you need to take a nicely formatted HTML newletter template and prepare it before sending because when you send HTML emails you can't reference an external .css file.

So, here's how to turn it into a command line script.

First, install, then write the script:

$ pip install premailer
$ touch ~/bin/run-premailer.py
$ chmod +x ~/bin/run-premailer.py

Now, you might want to do this differently but this should get you places:


#!/usr/bin/env python

from premailer import transform

def run(files):
    try:
        base_url = [x for x in files if x.count('://')][0]
        files.remove(base_url)
    except IndexError:
        base_url = None

    for file_ in files:
        html = open(file_).read()
        print transform(html, base_url=base_url)

if __name__ == '__main__':
    import sys
    run(sys.argv[1:])

To test it, I've made a sample HTML page that looks like this:


<html>
    <head>
        <title>Test</title>
        <style>
        h1, h2 { color:red; }
        strong {
          text-decoration:none
          }
        p { font-size:2px }
        p.footer { font-size: 1px}
        p a:link { color: blue; }
        </style>
    </head>
    <body>
        <h1>Hi!</h1>
        <p><strong>Yes!</strong></p>
        <p class="footer" style="color:red">Feetnuts</p>
        <p><a href="page2/">Go to page 2</a></p>
    </body>
</html>

Cool. So let's run it: $ run-premailer.py test.html


<html>
  <head>
    <title>Test</title>
  </head>
  <body>
        <h1 style="color:red">Hi!</h1>
        <p style="font-size:2px"><strong style="text-decoration:none">Yes!</strong></p>
        <p style="{color:red; font-size:1px} :link{color:red}">Feetnuts</p>
    <p style="font-size:2px"><a href="page2/" style=":link{color:blue}">Go to page 2</a></p>
    </body>
</html>

Note that premailer supports converting relative URLs, so let's actually using that:
$ run-premailer.py test.html https://www.peterbe.com


<html>
  <head>
    <title>Test</title>
  </head>
  <body>
        <h1 style="color:red">Hi!</h1>
        <p style="font-size:2px"><strong style="text-decoration:none">Yes!</strong></p>
        <p style="{color:red; font-size:1px} :link{color:red}">Feetnuts</p>
    <p style="font-size:2px"><a href="https://www.peterbe.com/page2/" 
     style=":link{color:blue}">Go to page 2</a></p>
    </body>
</html>

I'm sure you can think of many many ways to improve that. Mayhaps use argparse or something fancy to allow for more options. Mayhaps make it so that you can supply named .css files on the command line that get automagically inserted on the fly.

US License Plate Spotter (part 1)

July 9, 2012
4 comments JavaScript

This is part 1 in, hopefully, a series of blog articles about developing mobile apps with Javascript.

Screenshot
My app that I'm going to build is called "US License Plate Spotter". A dead-simple app where you tick off each US state once you see it.

In case you didn't know, in the USA, you mostly see "local" license plates because if you for example buy a car in Micigan but move to California, after about 3 months you have to re-license it with plates from the state you're living in. So here where I live, in California, you mostly see "California plates" but every now and then you see other plates such as Nevada, Washington or Floria. The further away, the less likely to be spotted.

Anyway, the first version is available right here: https://www.peterbe.com/uslicenseplates/index.html Code is on Github
It works best in smartphones like an Android or iOS app since it's built with jQuery Mobile

This is about 2 hours of work which is pretty quick but it was easy because I've used jQuery Mobile a lot in the past and this was more or less just getting familiar with the recent changes.
The app works fine and I even used it this last weekend to keep track of new license plates since Friday.

Next steps:

  1. Polish it a bit more with an icon, an about page and maybe add the date it was spotted
  2. Add "cloud storage" (at the moment it uses localStorage) using the Facebook API
  3. Compile an Android and iOS version with PhoneGap and see if I can launch it in some app stores.

UPDATE

Here's part 2 in the series.