Peterbe.com

This cost me some hair-pulling today as I was trying to write a custom test runner for a Django project I'm working on that creates a test Xapian database just for running the tests. Basically, you can't do this:

os.mkdir(database_file_path)

Because if you do you end up getting these strange DatabaseOpeningError exceptions. So, here's how you do it:

import xapian
xapian.WritableDatabase(database_file_path,
                        xapian.DB_CREATE_OR_OPEN)

Hopefully by blogging about this some other poor coder will save some time.

YUI Compressor apparently supports compressing CSS. Cool! I had to try it and what's even more cool is that it's the only other CSS minifier/compressor that doesn't choke on CSS hacks (the ones I tried). The only other CSS minifier/compressor is my very own slimmer. So, let's see what the difference is.

Running the YUI Compressor 10 times on a relatively small file takes 0.3 seconds on average. Running the same with python 2.6 and slimmer.css_slimmer takes 0.1 seconds on average. I think most of this time is spent loading the jar file than the actual time of running the compression.

I have some code that checks that a file is treated differently if some time has passed. In other words, it reacts if the modification time is more than it was before. The code uses os.stat(filename)[stat.ST_MTIME] to do this. The challenge was to mock os.stat so that I can pretend some time has passed without having to wait. Here was my first solution which made running many tests sloooow:

def test_file_changed(self):
    filename = 'foo.txt'
    expect_timestamp = int(time.time())
    ...run the unit test with 'expect_timestamp'...
    time.sleep(1)
    expect_timestamp += 1
    ...run the unit test with 'expect_timestamp'...

So, here's how I mock os.stat to avoid having to do the time.sleep(1) in the test:

def test_file_changed(self):
    filename = 'foo.txt'
    expect_timestamp = int(time.time())
    ...run the unit test with 'expect_timestamp'...
    import os
    from posix import stat_result
    def fake_stat(arg):
        if arg == filename:
            faked = list(orig_os_stat(arg))
            faked[stat.ST_MTIME] = faked[stat.ST_MTIME] + 1
            return stat_result(faked)
        else:
            return orig_os_stat(arg)
    orig_os_stat = os.stat
    os.stat = fake_stat

    expect_timestamp += 1
    ...run the unit test with 'expect_timestamp'...

I hope this helps someone else who's trying to do the same. It took me some time to figure out that os.stat is used by lots of various sub routines in the os module so it's important to only mock the relevant argument otherwise you might get unexpected problems.

One really annoying thing about PIL is that it's importable as Image and PIL. It leads me and other newbies to think if it's different. I don't want choices:

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
>>> import Image
>>> ?

When PIL/Image is put into standard lib, can we call the module: imaging?

I've just finished my first Twitter app. It's basically a just a about using OAuth to allow people to sign up to KungFuPeople.com without having to pick yet another password.

I simply took the oauth.py module by Leah Culver and wrapped it with some useful functions taken from a similar Twitter app we've done at work.

Unlike other Twitter apps for this one I'm using Twitter solely for handling authorization and authentication. That means that it has to work with the existing user + profile functionality but just side-step the sign up and login.

Next goal: Google OAuth

JSMIN - The Javascript Minifier is written in C and it does an excellent job of minifying Javascript code. After all, Douglas Crockford wrote it. I noticed there's a Python implementation of it so I wanted to see how it stacks up against my slimmer which is also written in Python.

For sake of argument I compiled the C version and ran that in my little benchmark and did so by using the subprocess module. Also, for the sake of comparison I threw in a run with YUI Compressor. Here are some quick results:

On js/signup-core.js
--------------------
js_slimmer
from 9708 to 6905 in 0.0245039463043 seconds
jsmin
from 9708 to 6720 in 0.0850019454956 seconds
jsmin.c
from 9708 to 6721 in 0.0026159286499 seconds
yuicompressor
from 9708 to 6102 in 0.914173126221 seconds

On js/zoom.js 
-------------
js_slimmer
from 5920 to 3712 in 0.0106379985809 seconds
jsmin
from 5920 to 3582 in 0.0582370758057 seconds
jsmin.c
from 5920 to 3583 in 0.00282216072083 seconds
yuicompressor
from 5920 to 2771 in 0.839382171631 seconds

On js/diypack.js
----------------
js_slimmer
from 21559 to 14059 in 0.0409741401672 seconds
jsmin
from 21559 to 13655 in 0.177556037903 seconds
jsmin.c
from 21559 to 13656 in 0.00346994400024 seconds
yuicompressor
from 21559 to 11638 in 0.891603946686 seconds

So, roughly, slimmer is 4 times faster than jsmin.py but fails to minify a couple of bytes. jsmin.c is about 6 times faster than slimmer.py but is awkward since it's in C. I guess jsmin.c is the way forward when you want speed and the best result. slimmer has the advantage of being all in python and PyPi and contains functions for CSS, HTML and XHTML as well.

It's clear the YUI Compressor does a wicked job at minifying but by running a .jar file every time in a subprocess is crazily slow if that matters for you.

If you're on the python-uk mailing list you will have already seen this but if you're not, here we go.

Fry-IT is hosting a Code Dojo in our offices. It's free and open to anyone. My colleague Nicholas has written up a little bit about what a Code Dojo is which should get you excited.

Details are available on this page which is also the place to go to secure your place. Currently there are 12 people who say they're coming and we've decided to cap the geek influx to 30 people.

Cheers,
Peter-san

I've been experimenting with the new Google Reverse Geocoding which allows you to get the location name and country and stuff from a latitude/longitude coordinate.

What I've been doing is comparing this with GeoNames. GeoNames is available from geopy in the reverse-geocode branch.

I wrote down a list of about 15 lat/long points and the result I expect from them (taken from an existing app I'm contemplating switching to Google Reverse Geocoding for) and ran a batch of timed tests on. These results might satisfy the impatient:

FAILURES:
geonames_json        0
google               0
geonames             12

TOTAL TIMES:
geonames_json        2.43582677841        0.143283928142 seconds/request
google               2.24999976158        0.132352927152 seconds/request
geonames             1.78063511848        0.104743242264 seconds/request

By popular demand I've made my little pyinotify wrapper available for download. It's nothing fancy really but damn useful and productive.

It relies on inotify (so you're stuffed on OSX and Windows) which makes it very fast and efficient (as opposed to periodic polling and file modification time comparisons).

At the moment it's actually quite generic for any command and any file but I'm hoping to take this to the next level with some magic dust that automatically only runs unit tests that fail or something. We'll see what happens.

Gun to your head; what would it take to make setuptools as a package author easy to use?

I've spent far too long time today trying to create a package for a little piece of code I've written. Because I can never remember all the bizarre options and commands to setup.py I tried to do it by following Tarek Ziade's wonderful Expert Python Programming but I still got stuck.

Granted, I did not read the f**n manual. Why should I have to? I've got more important things to do such as eating cookies and watching tv.