Nasty surprise of Django cache

December 9, 2008
10 comments Django

Caching in Django absolutely rocks. Much so because of its simplicity which lowers the threshold to the height of a snowflake. However the simplicity can cause some unexpected surprises.

I ran my site and clicked around until I found a bug I wanted to fix. Then I wrote the test and ran the testrunner but I was getting some really weird behavior. The view looked like this:


@never_cache
def club_page(request, clubname):
   club = _find_club(clubname)
   if club is None:
       raise Http404('Could not find the club')

   classes = cache.get('club_page_classes')
   if classes is None:
       classes = ClubClass.objects.filter(club=club).order_by('start_time')
       cache.set('club_page_classes', classes, CACHE_TIMEOUT)
   ...

What happened (and what took me a while to figure out) was that the memcache was still active and being used when running the tests since it's only within the hour that I started running the tests with completely different data. Here's how I solved the problem:


class ViewsTestCase(unittest.TestCase):
    ...
    def test_club_page(self):
       """ test rendering the club page """
       cache.delete('club_page_classes')
       ...

There must be an easier way to patch the testrunner to reset all cache before running the tests. Anybody?

bool is instance of int in Python

December 5, 2008
15 comments Python

I lost about half an hour just moments ago debugging this and pulling out a fair amount of hair. I had some code that looked like this:


result = []
for key, value in data.items():
   if isinstance(value, int):
       result.append(dict(name=key, value=value, type='int'))
   elif isinstance(value, float):
       result.append(dict(name=key, value=value, type='float'))
   elif isinstance(value, bool):
       result.append(dict(name=key, type='bool',
                          value=value and 'true' or 'false'))
...

It looked so simple but further up the tree I never got any entries with type="bool" even though I knew there were boolean values in the dictionary.

The pitfall I fell into was this:


>>> isinstance(True, bool)
True
>>> isinstance(False, bool)
True
>>> isinstance(True, int)
True
>>> isinstance(False, int)
True

Not entirely obvious if you ask me. The solution in my case was just to change the order of the if and the elif so that bool is tested first.

Finally got rid of the system beep

November 22, 2008
2 comments Linux

I have a Thinkpad T60p which is working really well for me. On it I've run various flavors of Ubuntu and as of a couple of weeks ago I put on Ubuntu 8.10 which has been working very well too except that I didn't get any options in the Preferences menu to switch off the damn loud system beep. The beep comes through the speakers and at a much much louder volume than any other sound or music.

I tried changing thins in BIOS and I tried installing various packages hoping one of them will give me the options. Finally I've found out how to disable it:


$ sudo modprobe -r pcspkr

This tip page showed me how to put it into /etc so it's applied all the time. Thanks Andy!

domstripper - A lxml.html test project

November 20, 2008
1 comment Python

I'm just playing with the impressive lxml.html package. It makes it possible to easily work with HTML trees and manipulate them.

I had this crazy idea of a "DOM stripper" that removes all but specified elements from an HTML file. For example you want to keep the contents of the <head> tag intact but you just want to keep the <div id="content">...</div> tag thus omitting <div id="banner">...</div> and <div id="nav">...</div>. domstripper now does that. This can be used for example as a naive proxy that tranforms a bloated HTML page into a more stripped down smaller version suitable for say mobile web browsers. It's more a proof of concept that anything else.

To test you just need a virtual python environment and the right system libs to needed to install lxml. This worked for me:


$ sudo apt-get install cython libxslt1-dev zlib1g-dev libxml2-dev
$ cd /tmp
$ virtualenv --no-site-packages testenv
$ cd testenv
$ source bin/activate
$ easy_install domstripper

Now you can use it like this:


>>> from domstripper import domstripper
>>> help(domstripper)
...
>>> domstripper('bloat.html', ['#content', 'h1.header'])
<!DOCTYPE...
...

Best to just play with it and see if makes sense. I'm not saying this is an amazing package but it goes to show what can be done with lxml.html and the extremely user friendly CSS selectors.

How to unit test the innards of a Django view function

November 15, 2008
3 comments Django

Seconds ago I got this running and haven't yet fully understood what I've just done but the results are exactly what I need and this is going to be great.

Basically, in Django you have views like this:


def _render(template, data, request):
   return render_to_response(template, data,
             context_instance=RequestContext(request))

@login_required
def club_page(request, year):
   variable1 = year / 4
   variable2 = variable1 * 2
   return _render("foo.html", locals(), request)

Now in my unit tests I don't want to have to call the view function and then have to dissect the resulting HTML just to figure out if the view function prepared the correct variables. So, here's my solution to this problem:

Truncated! Read the rest by clicking the link below.

Comic-Con 2008 photos

October 29, 2008
0 comments Misc. links

Comic-Con 2008 photos I'm not a comic book kinda guy at all and I don't know who half of these costumes are supposed to be but it's nevertheless very impressive. Looking through the thumbnails you spot a couple of people who seem to take it very seriously but a large majority seem to just enjoy themselves. Just goes to show that people always like to dress up.

Django vs. Java

October 25, 2008
1 comment Django

From the django-users mailinglist which I'm becoming more and more helpful in:


> Could you share approximately how big your project is? I know it's
> hard to find a real measure for this, but how about number of database
> tables?

A project I worked on over the summer used a Database that was 130
tables, and getting 1gb updates every 2 minutes. I was witting a new
web app to do calculations on the data and the company wanted to use
Java since thats what they knew best and had spend huge amounts of
money (1 mil +) to support with Sun Servers, and such. But I knew
python and django would be a better fit for this particular app, but
the boss wouldnt listen. So we had 10 Developers working on the Java
version (Including me) and over 3 months we got it about 85% done,
though it had no unit tests. During the same three months, I worked on
my own time after work and basically had no life for the whole time, I
was able to get the web app 100% complete with unit tests. That
convinced my boss that Django was a good fit.

The site is an internal app that I cannot give access to (And I
actually had to get permission to give what info I have), but I can
say that Django is a suitable framework for what you are looking for. 

Christ! 10 developers and no unit test!? Someone should remind them that you don't write unit tests for your bosses pleasure but for your own sanity and productivity.

I know that this quote is totally unscientific since Dj, as he says, can't back it up but it's a story interesting enough.

Flash advert hell

October 24, 2008
1 comment Web development

Flash advert hell I actually don't mind a bit of adverts on websites but sometimes it just gets too much. On this page I couldn't even read the text since as I scroll down the huge ad over the text scrolls with the page.

Feels greedy like they've just thrown more and more ads in without thinking about the original design.

World Plone Day here in London, England

October 21, 2008
0 comments Plone

On the 7th of November my company, Fry-IT is hosting the London arm of the World Plone Day in our office. It's basically an event for Plone developers, Plone companies and other people interested in Plone to meet up and promote or learn or share something about Plone. To quote the "FAQ":

"What is the goal of the World Plone Day?
The World Plone Day (WPD) is a worldwide event. Our goal is to promote and educate the worldwide public about of the benefits of using Plone in education, government, ngos, and in business."

The "invite" is here. I'm really looking forward to meeting everybody and put faces to people whose blogs and mailing lists posts I read. My colleague Lukasz has a nice map to our office and I recommend that you contact him to say you're coming.