Peterbe.com

At Euro DjangoCon I met lots of people and talked a lot about MongoDB as the backend. I even did a presentation on the subject which led to a lot of people asking me more questions about MongoDB.

I did mention to some people that one of the drawbacks of using MongoDB which doesn't have transactions is that you have to create and destroy the collections (like SQL tables) each time for every single test runs. I thought this was slow. It's not

Today I've been doing some more profiling and testing and debugging and I can conclude that it's not a problem. Creating the database has a slight delay but it's something you only have to do once and actually it's very fast. Here's how I tear down the collections in between each test:

class BaseTest(TestCase):

   def tearDown(self):
       for name in self.database.collection_names():
           if name not in ('system.indexes',):
               self.database.drop_collection(name)

For example, running test of one of my apps looks like this:

$ ./manage.py test myapp
...........lots.............
----------------------------------------------------------------------
Ran 55 tests in 3.024s

So, don't fear writing lots of individual unit tests. MongoDB will not slow you down.

Straight from DjangoCon 2010 here in Berlin. Slides from my talk on "Using MongoDB in your Django app - implications and benefits" are available as a HTML5 web page so you'll need one of those fancy browsers like Chrome to be able to view it. Sorry.

django-mongokit is the project you want to use if you want to connect your Django project to your MongoDB database via the pymongo Python wrapper. An alternative (dare I say competing alternative) is MongoEngine which is bridge between Django and straight to pymongo. The immediate difference you notice is the syntax. django-mongokit looks like MongoKit syntax and MongoEngine looks like Django ORM. They both accomplish pretty much the same thing. So, which one is fastest?

First of all, remember this? where I showed how django-mongokit sped past the SQL ORM like a lightning bullet. Well appears MongoEngine is even faster.

That's an average of 23% faster for all three operations!

The lovely people of Packt Publishing asked me to review Karen Tracey's latest book Django 1.1 Testing and Debugging.

I didn't actually read the book but rather skimmed it, apart from some selected parts and from what I read it's obvious that Karen has an ability to write to people who are not experts on the subject. Years of being a top contributor on the Django users mailing list must have something to do with it.

But here's the cracker. I didn't learn anything from this book (actually, I wasn't aware of the pp command in the pdb debugger). Is that a complaint about the book? No! It just means that the book was aimed at beginners and apparently I'm not a beginner any more. Great!

One thing I would have liked to see is more about testing strategy since this is something beginners often have problems with. I don't know if there even is such a word as "testing strategy" but I'm referring to the thinking behind what to test and more importantly sometimes what not to test. Beginners have a tendency to write tests for the most specific things and thus spending all their time assuring the most unrealistic scenarios are covered. Also, a lot of beginner tests I see check basic things like types which the semi-compiler will just automatically cover for you. Perhaps for a beginner, just getting some tests up and running this is a big step forward.

I'm a little bit disappointed that my lovely gorun wasn't mentioned in the book :) Perhaps the next version Karen?

In lack of a fancier solution here's how I solved a problem of knowing who was logged in when an error occurred. I'm building a Intranet like system for a close group of people and if an error occurs I get an email that reminds me to add more tests. So I fix the bugs and upgrade the server. But I often want to know what poor sucker was logged in at the time the exception happened so that I can email them and say something like "Hi! I noticed your stumbled across a bug. My bad. Just wanted to let you know I've fixed that now"

So to do this I installed a silly little piece of middleware:

from django.conf import settings
class ExceptionExtraMiddleware(object):
   def process_exception(self, request, exception):
       if settings.DEBUG:
           return
       try:
           logged_in_info = ''
           if request.user and request.user.is_authenticated():
               logged_in_info = "%s" % request.user
               if request.user.email:
                   logged_in_info += ' %s' % request.user.email
               if request.user.first_name or request.user.last_name:
                   logged_in_info += ' (%s %s)' % \
                     (request.user.first_name, request.user.last_name)
           if logged_in_info:
               request.META['ERROR-X-LOGGED-IN'] = logged_in_info
       except:
           # don't make matters worse in these sensitive times
           logging.debug("Unable to debug who was logged in", exc_info=True)

This means that when I get an email with the traceback and snapshot of the request object I get this included:

...
'ERROR-X-LOGGED-IN': u'anita (Anita Test)',
...

UPDATE

The code above had a bug in it. Doing an if on request.user will return true even if there is no logged in user. The safest thing is to change it to:

if request.user and request.user.is_authenticated():

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

fcgi (threaded)              640 r/s
fcgi (prefork 4 processors)  240 r/s (*)
gunicorn (2 workers)         1100 r/s
gunicorn (5 workers)         1300 r/s
gunicorn (10 workers)        1200 r/s (?!?)
uwsgi (2 workers)            1800 r/s
uwsgi (5 workers)            2100 r/s
uwsgi (10 workers)           2300 r/s

(* this made my computer exceptionally sluggish as CPU when through the roof)

I'm the proud creator of django-static which is a Django app that takes care of how you serve your static media the best way possible. Although some of these things are subjective generally this is the ideal checklist of servicing your static media:

1. Cache headers must be set to infinity
2. URLs must be unique so that browsers never have to depend on refreshing
3. The developer (who decided which media to include) should not have to worry himself with deployment
4. The developer/artist (who makes the media) should not have to worry himself with deployment
5. All Javascript and CSS must be whitespace optimized in a safe way and served with Gzip
6. All images referenced inside CSS should be taken care of too
7. It must be possible to combine multiple resources of Javascript or CSS into one
8. It must be possible to easily test production deployment in development environment without too much effort
9. A sysadmin shouldn't have to understand a developers Django application
10. A development environment must be unhindered by this optimization
11. Processing overhead of must be kept to a minimum
12. Must be possible to easily say which resources can be whitespace optimized and which can not

Following on from yesterday's blog about How and why to use django-mongokit I extended the exampleproject which is inside the django-mongokit project with another app called exampleapp_sql which does the same thing as the exampleapp but does it with SQL instead. Then I added a very simple benchmarker app in the same project and wrote three functions:

1. One to create 10/100/500/1000 instances of my class
2. One to edit one field of all 10/100/500/1000 instances
3. One to delete each of the 10/100/500/1000 instances

Here I'm going to explain how to combine Django and MongoDB using MongoKit and django-mongokit.

MongoDB is a document store built for high speed and high concurrency with a very good redundancy story. It's an alternative to relational databases (e.g. MySQL) that is what Django is tightly coupled with in it's ORM (Object Relation Mapping) and what it's called now is ODM (Object Document Mapping) in lack of a better acronym. That's where MongoKit comes in. It's written in Python and it connects to the MongoDB database using a library called pymongo and it turns data from the MongoDB and turns it into instances of classes you have defined. MongoKit has nothing to do with Django. That's where django-mongokit comes in. Written by yours truly.

If you, like me, have various projects that do things like OAuth on Twitter or Google or you have a development site that goes to PayPal. So you're doing some Django development on http://localhost:8000/foo and click, for example, to do an OAuth on Twitter with an app you have there. Then Twitter will redirect you back to the live site with which you've set it up. But you're doing local development so you want to go back to http://localhost:8080/... instead.

Add this bookmarklet: to localhost:8000 to your browser Bookmarks toolbar and it does exactly that.

Here's its code in more verbose form:

(function() { 
   a = function(){
     location.href = window.location.href.replace(/http:\/\/[^\/]+\//,
            'http://localhost:8000/')
   };
   if (/Firefox/.test(navigator.userAgent)) { 
     setTimeout(a,0)
   } else {
      a()
   }
 })()