Peterbe.com

Ever since we started building SnapExpense we ask people we meet how they do their business/travel expenses at their company. Almost always they say their software is annoyingly bad and it's a right pain in the ass to process their expenses. Today I heard another such horror story from a friend about: GlobalExpense (www.globalexpense.com). Apparently it's dreadfully slow to use and it never remembers or adapts to what you have entered previously.

So my friend went to the website on my computer to show me how crap it was but that didn't work because GlobalExpense doesn't work in Firefox! What?! On SnapExpense, about 30% of our visitors use Firefox (with 31% using Internet Explorer).

I'm glad to see that the competition is lagging behind. Gives me heart. Especially as I have recently added Google OpenID log in to SnapExpense so you can log in or register with your Google (or Yahoo! or any OpenID provider) account straight away.

This is going to be useful to keep handy when I move to America.

I've learned a couple of things this week on deploying my first site to use a user friendly OpenID.

My first revelation was when I realized that Google and Yahoo! have solved the usability stumbling block that you can use them as providers without having to know a personally unique URL. For example, for Yahoo! it's just http://yahoo.com which means that you don't need to offer a cryptic URL form and you can just show it as a logo image.

The second thing is that Google's hybrid OpenID + OAuth isn't as complicated as it sounds. It's basically a light extension to the OpenID "protocol" whereby you say, "while you're at it, also give me a OAuth token please so that I can connect back into Google's services later". What's important to understand though is that if you use this you need to know the "scope". scope is a URL to a service. Google Docs is a service for example and you need to search the web to figure out what the scope URL is for that service.

The third revelation was when I understood the difference between Simple Registration Extension (SREG) and Attribute Exchange (AX). Basically, AX is a newer more modern alternative and SREG was the first one. AX is better but some OpenID providers don't yet support it. Google for example, only supports AX. Key to be able to support not just Google's OpenID but any OpenID is that you can request both AX and SREG and whichever one works will be returned.

The fourth thing that helped a lot to understand was the Google's OpenID has a bug in its implementation of Attribute Exchange. Actually, perhaps it's a deliberate design choice they've made but in my opinion a bad one. Unless you say you require email, firstname, lastname, country etc. it won't return it. If you use the if_available directive you won't get it. Another bug/bad design choice is that Google seems to not forward the country attribute. It can happily do first- and last name but not country even if the documentation claims so.

The fifth thing is that python-openid is a lot easier to work with than you think. You don't need to do any crazy network checks or callbacks. For initiating the challenge all you're effectively doing is creating a long URL. If you don't like the API methods python openid offers, just add your own with:

redirect_url += '&openid.ax.mode=fetch_request' # etc.

After so many years since OpenID arrived, I'm only now excited about it. It's tonnes easier to implement than OAuth and now it's actually really pleasant to use as an end user.

According to my analytics the most popular Google search for getting to Crosstips.org is exactly crosstips.org which clearly proves that a lot of people type in the domain name in the search field these days. I do it too.

The second most popular search is "word whomp" which Crosstips has a dedicated page for. What's cute about that is that it was just a little side project I threw in yet it has grown to become one of the most popular features. You can never predict these kinds of things. I think the next thing I'm going to add is a Hangman solver which shouldn't be too hard.

So fellow Word Whomp cheaters, go for it!

I have a Zope web app that uses hand coded SQL (PostgreSQL) statements. Similar to the old PHP. My only excuse for not using an ORM was that this project started in 2005 and at the time SQLAlchemy seemed like a nerdy mess with more undocumented quirks than it was worth losing hair over.

Anyway, several statements use ILIKE to get around the problem of making things case insensitive. Something like the Django ORM uses UPPER to get around it. So I wonder how much the ILIKE statement slows down compared to UPPER and the indexed equal operator. Obviously, neither ILIKE or UPPER can use an index.

Long story short, here are the numbers for selecting on about 10,000 index records:

# ONE
EXPLAIN ANALYZE SELECT ... FROM ... WHERE name = 'Abbaye';
Average: 0.14 milliseconds

# TWO
EXPLAIN ANALYZE SELECT ... FROM ... WHERE  UPPER(name::text) = UPPER('Abbaye');
Average: 18.89 milliseconds

# THREE
EXPLAIN ANALYZE SELECT ... FROM ... WHERE  name ILIKE 'Abbaye';
Average: 24.63 milliseconds

First of all, the conclusion is to use UPPER instead of ILIKE if you don't need to do regular expressions. Secondly, if at all possible try to use the equal operator first and foremost and only reside on the case insensitive one if you really need to.

Lastly, in PostgreSQL is there a quick and easy to use drop in alternative to make an equality operatorish thing for varchars that are case insensitive?

UPDATE 2014

There is an answer to the question just above: citext

In lack of a fancier solution here's how I solved a problem of knowing who was logged in when an error occurred. I'm building a Intranet like system for a close group of people and if an error occurs I get an email that reminds me to add more tests. So I fix the bugs and upgrade the server. But I often want to know what poor sucker was logged in at the time the exception happened so that I can email them and say something like "Hi! I noticed your stumbled across a bug. My bad. Just wanted to let you know I've fixed that now"

So to do this I installed a silly little piece of middleware:

from django.conf import settings
class ExceptionExtraMiddleware(object):
   def process_exception(self, request, exception):
       if settings.DEBUG:
           return
       try:
           logged_in_info = ''
           if request.user and request.user.is_authenticated():
               logged_in_info = "%s" % request.user
               if request.user.email:
                   logged_in_info += ' %s' % request.user.email
               if request.user.first_name or request.user.last_name:
                   logged_in_info += ' (%s %s)' % \
                     (request.user.first_name, request.user.last_name)
           if logged_in_info:
               request.META['ERROR-X-LOGGED-IN'] = logged_in_info
       except:
           # don't make matters worse in these sensitive times
           logging.debug("Unable to debug who was logged in", exc_info=True)

This means that when I get an email with the traceback and snapshot of the request object I get this included:

...
'ERROR-X-LOGGED-IN': u'anita (Anita Test)',
...

UPDATE

The code above had a bug in it. Doing an if on request.user will return true even if there is no logged in user. The safest thing is to change it to:

if request.user and request.user.is_authenticated():

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

fcgi (threaded)              640 r/s
fcgi (prefork 4 processors)  240 r/s (*)
gunicorn (2 workers)         1100 r/s
gunicorn (5 workers)         1300 r/s
gunicorn (10 workers)        1200 r/s (?!?)
uwsgi (2 workers)            1800 r/s
uwsgi (5 workers)            2100 r/s
uwsgi (10 workers)           2300 r/s

(* this made my computer exceptionally sluggish as CPU when through the roof)

I don't know who these guys are but I found their pictures on Orange Snapshot (Orange customers Twitter thing for posting pictures) and it appears that they're cycling across the country and documenting this by taking pictures and sticking them up on Twitter. I actually often check out Snapshot just to get a feeling what's going on in the country at the moment. It was great when the snow hit us.

Why is this important? Well, in these modern times it means that it's very easy to make a traveling photo documentary without a film crew or some obscure travel blog which forces you to write 1,000 words for every city you visit. Good work on you lads! Keep it up!

I'm the proud creator of django-static which is a Django app that takes care of how you serve your static media the best way possible. Although some of these things are subjective generally this is the ideal checklist of servicing your static media:

1. Cache headers must be set to infinity
2. URLs must be unique so that browsers never have to depend on refreshing
3. The developer (who decided which media to include) should not have to worry himself with deployment
4. The developer/artist (who makes the media) should not have to worry himself with deployment
5. All Javascript and CSS must be whitespace optimized in a safe way and served with Gzip
6. All images referenced inside CSS should be taken care of too
7. It must be possible to combine multiple resources of Javascript or CSS into one
8. It must be possible to easily test production deployment in development environment without too much effort
9. A sysadmin shouldn't have to understand a developers Django application
10. A development environment must be unhindered by this optimization
11. Processing overhead of must be kept to a minimum
12. Must be possible to easily say which resources can be whitespace optimized and which can not

Came across these amazingly beautiful photos from the Katrina hurricane. I can't believe I missed these before! In a sense I wish I could have been there, but in a safe place.

We might be able to pool together and poison the earth with our pollutions but every now and then mother earth reminds us we're just ping pong balls in her game of tennis.