Peterbe.com

Because this took me a long time to figure out I thought I'd share it with people in case other people get stuck on the same problem.

The problem is that Mongoose doesn't support DBRefs. A DBRef is just a little sub structure with a two keys: $ref and $id where $id is an ObjectId instance. Here's what it might look like on the mongodb shell:

> db.questions.findOne();
{
       "_id" : ObjectId("4d64322a6da68156b8000001"),
       "author" : {
               "$ref" : "users",
               "$id" : ObjectId("4d584fb86da681668b000000")
       },
       "text" : "Foo?",
       ...
       "answer" : "Bar"
       "genre" : {
               "$ref" : "question_genres",
               "$id" : ObjectId("4d64322a6da68156b8000000")
       }
}

DBRefs are very convenient because various wrappers on drivers can do automatic cross-fetching based on this. For example, with MongoKit I can do this:

for question in db.Question.find():
   print question.author.first_name

If we didn't have DBRefs you'd have to do this:

for question in db.Question.find():
   author = db.Authors.findOne({'_id': question.author})
   print author.first_name

Today Felinx Lee and I launched TornadoGists.org which is a site for discussing gists related to Tornado (python web framework open sourced by Facebook).

Everyone in the Tornado community seems to solve similar problems in different ways. Oftentimes, these solutions are just a couple of lines or so and not something you can really turn into a full package with setup.py and everything.

Sharing a snippet of code is a great way to a) help other people and b) to get feedback on your solutions.

The goal is to make it a very open and active project with lots of contributors. I'll be accepting and reviewing all forks but hopefully control will be opened up to all Tornado developers. Also, since the code is quite generic to any open source project Felinx and I might one day port this to rubygists.org or lispgists.org or something like that. After all, Github does all the heavy lifting and we just wrap it up nicely.

This took me a long time to figure out so I thought I'd share.

Basically, I'm a newbie supervisor administrator and I was setting up a new config and I kept getting these errors:

# supervisord -n
2011-04-04 17:25:11,700 CRIT Set uid to user 1000
2011-04-04 17:25:11,700 WARN Included extra file "/etc/supervisor/conf.d/gkc.conf" during parsing
Error: Cannot open an HTTP server: socket.error reported errno.ENOENT (2)
For help, use /usr/local/bin/supervisord -h

The reason was that in my config I had the line:

[unix_http_server]
file=/var/lib/tornado/run/gkc.sock

but the directory /var/lib/tornado/run didn't exist. Creating that solved the problem.

Lesson learned from all this is that when specifying locations of .sock files always make sure the directories exist and that the current user can write to them.

I like the new Google Page Speed Online for it's simplicity. However, I threw it the URL of my Crosstips site http://crosstips.org and it only gave me a 80 out of 100 even though there were no high priority suggestions.

Seems hard to beat. Surely, to win over the remaining 20 points I don't have to tick all the medium and low priority suggestions.

This is helping me sooo much that it would a crime not to share it. It's actually nothing fancy, just a very convenient thing that I've learned to get used to. ff is an executable script I use to find files in a git repository. Goes like this:

$ ff list
templates/operations/network-packing-list.html
templates/sales/list_orders.html
$ ff venue
templates/venues/venues-by-special.html
templates/venues/venues.html
templatetags/venue_extras.py
templatetags/venues_by_network_extras.py
tests/test_venues.py

It makes it easy to super quickly search for added files without having to use the slow find command which would also otherwise find backup files and other junk that isn't checked in.

To install it, create a file called ~/bin/ff and make it executable:

$ chmod +x ~/bin/ff

Then type this code in:

#!/usr/bin/python
import sys, os
args = sys.argv[1:]
i = False
if '-i' in args:
   i = True
   args.remove('-i')
pattern = args[-1]
extra_args = ''
if len(args) > 1:
   extra_args = ' '.join(args[:-1])
param = i and "-i" or ""
cmd = "git ls-files | grep %s %s '%s'" % (param, extra_args, pattern)
os.system(cmd)

I've put all the static resources behind this site now on AWS CloudFront For example this: http://static.peterbe.com/misc_/Peterbecom/home/grey_face.1282513695.png

This site doesn't really have much traffic. About 50,000 pageviews per month. The bill for the last month: $0.55

I think I can afford that :)

This is a summary about the MongoUK conference held today here in London. It was great! Unlike other commercial conferences this one actually cost money but it was only about $50 and although there were free coffee cups, stickers and stuff this didn't feel at all like they were trying to sell themselves. The focus was on the technology. Great!

As often with these things, you realise that you have actually grasped a couple of things quite well now but it's also humbling in that there is so much you haven't grasped that other people are way ahead of you in.

What I learned

Yay! I've figured out how to properly unit tests my jQuery Mobile app that I'm working on. This app uses localStorage, localSession, lots of AJAX and works both offline and online. Through various hacks and tricks I've managed to mock things so that I can easily test the otherwise asynchronous AJAX calls and I can also test the little quirks of jQuery Mobile such as re-rendering <ul> tags after having changed the DOM tree.

I'm using the QUnit testing framework and I like it. The app isn't launched yet and the code is currently protected but once I get it nailed a bit more I'll blog about it more fully so other people can jump in unit testing their Javascript heavy jQuery Mobile sites too. For now I'm up to 75 tests and it's growing steadily.

Here's a little taster for how I mock the $.mobile, AJAX and the localStorage stuff:

module("Storage and AJAX", {
  setup: function() {
     localStorage.clear();
  },
  teardown: function() {
     localStorage.clear();
  }
});

var MockMobile = function() {
  this.current_page;
};
MockMobile.prototype.changePage = function(location) {
  this.current_page = location;
};

test("test Auth", function() {
  $.mobile = new MockMobile();
  var _last_alert;
  alert = function(msg) {
     _last_alert = msg;
  };
  $.ajax = function(options) {
     switch (options.url) {
      case '/smartphone/checkguid/':
        options.success({ok:true});
        break;
      case '/smartphone/auth/login/':
        if (options.data.password == 'secret')
          options.success({guid:'10001'});
        else
          options.success({error:"Wrong credentials"});
        break;
      default:
        console.log(options.url);
        throw new Error("Mock not prepared (" + options.url + ")");
     }
  };
  var result = Auth.is_logged_in(false);
  equals(result, false);

  Auth.ajax_login('peterbe@example.com', 'other junk');
  ok(_last_alert);
  ...

(Note: this is copied slightly out of context but hopefully it reveals some of the hacks and tricks I use)

Erann Gat reveals why he lost his mojo with Lisp

What caught my attention (for busy people who don't want to read the whole email):

I'm currently studying Lisp myself and it's hard. Really hard. I blame it on being spoiled with a programming language that I can work in without having to read the manual. With python's brilliant introspection I can use the interpreter to find out how a library works just by using help() and dir() without even having to read the source code. (not always true of course)

As we're entering the 21st century, the new contender "Usability" is becoming more and more important. Considering that I've now done Python for more than a decade I remind myself one of the reasons I liked it so much; yes, exactly that: Usability.

I was reading this article about linkfluence moving from CouchDB to Riak

Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.

It's the exact same perspectives I've always had on ZODB:

1) It's really convenient and powerful

2) It being a single HUGE file makes it hard to scale

3) Versioning can be nifty but it's often not needed and causes headache with the packing

4) It works great but when it cracks it cracks hard and cryptically