Peterbe.com

I was reading a thread about "Lisp development with macros faster than Python development?" on comp.lang.python when I stumbled across a little statement by Raymond Hettinger, a core Python developer:

"With Lisp or Forth, a master programmer has unlimited power and expressiveness. With Python, even a regular guy can reach for the stars."

Something I find myself doing very often is to download a .tar.gz or .tgz file that I want to unpack, but only in a subfolder. Some rather annoying gzips aren't collected in one folder so that when you unpack it lots of files are created in the current directory. Do you find yourself often doing this:

$ tar -ztvf Some-0.x.tar.gz
Some/file1.txt
Some/file2.txt
...
Some/file100.txt
$ tar -zxvf Some-0.x.tar.gz
Some/file1.txt
Some/file2.txt
...
Some/file100.txt

Or, in case they the gzip is badly organised:

$ tar -ztvf Foo-0.y.tar.gz
file1.txt
file2.txt
...
file100.txt
$ mkdir Foo; mv Foo-0.y.tar.gz Foo/; cd Foo/
$ tar -zxvf Foo-0.y.tar.gz
file1.txt
file2.txt
...
file100.txt
$ cd ..

Boy did that shut me up! The \b special character i python regular expressions is so useful. I've used it before but have forgotten about it. The following code:

def createStandaloneWordRegex(word):
   """ return a regular expression that can find 'peter'
   only if it's written alone (next to space, start of 
   string, end of string, comma, etc) but not if inside 
   another word like peterbe """
   return re.compile(r"""
     (
     ^ %s
     (?=\W | $)
     |
     (?<=\W)
     %s
     (?=\W | $)
     )
     """% (re.escape(word), re.escape(word)),
           re.I|re.L|re.M|re.X)

can with the \b gadget be simplified to this:

def createStandaloneWordRegex(word):
   """ return a regular expression that can find 'peter'
   only if it's written alone (next to space, start of 
   string, end of string, comma, etc) but not if inside 
   another word like peterbe """
   return re.compile(r'\b%s\b' % word, re.I)

Quite a lot simpler isn't it? The simplified passes all the few unit tests I had.

My friend Jacob from Galdrion taught me about "positive lookbehind assertion" and "lookahead assertion" when writing regular expressions in Python. It was new to me and I can't believe why I didn't read up on this more earlier because they're really useful. I've now got a usage for these which I use to find words that written on their own. For example, in the string "peterbe" the word "peter" doesn't exist really. You only want to find your words when they're written alone. You can't rely on it being spaces always on both sides either. You might find brackets, fullstops, end-of-string, you name it.

Enough chatting about it, I have now put together a little something that does this properly which I for no real reason call:

the Jacobian highlighter

Please give it a spin to see if it behaves like you expect it too. If you find any problems, let me know and I'll fix it and add them to the unit tests.

Next on the todo list for this is Unicode support of course.

I find myself often running a grep command with the -n which means that when it shows the filename it also shows the line number it matched it on. Then what I often do is that I open the found file with my editor and move to that line with the gotoline feature. Here's a cute little bin Python script that expands the result a little bit. Try:

$ grep -rin 'def set' *      
frintranet/sequence/tests/testSequence.py:26:    def setUp( self ):
slimmer/tests/testSlimmer.py:37:    def setUp(self):
slimmer/tests/testSlimmer.py~:37:    def setUp(self):
slimmer/tests/testSlimmer.py.bak:42:    def setUp(self):

Do you, like me, use the powerful print function to debug your Python code? Do you, like me, sometimes forget to remove lines with print and ship this out to less development-like environments? Do you, like I did, need a way to check for forgotten print statements?

I've written a dumb-simple script that scans through a directory for print statements. It uses a recursive grep command so this won't work for you hopeless Windows people :)

Download it and put it in your ~/bin (you might want to read it through) then all you have to do is to run it against a directory where there are lots of .py files:

$ find_print_statements.py ~/dev/mypythonmodule

If it doesn't work perhaps you can patch it and let me know. This might not be for the faint-hearted. There might be serious security concerns unless you have full control over your users and su account. I take no responsibility if this script rm -fr / your hd.

Did you see my mentioning about addhrefs last week? addhrefs is a python module that turns a text full of URLs and email addresses into links. For example:

>>> from addhrefs import addhrefs
>>> print addhrefs("""Visit www.peterbe.com 
and email mail@peterbe.com""")
Visit <a href="https://www.peterbe.com">www.peterbe.com</a>
and email <a href="mailto:mail@peterbe.com">mail@peterbe.com</a>

Then I saw this instruction on google.com about Preventing comment spam They suggest you add a rel="nofollow" to links in your comments on your blog and with a bit of luck this will reduce the amount of comment spam you get on your blogs. So, how to do that?

Python gurus are going to laugh at me for this one but I learnt it today; how to check if an object is callable. Before this I thought I was clever with this:

>>> def foo(): return "s"
>>> hasattr(foo, '__call__')
True
>>> hasattr(foo(), '__call__')
False

But there was an easier way. I discovered this by accident and looked it up in the documentation after. Use the builtin function called 'callable()':

>>> def foo(): return "s"
>>> callable(foo)
True
>>> callable(foo())
False

D'oh!

I've now made some improvements to my little addhrefs script. If you don't know what this is, read Add links to a text (take II) and Add links to a text with URLs which explains what this does.

The latest changes are that you can now pass your own function for turning an email or a URL into a link. For example:

>>> from addhrefs import addhrefs
>>> def emailify(e):return '<a href="/sendemail?to=%s">%s</a>'%(e,e)
>>> print addhrefs("Hello foo@bar.com", emaillinkfunction=emailify)
Hello <a href="sendemail?to=foo@bar.com">foo@bar.com</a>

You can also do the same with URLs by filling the urllinkfunction parameter.

Download: addhrefs-0.6.tgz

UPDATE Bug fixed version: addhrefs-0.7.tgz

If you do a speed report on Plone.org you notice that the three biggest files it services are plone.css, plone_javascripts.js and index.html. If you run these through my little slimmer whitespace optimizing program in Python you get the following results:

$ python slimmer.py --test http://plone.org/plone.css
Took 0.016 seconds
Bytes before: 29355
Bytes after:  20265
Bytes saved:  9090  (69.0% of original size)

$ python slimmer.py --test http://plone.org/plone_javascripts.js
Took 0.057 seconds
Bytes before: 27251
Bytes after:  19974
Bytes saved:  7277  (73.0% of original size)

$ python slimmer.py --test http://plone.org/ 
Took 0.029 seconds
Bytes before: 37186
Bytes after:  26466
Bytes saved:  10720 (10K)  (71.0% of original size)