Handwriting on the Sky

New Blog

Tue, 08 Apr 2008 05:29:59 GMT

Those of you following me on blendix already know this, but I have a new blog.

While I may post something here occasionally from now on, it will be a much more personal flavor; my writing about work, about programming, and about technology will move there.

Masters of the planets (both python and twisted) please update your feed sources; you can feel free to drop this livejournal in favor of the new URL.

Divmod: Reloaded

Fri, 28 Mar 2008 00:05:11 GMT

Hot on the heels of the Twisted release, Divmod has a new, and hopefully much more comprehensible, sight design and layout.

Check it out over at divmod.org.

I've long been ashamed of the default-Trac look and the opaque information layout on Divmod's site, and I'm really happy to have the way we greet the world be spruced up.

This is mostly the work of the unstoppable Duncan McGreggor; this is just his latest work in improving Divmod's communication with our community and our customers — and it won't be his last.

(As with any new site design, the topic isn't entirely a joke: your browser's probably cached some stuff it wasn't supposed to, so if you've been visiting our site a lot, re-load for the full effect...)

Upgrade now!

Thu, 27 Mar 2008 19:35:13 GMT

In case you haven't heard through some other channel already, Twisted 8 is out.

In addition to numerous fixes and features, this release also includes a new release system for Twisted itself; this (hopefully) means that we won't have another year-long release drought. We're planning to do another release in less than 3 months.

This means that new Twisted features will be available faster, but it also means that if you're writing some software that uses Twisted, upgrade now! We try very hard to make sure that each new release is mostly compatible with the one that comes before it, so that your upgrade should be painless. Especially if you have good unit tests.

However, this compatibility doesn't extend infinitely. There are at least a few twisted developers who would really like to drop some of our years of accumulated cruft and break compatibility with older versions.

If you upgrade now, your migration process will be gradually fixing a few deprecation warnings. If you wait for 3 or 4 more minor releases, upgrading all at once will mean that anything which has changed will start off broken, and your tests might not even run until you've fixed a bunch of things.

Of course, by "will", I mean "should" - we're not perfect, but we'll fix upgrade issues in micro releases if you find them and report them.

Open Source 3D Massively Multiplayer Game Infrastructure using Twisted

Thu, 20 Mar 2008 02:34:07 GMT

sirgolan wasn't at PyCon, but he totally should have been.

I (and others) have been working on him for years to release this stuff as open source, and he's finally done it!

Go check it out the Multiverse 3D trac site, and get the code!

Do it!

Do it now!

(and then start submitting patches to get its persistence layer ported back over to Axiom because I told Mike not to block the release on that but it should totally be using Axiom and he only switched away from it because he didn't understand quite how it worked...)

Against the Alexandria Library Migration Strategy

Wed, 19 Mar 2008 07:22:37 GMT

Some library maintainers, when faced with the impending incompatible changes in Py3K, decide that it's time to burn their library down and start over with a new, incompatible version. "Python is changing", they say, "so people are going to have to do a bunch of maintenance anyway. What a great opportunity to force them to do all that maintenance that they should be doing for our library too!"

I'm saying it's wrong. But don't take my word for it: Guido says it's wrong. Before it became cool to do it, Martijn Faassen was saying it was wrong.

Guido didn't just blog that it was wrong, though. He was so concerned that this message get out publicly that he repeated himself in a mailing list message to python-dev to make sure that people who don't read blogs would get the message.

This might strike some library maintainers as unfair. If Py3K is just breaking compatibility, why can't you?

First of all, even if Py3K were really "just breaking compatibility", there is still the issue of careful timing. You should read Guido's post and understand Ima Lumberjack's plight; he explains it exactly as I would. When your users are doing maintenance to upgrade something, they only want to upgrade one thing, so they know what's going wrong when they encounter problems. And if you're making incompatible changes, they will encounter problems, regardless of how cool and well-documented your new API is.

But, if you look closely, you will find that there's another reason that your library doesn't play by the same rules that Py3K does. It's because Py3K is actually doing a lot more than just breaking compatibility.

I've been a critic of this effort in the past (and I still occasionally grumble about a thing or two) but the bottom line is that the core Python team is not just willy-nilly breaking stuff. Let me enumerate the huge amount of work they're doing to make sure that people can have a reasonable migration experience to Python 3:

The Python core team have written and are maintaining a source-to-source translation tool to assist in the transition. Does your compatibility-breaking project have source-to-source translation, or in fact any tool support for migrating between different versions of the library?
The Python core team are developing a compatible backport of 99% of their features: Python 2.6 is effectively "python 3 lite"; you don't need to upgrade all the way to the incompatible version to get a lot of the new features. Does your compatibility-breaking project include a (at least mostly) compatible backport of all of your new features to an actively developed, "older" version?
The Python core team is providing long-term support for the previous version so that people can migrate at their own pace and not be left out in the cold. Is your compatibility-breaking project planning to provide a decade worth of bugfixes, security patches, and feature backports to your older versions?
The Python core team is providing comprehensive deprecation warnings explaining each new feature, and how you get there from the old feature. Is your project going to provide documentation like that?

It's also worth noting that Python's dependencies are not going through any kind of compatibility earthquake while they're doing this, so they are not in the same position that you (as a Python application author) are.

So, if you answered "yes" to all four of those questions — you still don't have the same excuse that Python does, because their dependencies are not changing incompatibly. So don't do it. But if you were thinking about breaking compatibility at the same time as a dependency, then you probably didn't.

Of course, you can drop deprecated stuff and break compatibility if your user community will tolerate that. Just don't do it in the same version where you decide to support Py3K. Users should have the ability to get a compatible version that will work in 2.x and 3.x so that when they translate their own source code, they don't have to learn new methods right at that moment.

Back from PyCon

Wed, 19 Mar 2008 01:47:06 GMT

Summary: PyCon 2008 was a great time. I didn't go to a single talk, except a few of the keynotes; I spent pretty much my entire time cross-pollinating with other projects and plugging the until-recently-secret Twisted Software Foundation - our nickname for the Twisted project's membership of the Software Freedom Conservancy (TSF/SFC).

I briefly addressed the audience (of over one thousand python users) to kick off the TSF announcement, mostly to introduce Duncan. However, someone managed to snap a picture of me that I think captured the feeling of awe that we've come so far in such a relatively short time.

I spoke to about 400 people at the conference, and I have a lot to follow up on. I also have a day job, 阿as those of you that I spoke to about Mantissa rather than Twisted know ;-).

If I talked to you about something at the conference, please don't hesitate to send me email reminding me about it. Ideally, send me email reminding me in one to three weeks. I have almost 1000 unread emails right now and while I try to be rigorous about using some unique features of Divmod's mail system to make sure I reply to each one, there is a certain volume of communication which no tool can help me cope with.

My first priority is blogging about interesting aspects of the conference in the next few days, before I've forgotten.

That Ain't Workin'

Thu, 06 Mar 2008 11:19:27 GMT

I'm a big fan of Nine Inch Nails. Not quite to the degree of buying every Halo, but I have a number of his albums. So of course I've been intrigued by his latest offering. Today, while looking at Nine Inch Nails "Ghosts I-IV' website, I noticed an interesting bit of information:

We have SOLD OUT of the 2500 Limited Edition Packages.

My great-uncle is fond of a saying. "It is better to be rich and healthy than to be poor and sick." Seeing this, I was reminded of it. It's not quite as catchy, but it's better to be customer-friendly and a huge success than reviled as corrupt and a failure.

The music and movie industries have been telling us for the last few years that digital restrictions are required in order to save their businesses from destruction. Well, Mr. Reznor has proven them wrong quite dramatically. Before I continue, let me address an obvious objection up front - I realize he's a superstar. However, the spokespeople for the RIAA that support their claims are also superstars: Lars Ulrich is hardly a starving artist laboring in obscurity. I'm not saying everyone can do what he did, only that the people who are already rich in the music industry can continue to be rich without the bullshit that they claim is critical.

The "Limited Edition", for those of you not up on the latest NIN happenings, is a three hundred dollar version of the album, containing a bunch of extras and a signature from Mr. Reznor himself. The full album, in lossless, non-restricted format, costs five dollars. There were 2500 copies of the limited edition.

Let me emphasize for those of you who might not be quite as up on the terminology that "lossless" formats (which NIN is selling for $5 here) are the highest quality format that it is possible to distribute over the internet. Other music producers, out of fear for eating their CD revenues, have mostly refused to provide digital copies of their music of this quality.

Also, to compare pricing: Apple typically charges 99¢ for a DRM-free song: it's not lossless, but it's 256kbps, which is fairly high quality. (I would not believe it if someone told me they could hear the difference, but there is a marginal difference in the perception of value here.) There are 36 songs on "Ghosts". $5 is roughly 14% of $35.64.

So now that I've established that NIN is selling higher quality goods, in a customer-friendly way, for a fraction of the price of the competition, let's do some math:

march 6, today,
minus the march 4th (the date that the "ghosts" website became fully operational (according to wikipedia),
is two days, times
2500 copies
times 300 dollars
equals SEVEN HUNDRED AND FIFTY THOUSAND DOLLARS IN TWO DAYS.

Trent has now proven that if you are a superstar, you can make three quarters of a million dollars in two days, on a ridiculously expensive premium edition alone. This is to say nothing of the people who bought, and continue to buy, the $75 version, the $10 version, or the $5 version. This says nothing of the people who are buying it through Amazon.

Coincidentally, it also proves that you don't need any RIAA thugs to help you do this, or "market" your work, assuming people already know who you are. You just need a web server, and a swimming pool big enough to put a million dollars.

Highlighting buried treasure in Twisted

Thu, 07 Feb 2008 10:17:55 GMT

I've previously blogged about twisted.python.modules, but it assumes you know about another API inside Twisted, twisted.python.filepath. Unfortunately this module is rather under-documented and under-publicized, despite being extremely useful. Unlike a lot of Twisted, much of the code in twisted.python can be extracted and used by itself, regardless of whether the program in question is networked or even event-driven. This is especially true of FilePath, which is completely blocking, although sometimes I wish there were at least a version of it that wasn't.

A common sort of script that deals with a filesystem is to open each file in a directory hierarchy with a given path and do something to its contents. For example, let's write a program that prints out a list of all Python modules (with a .py extension) in a tree which contain shebang lines.

Here's the script using good old os.path:

import sys
import os

def os_shebangs(pathname):
    for dirpath, dirnames, filenames in os.walk(pathname):
        for filename in filenames:
            fullpath = os.path.join(dirpath, filename)
            if (fullpath.endswith(".py") and
                file(fullpath, "rb").readline().startswith("#!")):
                yield fullpath

def os_show_shebangs(pathname):
    for path in os_shebangs(pathname):
        sys.stdout.write("%s: %s\n" % (
                path,
                file(path, "rb").readline()[2:].strip()))

if __name__ == '__main__':
    os_show_shebangs(sys.argv[1])

Pretty normal looking python code; not too much wrong with it. At 20 lines and 596 characters long, it's not too complex.

Now let's have a look at a similarly idiomatic version using FilePath:

import sys
from twisted.python.filepath import FilePath

def shebangs(path):
    for p in path.walk():
        if (p.basename().endswith(".py") and
            p.open().readline().startswith("#!")):
            yield p

def showShebangs(pathobj):
    for path in shebangs(pathobj):
        sys.stdout.write("%s: %s\n" % (
                path.path,
                path.open().readline()[2:].strip()))

if __name__ == '__main__':
    showShebangs(FilePath(sys.argv[1]))

At 18 lines and 471 characters, it's almost exactly 20% smaller than the version that uses os.path. However, a small space savings is hardly the most interesting property of this code. The advantages over the version that uses os.path:

It's easier to test. You can use a fake FilePath object rather than needing to replace the whole "os" module and the "file" builtin.
It's easier to read. You need fewer names; rather than os, os.path, and builtins, the code talks mainly to one object.
It's easier to write. How many of you honestly remembered that "dirpath, dirnames, filenames" is the order of the tuples yielded from os.walk?

It's easier to secure. If you wanted to allow untrusted users to supply input to the os.path version, you need to be very, very careful. What about "/"? What about ".."? With FilePath, you simply supply the input to the 'child' method, and...

>>> from twisted.python.filepath import FilePath
>>> fp = FilePath(".")
>>> x = fp.child("okay")
>>> y = fp.child("..")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "twisted/python/filepath.py", line 308, in child
    raise InsecurePath("%r is not a child of %s" % (newpath, self.path))
twisted.python.filepath.InsecurePath: '/home' is not a child of /home/glyph
>>> z = fp.child("hello/world")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "twisted/python/filepath.py", line 305, in child
    raise InsecurePath("%r contains one or more directory separators" % (path,))
twisted.python.filepath.InsecurePath: 'hello/world' contains one or more directory separators

It's easier to extend. As of revision 22464 of Twisted (i.e. the next release) you can replace twisted.python.filepath.FilePath with twisted.python.zippath.ZipArchive, and this exact same code can operate on zip files.

Not only does FilePath provide these benefits, it has very few dependencies. Even if you don't like Twisted much, you can use twisted.python.filepath by copying only 3 modules into your project (twisted.python.filepath, twisted.python.win32, and twisted.python.runtime) and twiddling the appropriate imports to be relative. Since FilePath is only one import for your code, and mostly consists of method calls, it will easily work with Twisted's version or your own. So, share and enjoy!

Do Not Want

Thu, 31 Jan 2008 14:31:34 GMT

I do not want a book called "The Ghost Brigades", but someone thought I did, for a minute. Those of you following this via my Blendix activity page will know what I mean.

One of the interesting things about working on Blendix is that we get to see all the ways in which other services export bad data. Amazon is particularly weird about it, though. As we were testing our code we saw a variety of bits of bad data intermittently published, some of which were fusions of the names of random products, some of which were actual products that had nothing to do with the user in question. I'm a little sad that my first bogus entry was a real product, though. Some of the incorrect entries we saw during testing were pretty hilarious.

Has anyone else out there used to using Amazon's various APIs to pull wishlist data and seen similar results? Is there any way to work around it, or to recognize the bogus data? Advice is welcome.

It Will Blend

Fri, 18 Jan 2008 23:25:43 GMT

So, as you may already have heard, we did a thing. While I've been able to tell a few people, I've been waiting to talk about it publicly for a while.

We've gradually begun to launch Blendix. Blendix aims to be the one site that you need to visit every day to tell you what's happening in your world. Right now, it's an aggregator for various kinds of data around the web. You can pull in the usual suspects (RSS, ATOM), but it's a bit more than just an RSS aggregator: today, it understands a few more specific things (last.fm tracks, amazon wishlists, yahoo weather, flickr photos), and lots more are planned. Although I'm practically bursting with awesome ideas for its future development, I will try to refrain from commenting too much on those plans. As I've said before, software is like frisbee — predictions more specific than "hey, watch this!" can be dangerous :-). However, since I know it's the first thing you'll all suggest, I can say that yes, there will be richer integration with social networks.

In brief, it works like this. You log in, and you create some people. Some feeds may be automatically discovered based on their email addresses, and you can add your own. Maybe you subscribe to some people: if you're looking for one to subscribe to, may I suggest "Glyph Lefkowitz". I hear he's pretty interesting. Finally, you visit your "dashboard" page, which — thanks to the magic of Athena — will update whenever blendix detects that one of the people you maintain or are subscribed to publishes some new data. You can expect to see more of that magic as it develops.

A word of warning, though: it doesn't work with Safari (or IE, but I don't imagine that a lot of you are using IE). We're working on it, but for the time being, Firefox is strongly recommended. (Firefox for the mac works fine.) Most of the work involved in supporting Safari is in Nevow, which is all open source, so if you are familiar with these sorts of problems, please submit patches!

This is our first live, fully public deployment of a Mantissa server, and I'm really glad to have it out there. We are, of course, working through the usual kinks of getting our first batch of users ("beta" isn't a web 2.0 buzzword for nothing), but I'm fairly pleased with it so far.

Blendix itself isn't open source — yet. We've mainly been keeping the product as a whole behind closed doors as a matter of expediency. We didn't want to support an API that was heavily in development. However, one of my goals as we get closer to a bigger launch is to get enough of the code out there for you folks in the community to write extensions and improvements for it. There's already enough for some things (like supporting Safari!), even at the application level. For example, a big chunk of Blendix is the "Person" object, which is available in the public Mantissa code, along with UI for editing, browsing, and viewing.

I'm really glad to have something "out there" to share with you all, and I'd like to encourage you to share back. Please check out Blendix, and make liberal use of the information you find under the "Contact" link at the bottom of every page. Let me know what you think, especially if you're a programmer and you've got some ideas for hacking on the code. This will be especially useful as get into the initial phase of pushing the core out to the community. Also, we really want to make sure the experience is as bug-free as possible, so let us know about any problems you have.

Export for Python

Wed, 02 Jan 2008 06:43:31 GMT

I've started playing around with minor projects in my personal launchpad space, partially to try out bzr.

Most recently I wrote a hack, temporarily named "pyexport", which allows you to control the names which your library module namespaces export to application code.

So far, I've implemented a few features.

export.alias(), which registers an alias for a method in another module that will not be imported until that module is imported,
export.explicitly(), a convenience function which makes cooperating with __all__ easy
export.internal(), which marks a module as "internal", and warns any application code (code outside the package which defines the module) which tries to import it
export.restrict(), a method which prevents "leakage" of extraneous imported or private names - for example, if you have a module 'foo' which imports 'sys', you can normally do 'from foo import sys' in Python and get a result.
export.singleton(), which replaces the calling module with a proxy that shares a namespace between the given singleton and the module itself.

It'll be some work to turn this rough prototype into something really usable for a large system like Twisted; at the very least it will need to be rebuilt test-first and integrated with the pydoctor and pydoc documentation tools. Let me know what you think, and if I should pursue it!

Twenty Post-Dollars

Wed, 02 Jan 2008 06:32:11 GMT

I finally hit the limit of my flickr account; I've posted 200 pictures, and so, starting from the beginning of my photo stream, some of them are going to start disappearing.

I have about 100 more pictures to post, but I've been intentionally keeping my account free, because I'm not sure if it's actually worth my time (or money) to use the features of a "pro" account if nobody is really listening.

So I'm wondering, are there 20 people out there — one for each of the dollars it would take to upgrade my account — who actually care about what I'm posting on flickr? No need to get effusive, just ping me in the comments (or email) to let me know I should upgrade and keep posting pictures.

It Was A Bullet

Mon, 31 Dec 2007 21:11:14 GMT

Tycho says:

More than anything else, I think it was installing Vista that made me hate PC gaming. The constant, system-level interruptions, the impaired compatibility, and most of all the savage kick to my framerate's exposed groin made me wonder what precisely in the fucking fuck I was doing screwing around with this onyx monolith. I knew I was just going to have to upgrade eventually (no), and I wanted to see if there was anything to this DirectX 10 thing (no), and I wanted to see what the Windows version of Live was like (a warcrime) so I bit the bullet. I shouldn't have. It was a bullet! That should have been my first clue.

Is this Microsoft's secret strategy with Vista? To make everyone by an Xbox because they're sucking the life out of the PC ecosystem? It hadn't yet occurred to me that Vista was supposed to be bad.

Looks like I'll be going to PyCon after all

Tue, 18 Dec 2007 04:23:42 GMT

Ying and I had been planning to have our wedding on the 11th anniversary of our first date: on march 15th, 2008. I've told a few people about our plans, although invitations haven't gone out yet.

However, we're going to have to delay our plans. In addition to the usual logistical issues and scheduling conflicts, one thing we didn't count on was that planners tend to be exceptionally busy during the holiday season and couldn't really return our calls.

I'll let you all know as soon as I can when the new date is, but we don't want to set one until we're sure that we can actually make sure the plans will happen on time. I appreciate all the offers of help — and believe me, we will call all of you with favors when the time comes — but we can't really wait any faster, no matter how much help we have ;).

The Fear That Haunts My Dreams

Mon, 26 Nov 2007 08:25:00 GMT

The good news is that in 2015 we will have a good distributed operating system and programming language; the bad news is that they will be Firefox and JavaScript.

The xUnit Paradox

Thu, 13 Sep 2007 13:04:28 GMT

Billions of years ago, at the dawn of the digital age, Kent Beck invented the idea of testing computer programs and wrote a paper about it called "Simple Smalltalk Testing: With Patterns". In it, he expressed an API which is now colloquially known as "xUnit", because there are implementations for various languages called xunit, where x is a reference to the language of implementation: for example, Java's "JUnit", Python's "PyUnit", and the C++ "CppUnit".

Many test-driven developers who use an xUnit implementation eventually become test-framework developers. After some time using an xUnit implementation, they realize that it lacks certain features which would make it more useful, and then writing their own testing tool. Such attempts, in Python, include:

Twisted's own trial tool, for whose genesis I am to blame.¹
py.test
TestOOB
nose
doctest

Unfortunately, many of these test frameworks are written only looking at the problems with xUnit, and not realizing its benefits.

There are two things that every potential test framework developer needs to know about xUnit.

The xUnit API is great.

You need a structured, well-specified way to manipulate tests. Maybe you don't realize it yet, but when you have a test suite with a hundred thousand tests, you will want to selectively run parts of it. You will want to be able to arbitrarily group tests differently, change their grouping, ordering, and runtime environment. Most importantly you'll want to do all this in a program, not necessarily with a GUI or command-line tool. To write that program, you need a coherent, well-designed, stable API for interacting with tests as first-class objects.

The biggest thing that xUnit does right is that it exists. It is a structured API for interacting with tests as first class objects. Many attempts to implement specific features end up architecturally breaking or ignoring this API, or adding extra, implicit stuff, which will "just work" if you use the particular TestCase class that came with your tool of choice, but break if you try to customize it too heavily or start overriding internal methods.

xUnit also factors some important responsibilities away from each other. A test case is different from a test result; a test suite contains multiple tests. The magic that is often associated with the 'testXXX' naming convention aside, a single test case object represents a single test, and multiple runs of the same tests must create multiple test objects. These might seem obvious, but they are all important insights which must be preserved: and problems occur when they don't seem quite so obvious. As James Newkirk said on his blog a few years ago: "I think one of the biggest screw-ups that was made when we wrote NUnit V2.0 was to not create a new instance of the test fixture class for each contained test method." Trial had this screw-up as well, and while it has been fixed, the '-u' option will still re-use a TestCase object to run its own method again for the second invocation.

Sadly, xUnit is not the solution to all of your problems. There is something else you need to know about it.

The xUnit API is terrible.

The xUnit API is missing a lot of features that it really needs to be a generally useful test-manipulation layer. It is because of these deficiencies that it is constantly being re-invented or worked around. If it did some more of these things right, then the ad-hoc extensions which don't conceptually work with it properly wouldn't keep springing up.

The main problem with xUnit that is not simply a missing feature is that it uses the "composite" pattern, rather than the "visitor" pattern, to implement test suites. If you want to discover what test cases that a test suite contains, you are supposed to call 'run' and it will run them for you. It is impossible to programmatically decompose a suite without stepping outside of the official xUnit API.

For example, let's say you wanted to have a "runner" object, which would selectively identify tests to be run in different subprocesses. Without getting involved in the particular implementation details of an xUnit implementation, there's no way to figure out how many tests are going to be run when you have discovered and invoked a TestCase; you just call 'run' and hope for the best. Of course, just about every actual xUnit implementation cheats a little bit in order to do things like generate progress bars to figure out how far done with the test run you are; but it's always possible - and intentionally supported, even - to generate tests on the fly within 'run' and run them.

xUnit has no intermediary representation of a test result. A vanilla xUnit API assumes that a result is the same thing as a reporter: in other words, when you call a method on the result, it reports it to the user immediately. This absence of a defined way to manipulate test results for later display means that you have to take over the running of the tests if you want to do something interesting with the report; it's not possible in a strict xUnit implementation to cooperatively define two separate tools which analyze or report data about the same test run.

The use of the composite pattern is linked to the lack of an intermediary "test discovery" step. In Beck's original framework, you have to manually construct your suites out of individual test case objects and call a top-level function that then calls them all, although most xUnit implementations accepted as "standard" these days will provide some level of convenience functionality there. For example, all the implementations I'm aware of will introspect for methods that begin with "test" and automatically create suites that contain one test case instance per method.

xUnit has no notion of cooperating hooks. If you want to provide some common steps for setUp and tearDown in both library A and library B, you have no recourse but to build complicated diamond inheritance structures around "a.testsupport.TestCase" and "b.testsupport.TestCase", or give up and manually call functions from setUp and tearDown. This is where Trial has gotten into the most trouble, because it sets up and tears down Twisted-specific state as part of the tool rather than allowing test cases to independently set it up.

The Alternative Is Worse

Those who have not learned from the benefits that xUnit provides, however, will be doomed to repeat its mistakes, and often make even more. The alternative - let's call it "AdHocUnit" - is very popular. It's to start glomming features into random parts of the test manipulation API to support specific use-cases without attention to the overall design of the system.

Twisted's Trial, the Zope Test Runner, Nose, and I'm sure quite a few other testing tools in Python all do this, and the result is a situation where the concepts are all basically compatible, but you can't write a test that uses features from two of these systems at once - and if you're a person (as there are many such people) who write code that relies heavily on both Twisted and Zope, this can be painful.

You can definitely tell when you've got an AdHocUnit implementation when your APIs are internally throwing around strings that represent file names, module names, and method names, without reference to test objects; when you've got global, non-optional start and stop steps (not in setUp or tearDown methods) which twiddle various pieces of framework state, or when you've started coupling the semantics of 'run' to some ad-hoc internal lists of tests or suites. You can tell you have an AdHocUnit implementation when your test discovery is convenient, but hard-coded to a particular idiom of paths and filenames. Most of all, you can tell you've got an AdHocUnit implementation when you've got some things called "TestCase" objects which claim to be xUnit objects that subclass from a TestCase class but mysteriously cannot be run by another xUnit framework in the same language.

What To Do Now

The object-oriented programming community needs a better API which is as high-level and generic as xUnit. Anyone looking to "fix" xUnit should be careful to create a well-defined structure for tests and document the API and the reasons for every decision in it. It's interesting to note that the (very successful) Beck Testing Framework was originally presented as a paper, not as an open source project.

It might seem like testing APIs don't require this kind of rigor. They seem deceptively simple to design: at first, all you need to do is run this bit of code, then that bit of code, and make sure they both work. For a while all you're doing is piling on more and more tests; making sure they all work.

Then, one day, you want to start doing things with all these tests. You want to know how long they take to run, how much disk space they use, how many of them call a certain API. You want to run them in different orders to make sure that they are in fact isolated, and don't interact with each other. You'll find out, as I did, that huge numbers of your tests are badly written because your first ad-hoc attempt at the framework was wrong.

Unlike most frameworks, you can't gradually evolve them and depend on your tests to verify their behavior, because by changing the testing framework, you might have changed the meaning of the tests themselves. Having a well-tested test framework can help, of course, but while you can test the test framework, you won't be testing your tests.

Of course, everything is possible in this world of infinite possibilities. Test frameworks can, and do, evolve; but the process is slower, and more painful than other kinds of evolution. So, when you're looking to write your own conveniences for testing, don't throw the baby out with the bathwater: keep what your xUnit implementation does well, retain compatibility with it, and build upon it.

Acknowledgments

I'd like to thank Jonathan Lange, who encouraged me to consider the benefits of xUnit in the first place, and the Massachusetts Bay Transportation Authority, without whose constant delays and breakdowns I wouldn't have had time to have written this at all.

1: I didn't really start 'trial', but a few nasty hacks in the redistribution of pyunit. In my defense, it predated 'unittest' as a standard-library module. Jonathan Lange, its current maintainer, was the one who made it an independent tool. Thanks to him, it is now actually compatible to a large extent with the standard 'unittest' module.

Thank You, Microsoft!

Sun, 02 Sep 2007 20:33:10 GMT

It looks like the ever-popular Vista may be the impetus which drives game developers towards open platforms. Jonathan Blow, the author of the long-delayed, very original, and kind of quirky indie game "Braid" has decided that, thanks to Vista being so annoying, he is going to develop and release Braid on Ubuntu.

Until now, only relatively unknown developers such as Id Software have bothered to release or develop games on the Linux platform, but now that the more popular indie developers are starting to look at it, the commercial mainstream might not be far behind. Hooray!

There is a flash of light! Your PYTHON has evolved into PSYDUCK!

Sat, 01 Sep 2007 09:19:52 GMT

I guess if you don't read any other python-related blogs (or news, or mailing lists, for that matter) you might not already know that the first alpha of Python 3.0 has been released.

A sample of the release notes:

There are a few memory leaks
SSL support is disabled
Platform support is reduced
There may be additional issues on 64-bit architectures
There are still some open issues on Windows
Some new features are very fresh, and probably contain bugs
IDLE still has some open issues

In other words, this is not a product for general consumption, and is not labeled as such. Do not use it expecting to be able to get real work done with it. This release is to help the Python development team find bugs and get feedback from the community.

I've already blogged about my inability to get excited about Python 3.0. Now that it's begun to arrive, I can more clearly see the scope of the work required to get in sync with it, and the new features that it is actually going to encompass. It's a staggering amount of work.

Jean-Paul Calderone has prepared a preliminary run of the 2to3 tool over the Twisted codebase. This includes a 820 kilobyte diff, which took 12 minutes to produce (on a fairly fast, modern piece of hardware). However, this is not even a complete run, because the 2to3 tool cannot even parse two of the files in the repository, despite the fact that they are all valid python. Many of the transformations (especially in the area of the unicode/str conversion) are almost certainly going to drastically change the semantics our test suite, if not break it - although enough of Twisted's dependencies are missing on 3.0 that I haven't even had an opportunity to try.

I still hold out hope that the 3.0 branch will gradually be abandoned, as these changes are rolled back into older versions (2.6+) of Python and gradually phased in, with their deprecated alternatives being gradually phased out. Right now, though, the plan is to continue parallel development in the 2.x series until 3.0 is ready to "take over", although I'm not sure how that determination will be made.

While I wish I could be more excited about something in the 3.0 roadmap, it worries me that some of the excitement I see from others is enthusiasm for the idea of using it as an excuse to break their own users' software too. If you've written a library for Python, please consider that its users are going to be having a hard enough time upgrading from python 2.x to 3.x; you should really try to provide the smoothest migration path from here to there, and keep your APIs as compatible as possible.

Although I'd like to say something nice and congratulatory, the thought of spending a year just pushing little piles of syntax into other little piles of syntax, even with the help of a tool like a hypothetically-much-more-advanced 2to3, is honestly just depressing. I'd have to get Twisted (and Axiom and Nevow and Mantissa and Quotient and a handful of proprietary projects that I work on) to work on Python 3.0 before I can use it. If I'm going to work on Twisted, I have a lot of other things I'd rather be doing. So my plan, for the moment, is to ignore Python 3.0 as long as I possibly can.

But hey, that's what the magic of open source is supposed to be about, right? Do you like Python 3.0? Do you want to see Twisted (and the various Divmod projects) eventually support it? A fairly substantial portion of the diff in question is a litany of non-controversial stylistic changes to update old, and sometimes creaky parts of the codebase. For example, there are a bunch of usages of the 'print' statement that need to be transformed; you might consider submitting a patch which simply removes all usages of "print", since we probably shouldn't be using that syntax anyway. That will reduce the size of the changes that we need to consider, generate, and apply in order to be 3.0 compliant. and probably improve the code's cleanliness quite a bit. Once those parts are applied, and thereby removed from the output of 2to3, we can have a better view of what work is actually necessary to support 2.x and 3.0 versions simultaneously from the same codebase.

Of course, comprehensive test coverage is something else frequently brought up when talking about the 2-to-3 migration. Any patches which increase or improve our test coverage will alway be greatly appreciated regardless of any migration issues.

The one prospect that appeals to me is that, if 3.0 successfully adheres to the vague promise of breaking backwards compatibility "just this once", Python may move towards being a real platform instead of simply a tool that you run on another platform. Of course, backwards incompatible changes can be a bit like potato chips - "you can't eat just one" - but I trust that the Python team can stick to it if they see the value in it and have made the incompatible changes they think are significant and necessary.

A few days ago I ran Neverwinter Nights on my Ubuntu (feisty) machine and was pleased to discover that despite the fact that there are new versions of SDL, X11, the Linux kernel, GNOME, ALSA, and a dozen other dependencies (all of which are dynamically bound) since five years ago when it was written, it still runs beautifully, with no configuration or re-installation on my part. Getting that kind of reliability from Python, and being able to provide it for Twisted, would be worth a fair amount of time spent overhauling syntax.

Pondering Python Path Programming Problems

Thu, 23 Aug 2007 04:33:24 GMT

Most Python programmers are at least vaguely aware of sys.path, PYTHONPATH, and the effect they have on importing modules. However, there's a lot of confusion about how to use them properly, and how powerful these concepts can be if you know how to apply them. Twisted - and in particular the plugin system - make very nuanced use of the python path, which can sometimes make things that use them a bit hard to explain, since there isn't a well-defined common terminology or good library support for working with paths, except to the extent that they are used by importers.

This article is really about two things: the general concept of paths, and the Twisted module "twisted.python.modules", which provides some specific implementations of my ideas about the python path.

First of all, why should you care about python paths? To put it simply, because very bizarre problems can result if you use them incorrectly. Also, you need to know about them in order to use Twisted's plugin system effectively, and of course you want to use Twisted, right? :)

What kind of problems? Even very popular, well-regarded Python packages by very experienced Python programmers sometimes mess this up pretty badly. Here's a simple example of what can go wrong with a package you probably know of, the Python Imaging Library:

>>> import Image
>>> import PIL.Image
>>> img = PIL.Image.Image()
>>> Image.__file__
'/usr/lib/python2.5/site-packages/PIL/Image.pyc'
>>> PIL.Image.__file__
'/usr/lib/python2.5/site-packages/PIL/Image.pyc'

Here we can see that you can import PIL's "Image" module as either "PIL.Image" or simply "Image". Both these modules are loaded from the same file. On the face of it, this is simply a convenience. But let's dig deeper:

>>> PIL.Image == Image
False

The modules aren't the same object! This has some nasty practical repercussions:

>>> isinstance(img, Image.Image)
False

For example, Image objects created from one of these PIL modules do not register as instances from the other, even though they're all the same code. Worse yet, this mistake can become "sticky" if you use them along with a module like pickle, which carries the module and class name into the data:

>>> from cPickle import dumps
>>> img2 = Image.Image()
>>> dumps(img)
"(iPIL.Image\nImage\n ...
>>> dumps(img2)
"(iImage\nImage\n ...

Many Python features and packages depend on matching types. Zope Interface, for example, will not let you use adapters for one Image type for the other, the objects will not compare equivalent even if they really are, and so on. And none of this is a bug in the code! Why does it happen?

PIL is a package; that is, a directory with Python source code and an "__init__.py" in it, named "PIL". However, it also installs a ".pth" file as part of its installation. ".pth" files are one way to add entries to your sys.path. This particular one adds the "PIL" directory to your path, which means it can be loaded from two entries: as a package, from your "site-packages" directory.

This isn't to pick on PIL or the Effbot; I've seen lots of projects which have a "lib" directory with an __init__.py and change its name at installation time, or inconsistently reference subpackages with relative and absolute imports, or do any number of things which are just as bad. I hope that I've convinced you not to do the same thing with your project, but I won't dwell on the problem here, since I have a solution handy.

Unless you already know what is going on (although I'm sure many of you reading this already do), this can be a bit confusing to figure out. You can use twisted.python.modules to ask this question rather directly. Here's how:

>>> from twisted.python.modules import getModule
>>> imageModule = getModule("Image")
>>> pilImageModule = getModule("PIL.Image")
>>> imageModule.pathEntry
PathEntry<FilePath('/usr/lib/python2.5/site-packages/PIL')>
>>> pilImageModule.pathEntry
PathEntry<FilePath('/usr/lib/python2.5/site-packages')>

Here we're asking twisted.python.modules to give us objects that represent metadata about two modules, without actually loading them. The attribute here is the 'pathEntry' attribute, which tells us what entry on sys.path the module would be loaded from, if it's imported.

>>> import sys
>>> pilImageModule.isLoaded()
False
>>> imageModule.isLoaded()
False
>>> 'PIL.Image' in sys.modules
False
>>> 'Image' in sys.modules
False

Look, no modules!

Of course, if we wanted to load those modules, it's easy enough:

>>> pilImageModule.load()
<module 'PIL.Image' from '/usr/lib/python2.5/site-packages/PIL/Image.pyc'>
>>> imageModule.load()
<module 'Image' from '/usr/lib/python2.5/site-packages/PIL/Image.pyc'>

You can also get lists of modules. For example, you can see that the list of modules in the "PIL" package is suspiciously similar to the list of top-level modules that comes from the path entry
where the "Image" module was loaded:

>>> pilModule = getModule("PIL")
>>> pprint(list(pilModule.iterModules())[:5])
[PythonModule<'PIL.ArgImagePlugin'>,
PythonModule<'PIL.BdfFontFile'>,
PythonModule<'PIL.BmpImagePlugin'>,
PythonModule<'PIL.BufrStubImagePlugin'>,
PythonModule<'PIL.ContainerIO'>]
>>> pprint(list(imageModule.pathEntry.iterModules())[:5])
[PythonModule<'ArgImagePlugin'>,
PythonModule<'BdfFontFile'>,
PythonModule<'BmpImagePlugin'>,
PythonModule<'BufrStubImagePlugin'>,
PythonModule<'ContainerIO'>]

As you might imagine, the ability to list modules and load the ones that seem interesting is a great way to load plugins - and that's exactly how Twisted's plugin system is implemented. While the plugin system itself is a topic for another post (or perhaps you could just read the documentation) the way it finds plugins is interesting.

For example, let's take a look at the list of Mantissa plugin modules I have installed:

>>> xmplugins = getModule('xmantissa.plugins')
>>> pprint(list(xmplugins.iterModules()))
[PythonModule<'xmantissa.plugins.adminoff'>,
PythonModule<'xmantissa.plugins.baseoff'>,
PythonModule<'xmantissa.plugins.free_signup'>,
PythonModule<'xmantissa.plugins.offerings'>]

This simple query is actually an incomplete list. It's just the modules that come with Mantissa itself. Python has a special little-known rule when loading modules from packages, and twisted.python.plugins honors it: if there is a special variable called "__path__" in a package, it is a list of path names to load modules from. However, twisted.python.plugins doesn't load modules unless you ask it to, so it can't determine the value of that attribute. As it so happens, twisted.plugins uses the __path__ attribute in order to allow you to keep your development installations separate, so twisted.python.plugins can't determine all the places you might need to look for plugins without some help. Let's just load that package so we can look at its __path__ attribute:

>>> xmplugins.load()
<module 'xmantissa.plugins' from '/home/glyph/Projects/Divmod/trunk/Mantissa/xmantissa/plugins/__init__.pyc'>

Now that we've loaded it, let's have a look at that list:

>>> pprint(list(xmplugins.iterModules()))
[PythonModule<'xmantissa.plugins.adminoff'>,
PythonModule<'xmantissa.plugins.baseoff'>,
PythonModule<'xmantissa.plugins.free_signup'>,
PythonModule<'xmantissa.plugins.offerings'>,
PythonModule<'xmantissa.plugins.mailoff'>,
PythonModule<'xmantissa.plugins.radoff'>,
PythonModule<'xmantissa.plugins.sineoff'>,
PythonModule<'xmantissa.plugins.hyperbolaoff'>,
PythonModule<'xmantissa.plugins.imaginaryoff'>,
PythonModule<'xmantissa.plugins.blendix_offering'>,
PythonModule<'xmantissa.plugins.billed_signup'>,
PythonModule<'xmantissa.plugins.billoff'>,
PythonModule<'xmantissa.plugins.derivoff'>]

That's my full list of Mantissa plugins, including my super secret Divmod proprietary plugins.

This list is generated because plugins packages use a feature (which was previously kind of a gross hack but will be an officially supported feature of the next version of Twisted) to set their path to every directory with the same name as the plugin package which is not also a package on your python path. In other words, if you have 2 sys.path entries, a/ and b/, and one package, x.plugins, in b/x/plugins/__init__.py with this trick in it, then if you have a file b/x/plugins/foo.py, it will be considered to contain the module "x.plugins.foo". This requires that you do not have a file b/x/__init__.py or b/x/plugins/__init__.py. If you do, this hack will treat the two paths the same way that Python does: duplicate packages in your path, so the package in a/ is loaded and the package in b/ is ignored.

The distinction between packages and path entries is why all the Twisted and Divmod projects conventionally have capitalized directory names but lowercase package names. "Twisted" is where your path entry should point; "twisted" is the python package that is loaded from that path entry. "Twisted" should never have an __init__.py in it. "twisted" always should. This goes the same for "Axiom" and "axiom", "Mantissa" and (the unfortunately named) "xmantissa". You will sometimes encounter other examples of this style of naming floating around the web.

When using Twisted and Divmod infrastructure, keeping this distinction is clear is critical, because otherwise it is difficult to develop plugins independently. You probably don't want to copy your development plugins into your Twisted installation - they're part of your source repository, after all, not ours. However, keeping the distinction clear in your mind will avoid lots of obscure problems with duplicate classes and naming, so it's generally a good idea even if you don't like our naming conventions.

Please let me know in the comments which parts of this post you found useful, if any. I know it's a bit rambling, and covers a number of different topics, some of which may be obvious and some of which might be inscrutable. I've experienced quite a bit of confusion when talking to other python programmers about this stuff, but I'm not sure if it was my awkward explanation of Twisted's plugin system or some inherent issue in Python's path management.

Not Just The Faithful

Wed, 22 Aug 2007 01:54:29 GMT

As I've said before, Microsoft Windows Vista is a terrible disaster which I hope I never have to deal with in any capacity, professional or otherwise. I suspect that it is inevitable, but I will resist it for as long as possible.

The FSF has a campaign, "BADVISTA", to educate end-users about the ways in which Vista is limiting your freedom more aggressively than any other commercial software product to date. Unfortunately this can sometimes sound a bit ... overdramatic, even if it is pretty much all true. For example, a prominently featured quotation:

Windows Vista includes an array of “features” that you don't want. These features will make your computer less reliable and less secure. They'll make your computer less stable and run slower. They will cause technical support problems. They may even require you to upgrade some of your peripheral hardware and existing software. And these features won't do anything useful. In fact, they're working against you.
- Bruce Schneier, "DRM in Windows Vista", from Schneier on Security

I recently had the experience of talking to a Regular User in a consumer electronics store about his vista "upgrade". His "computer guy" had told him that Vista was like XP, but better. Little did he know that the "better" would mean that the computer ran visibly slower, had reduced functionality, and required the purchase of newer, more expensive hardware.

Of course, I gave him my rant about the other reasons he shouldn't have upgraded, and the poor guy turned white as a sheet. I don't think he's going to be purchasing any more "upgrades" from his "computer guy".

But, what does the other side have to say about this fancy new operating system? Surely there are some worthwhile new conveniences that we are trading this freedom for? Let's see what one ex-Microsoft employee and prominent Windows developer has to say about it:

"I've been using Vista on my home laptop since it shipped, and can say with some conviction that nobody should be using it as their primary operating system -- it simply has no redeeming merits to overcome the compatibility headaches it causes. Whenever anyone asks, my advice is to stay with Windows XP (and to purchase new systems with XP preinstalled)."
- Joel Spolsky, "Even the Office 2007 box has a learning curve", from Joel On Software

... and there you have it. Friends don't let friends use Vista.

Pet Peeve

Sun, 15 Jul 2007 19:22:34 GMT

The word "depreciate" means "to lessen the price or value of". This is an accounting jargon term referring to the process by which assets lose value over time. It is pronounced 'Dee Pree Shee Ate".

The word "deprecate" means "to express disapproval of" or "to urge reasons against; protest against". This is a programming jargon term describing the process by which APIs become less favorable over time. It is pronounced "Deh Preh Kayt".

These words, while they have similar meanings, are not synonyms. Please do not confuse them, especially when using their jargon senses. It sounds like nails on a chalkboard to me, having worked on accounting software. I would like to be able to use phrases like "a deprecated depreciation function" without eliciting bewilderment.

Both Java and Python consistently use "@deprecated", and "DeprecationWarning". English usage of these terms may be shifting, but "DepreciationWarning" or "@depreciated" will still get you runtime or compiler errors, so please stick to "deprecate" consistently while talking about code.

Thank you.

Mindful Link Propagation

Sat, 14 Jul 2007 17:45:13 GMT

It occurs to me that there may still be a few Python people who read this blog but have not yet discovered JP Calderone's.

If you are such a person, he just did an excellent write-up of the practical implications of Python's rich comparison operators. Check it out.

Functional Functions and the Python Singleton Unpattern

Sun, 08 Jul 2007 02:47:17 GMT

Have you ever written a module that looked like this?

subscribers = [] def addSubscriber(subscriber): subscribers.append(subscriber) def publish(message): for subscriber in subscribers: subscriber.notify(message)

And then used it like this?

from publisher import publish class worker: def work(self): publish(self)

I've done this many times myself.

I used to think that this was the "right" way to implement Singletons in Python. Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.

Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have. As Wikipedia points out, "It is also considered an anti-pattern since it is often used as a euphemism for global variable."

The module above is brittle, and as a result, unpleasant to test and extend.

It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.

It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish". Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).

It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up. If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.

Unfortunately, this probably doesn't seem particularly bad, because bad examples abound. It's just the status quo. Twisted's twisted.python.log module is used everywhere like this. The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.

And there's a good reason that this keeps happening. Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places. It seems convenient to have it available, and it basically works.

Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.

First, make your "global" thing be a class.

class Publisher: def __init__(self): self.subscribers = [] def addSubscriber(self, subscriber): self.subscribers.append(subscriber) def publish(self, message): for subscriber in self.subscribers: subscriber.notify(message) thePublisher = Publisher()

Second, decide and document how "global" you mean. Is it global to your process? Global to a particular group of objects? Global to a certain kind of class? Document that, and make sure it is clear who should use the singleton you've created. At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing, Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.

Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly. For example:

from publisher import thePublisher class Worker: publisher = thePublisher def work(self): self.publisher.publish(self)

In this example, you now have a clean point of integration for testing and extending this code. You can make a single Worker instance, and change its "publisher" attribute before calling "work". Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker. If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.

Does this mean you should make everything into objects, and never use free functions? No. Free functions are fine, but functions in Python are for functional programming. The hint is right there in the name. If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.

Once you've started adding mutable state into the mix, you're into object territory. If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods.

Ultimate Quality 功夫 System

Fri, 15 Jun 2007 23:53:31 GMT

Often it is the means that justify the ends: Goals advance technique and technique survives even when goal structures crumble.
- Alan Perlis, Epigram 64, "Epigrams in Programming"

How does a computer programmer get better at computer programming?

In a very broad sense, there is some consensus on the answer to this question. It's like the answer any other skill. "Do a lot of it." Much like other skills, though, while necessary, this condition is not sufficient. If you want to get really good at punching, you can punch a brick with your thumb inside your first ten thousand times, and someone who has learned to do it correctly and had a fraction of the equivalent amount of practice will be a lot better at punching than you.

So, to expand on that consensus, to get better at programming, you have to learn to do it right, and then do a lot of it, hopefully with someone correcting your mistakes and helping you do it better each time. Like any other kind of serious training.

This concept is pretty old. Shortly after the dawn of time, the dragon god-emperor pioneered this concept when he invented a martial art that is now popularly known as "Kung Fu", or, literally, "achievement through great effort". The term does not necessarily refer specifically to martial arts, and can (in both Chinese and English) refer simply to the fact that someone has achieved something exceptional through a lot of effort and practice.

Specious metaphors between computer programming and martial arts (and various eastern philosophies) have existed for almost as long as computer programming itself. As a cocky and callow youth, I occasionally had these metaphors generously applied to me, but they rang hollow even in my hubristic state of mind. I knew that despite any talents I might have, at the very least, I hadn't spent a lifetime of effort achieving a near-supernatural mastery of computers. I had just messed around with C++ and Java a little bit. The appellation of "master" was inappropriate, but I didn't know how one really did go about getting to that point. I had already met a few folks in the programming profession who were old, but certainly not yet wise, and I wanted to learn how not to end up like that.

Whenever two programmers meet to criticize their programs, both are silent.
- Alan Perlis, Epigram 101, "Epigrams in Programming"

I first noticed that the parallel between UQDS and these various arts and philosophies when a friend observing Divmod from afar mentioned that our review process was intimidating. Intrigued, I conducted a straw poll among my programming friends who are not deeply immersed in the eternal struggle of the Twisted or Divmod codebases, and found that there was a fairly consistent impression: the exchange of comments between authors and reviewers was highly unusual, intimidating, and impressive.

It's unusual because most people cannot read code well enough to comment in as much depth. Code review is becoming more popular, but it's still rare. Even intense, newfangled "Agile" processes like Extreme Programming focus more on discussing code as it's being created than reading code which has already been written.

The exchange is intimidating because many of the comments are extremely fine-grained and not very forgiving. It isn't uncommon to see comments about a tiny private method missing documentation, or test coverage doesn't hit a single error condition. This causes the reader to muse on the ways in which their own code would fail if it were subjected to such intense scrutiny.

Finally, the exchange is impressive because the factors which make it intimidating make the programmers doing it seem extremely skilled. If you're looking at work which is being subjected to criticism only on details which seem insignificant, it implies that it is in every other way correct. To some extent this impression is illusory, because the reviewers are only human and may be missing more important things. For something as complex and intricate as software, though, it's impressive assuming that the reviewers tried at all.

I can't pretend that UQDS is an entirely unique process in this regard. The Python and Subversion projects practice regular code reviews in public, and those are just the ones I know off the top of my head. This isn't a phenomenon unique to open source either - I know that both Canonical and Google do fairly comprehensive code review on their closed-source products. I've heard milder expressions of awe - although perhaps they were only milder because they came from friends - about those projects and companies.

While it isn't unique, UQDS is particularly intense. Some other processes try to be "reasonable" - which is, on the face of it, a reasonable thing to do. Small, "obviously correct" fixes are sometimes allowed without tests. "Security" fixes are sometimes allowed in for expediency because of their urgency. Some committers are sometimes exempt. Some reviews are done after the fact. Documentation is often optional, since reviews are meant to enforce correctness, not every aspect of quality. For better or worse, UQDS sets out a much stricter guidelines: everything must be tested. Everything must be documented. Nobody is allowed to bypass the process.

It isn't all roses. Especially when a contributor needs to update an area of code not originally developed under this regime, it can be frustrating to need to document and test and correct code that isn't really related to their goal. Sometimes the overabundance of the opportunity for review creates interminable discussions about a design, when there is really a need to just go with a solution that works for now. There may be some room to adjust the constraints to place less of a burden on those who want to contribute small improvements, and that's difficult without compromising on overall quality.

Despite their drawbacks, these rules plus a public ticket tracker add up to something that the industry of martial arts has known about the power of for quite some time: the exhibition. If you're going to run a martial arts studio you need to attract students, and to do that you need to "wow" people every so often. Martial arts are about defense, mind-body integration, health and fitness, and self-discipline, but putting that on a poster isn't going to fill up the thursday evening class. Sometimes you need to get your school together and show off just how bad you would be able to kick some ass with your skills, you know, just in case anybody was wondering.

I am quite proud of the fact that I've received numerous unsolicited comments about the quality of the code itself within certain parts of Divmod and Twisted. Like any programmer I've received my share of complaints about my work, too (especially the stuff that precedes all this intense review), but as Steve Yegge famously observed, most programmers who are famous for their programs don't actually write programs; they write about programs, or they wrote some programs nobody has the code to.

I'm almost more pleased with the idea that I could help pave the way for programming to have some real rock-stars on the merit of their code than the idea of being a rock-star myself.

Dealing with failure is easy: Work hard to improve. Success is also easy to handle: You've solved the wrong problem. Work hard to improve.
- Alan Perlis, Epigram 101, "Epigrams in Programming"

To close, there are a couple of skills that I think UQDS promotes which there might be other ways to promote. If you want to develop a competing "school", these are things to think about:

Reading code. Open source is great because you can read the code, but UQDS gives you a reason to do the reading. Not just skim, not participate in the writing, but really read and analyze.
Communicating with other programmers. Another thing that UQDS has in common with a martial art is that it's adversarial. You throw the "punch" of a patch, the reviewer "blocks" with a branch, and so on. You don't need to be a social butterfly to get this right, but you do need to be very focused on the technical communication skills required to explain your ideas.
Testing. Test-driven development is an excellent trend. UQDS doesn't have much to add to that beyond enforcing it, and making the application of TDD the easiest way to get through reviews which deal with issues like test coverage. (If you test everything before you write it, of course your tests are going to cover everything before you put it into review.)

I'm not a martial artist. The closest I've come was years ago when I was a fencer, and although I hope to pick up a sword again some time soon, I am no expert. Still, I have a great deal of respect for Kung Fu, and I hear a fair amount about it from a fellow I know who did some training in it, and I think this is a bit more accurate of a comparison than the earlier attempts that I mentioned; so if you're a twelfth dan grand master in ultimate ninjitsu, please don't explode my head with your psychic powers if you think I'm wrong.

A disagreement in the comments would be sufficient.

50/50 again!

Tue, 12 Jun 2007 18:51:19 GMT

Or, 72/72 to be precise.

Twisted is once again in danger of losing the engineship of your Internet! If you haven't voted on this Jyte claim already, please do so now!