Log in

No account? Create an account
entries friends calendar profile Scribbles in the Dark Previous Previous Next Next
Seventh System Effect - Please Visit http://glyph.twistedmatrix.com/ - This Blog Is Closed.
Sorry about the ads, I can't turn them off.
Seventh System Effect
It's somewhat official now, so I guess I have to announce it: Divmod is doing a massive refactoring of our application, starting with the database. Work began a week ago in my Quotient sandbox, and has been continuing around the clock since then.

Progress is now visible in a more public location:
the unified Divmod SVN repository at http://divmod.org/svn/Divmod/trunk/
I would have said "rewrite" rather than "refactor", but of course Everybody knows that's stupid. Plus, we are mostly migrating our old code base and cleaning it up along the way; the only component getting a complete rewrite is Atop - the rewrite of which is so fundamentally better that we came up with a better name: "Axiom".

To clarify the naming situation: the new Divmod repository has Axiom and a new version of Mantissa, Python package name 'xmantissa' to avoid package name conflicts during the transition period. eventually it will contain a package for Quotient, Sigma, and several other things. During the transition the Python package names will all start with 'x' but I will still refer to them by their project names, since the older projects will go away and the module names will eventually change.

Initially I was very concerned when we began the experiment that lead to this code revolution. It began, as bad ideas are wont to do, as a joke. I mentioned some client work that we are doing to JP, (who had already been rewriting some things) and my difficulty in choosing an appropriate persistence solution given some of the maintenance issues we'd been having with Atop.

Exasperated after an hour of discussion, JP said, "Why don't you just use SQLite". Now, I'd looked into it some time ago and (the ostensible punchline of the joke) it was garbage. However, that was SQLite version 1, not SQLite3, which has a different API and several critical features that made it considerably more appropriate to our tasks.

A few hours later I had a working prototype of maybe half of the functionality from our existing database. I was suitably impressed; SQLite was giving us all the benefits of SQL (ad-hoc queries and indexing, relational operations on data, a fast query engine) without any of the drawbacks (difficult to customize, unportable server, fragile and time-consuming deployment). I realized that we had got something radical on multiple levels.

In the past I've been very conservative about telling people when and whether to use Divmod's open-source released software, With this new system, I say: jump in. Use it. Only a week into implementation it might be a bit premature to launch a production system with it just yet, but indications are very positive that we will be able to do just that within the month.

The code is shorter, clearer to read, easier to maintain, and the database is Pickle Free℠.

(Okay, it's not really a service mark, but it should be. Pickle is the winner for causing problems for us.)

We are building from our experience with 5 previous persistence systems, 3 previous plug-in frameworks, and 4 previous authentication databases. A curious side-effect of all that experience - and the effect that the title is referring to - is that certain development methodology concepts become irrelevant. Most notably, "YAGNI", from XP, is no longer of any use: we know exactly how much extensibility we need. At every point in implementing this system we have known whether to fuse a component together because we'd built unnecessary additional complexity into previous systems, and where to use a plug-in architecture because we'd needed to inject ugly code into the middle of a monolithic routine.

As a result, where our architecture was heavily monolithic before, now it is almost entirely composed of plugins. It is so plugin-happy, in fact, that there is a database with Service plugins in it, which activate when the database is started from twistd; it contains its own configuration, including port-numbers, so nothing need live in a text configuration file. The web application system is built around this as well; so there is a plugin lookup for invoking raw nevow IResource implementors without sessions (for example, for XML APIs), then IResource implementors which do require a session, then IResource implementors which are specific to a particular user, and finally Fragment instances which plug into a generic hierarchical navigation system. At each level there is a distinct and clear place to put new plugin code, and large portions of it are self-similar. For example, you can install the hierarchical navigation both onto a public site and onto a user's private application pages, since the "web server" implementation is an IResource which can be installed either onto a toplevel database, or any user's personal database.

Oh, and did I mention - LivePage support built right in?

The net result of this is that you can build themeable, multi-user web applications with the code that's in SVN right now. The example isn't visually appealing, but the code is nice, and it's composed from a stack of plugins.

I'm very excited about the possibilities of what we'll be transforming our system and our application into within the next few weeks. I'd like to invite everyone who has been interested in Divmod's open source work in the past to have a look at the new repository, and consider coming to #divmod on irc.freenode.net to look for something to hack on. Considering the higher-level and easier-to-understand nature of the implementation of Axiom vs. Atop, I would also love it if we could find some people to help us document it right from the start.

So - anybody out there looking for an open source project's website to maintain?

Tags: , , ,
Current Mood: working
Current Music: Another One Rides The Bus (by Weird Al Yankovic on "Permanen

4 comments or Leave a comment
From: ianbicking Date: July 31st, 2005 06:03 pm (UTC) (Link)


Are you going to write documentation to go along with your refactoring? That can be a most revelatory experience, nearly as big an improvement on API design as unit testing.
glyf From: glyf Date: August 1st, 2005 07:53 am (UTC) (Link)

Re: Docs...

At what point in the development schedule would you recommend doing this? I'd like to establish some better documentation habits - I've gotten my testing habits in pretty good shape because I've noticed that writing tests can actually accelerate the development process if you do it before you write code (in cases where the design is clear) or after you've got enough code scaffolding to determine what the shape of the system is (in the cases where you're making it up where you go along).

Is there a technique with which documentation can actually speed things up or at break-even in terms of its benefit to the development process? I'm operating on not just one, but 3 *simultaneous* really tight schedules, so I don't have the luxury of lots of extra time...
glyf From: glyf Date: August 1st, 2005 07:54 am (UTC) (Link)

Re: Docs...

I also ask because I've had several bad experiences with writing copious documentation for parts of systems that got thrown out, and having sparse docs for the parts that got kept...

(Yeah yeah, I know - document everything well and you won't have that problem :))
From: ianbicking Date: August 1st, 2005 05:12 pm (UTC) (Link)

Re: Docs...

Yeah... it's always hard in practice as well. Interfaces are certainly a good place to start, but I imagine you are already doing that. That's like an outline of what is essential, and what is just implementation. Even though I'm not using interfaces for anything functional, I've started writing them just for documentation. Adding notes on contracts, expected types, etc., could be good.

The other being the illustration of a reasonable use case. Formal interfaces keep the interface clean, but it's easy to leave the user with lots of boilerplate to write. I think writing the example as a doctest can be good -- more tests are always good, of course, and system tests are important even with extensive unit tests. They tell you when you've changed what outsiders will see; unit tests are so granular there's little resistance to changing the tests to fit the code.

It can be a pain in the ass to make a full example doctestable, but I think that's an easy effort to justify really.
4 comments or Leave a comment