Log in

entries friends calendar profile Scribbles in the Dark Previous Previous Next Next
Slaying Medusa - Please Visit http://glyph.twistedmatrix.com/ - This Blog Is Closed.
Sorry about the ads, I can't turn them off.
Slaying Medusa
That is not dead, which can eternal lie
and in strange aeons, even death may die

Apparently some guy is trying to resurrect Medusa. I wasn't going to say anything until he mentioned Twisted directly, calling it "unprofitable", and "a complex maze of orthogonal but impractical interfaces", but IT'S ON NOW.

Medusa was great for its time. I read the whole thing before I started work on Twisted. In fact, Medusa can be credited with opening my eyes to the problems with threads. Somewhere, locked in a lead drum, encased in concrete, the very first version of Twisted Reality for Python can be found, using threads for concurrency and pickle for serialization.

However, even when I was starting with Twisted - circa 1999 - Medusa was already showing its age.

The main problem with Medusa is that it does too little. For example, Windows sockets are different from UNIX sockets in a variety of ways you probably aren't going to anticipate if you are calling self.recv and such in an asyncore.dispatcher. Those are just the differences in sockets - Twisted handles threads, pipes, and processes too, which a cursory look at the new "Allegra" reveals only rudimentary support for.

95% of the time, you won't notice these problems. As they're related to operating-system error reporting, they tend to show up only when your program is under load. In other words, you won't notice any issues with your program until a lot of people are using it and there is a huge amount of pressure on you to fix it immediately. Luckily Allegra has no automated tests, so you won't have to be troubled by finding out about bugs too early. Twisted has more than 2500 tests run on 4 different platforms with each commit.

Don't believe me that there will be issues with your super-simple cross-platform select loop? By way of an example, after years of maintaining their own networking core, BitTorrent has started using Twisted, with the unobtrusive changelog message of "TCP Stack flaking out bug fixed, using Twisted".

Even given the testing situation, Medusa (I'm sorry, "Allegra") is, indeed, quite a bit simpler than the full Twisted suite, and an adequate solution to 95% of the asynchronous-socket-programming problems out there. However, it's only adequate, whereas I think Twisted is excellent.

As per my last post, there is a lot of complexity in some projects surrounding Twisted, and some of it is unnecessary. That complexity isn't there if you are just trying to get basic asynchronous I/O going, though. If you are considering Medu^HAllegra for some reason, have a look here first: Writing Servers with Twisted. If you can't figure that out, I've just saved you some time: you should just stop trying to write networked programs. "a complex maze", "impractical"??? The simplest working server is just 4 lines of code: an import, a class statement, a method definition, and a method body. You can really be productive that fast. I don't think it's even possible to be simpler than that in a Python framework.

A similar document for Medusa, Asynchronous Socket Programming, has more than twice as many words, and the simplest example is more than 4x longer.

As to "unprofitable", well, let me just direct you to a page that describes people's profitable experiences with Twisted: including the quote "The quality of the Twisted networking core is unmatched in the open- or closed-source arenas".

I'm glad to see someone doing a project that aims to do something similar to Twisted, since that indicates it's a thing that people need doing. I'd love to push for some interoperability standards if a different framework were to emerge that had different implementation strengths than Twisted does (and goodness gracious, it could certainly be faster. Twisted can only handle a few tens of thousands of requests per second on high-end hardware, C++ servers are in the millions these days).

However, there are a few things such a framework would have to get right first:
  1. Write some automated tests. Seriously, it's 2006 already, unit tests are an accepted best practice pretty much anywhere that people care about writing programs that work even some of the time.
  2. Be clear about what you're trying to do differently, or don't say anything at all. Here's a hint: any phrase involving the word "lightweight" is not specific. Reading the descriptions of Python frameworks (although this mainly applies to tiny one-off web frameworks) I am reminded of the descriptions of WindowMaker themes at the beginning of the themes craze five years or so ago. "It's a lightweight, simple, fast, clean, dark theme, with a picture of Neo on the desktop" probably described 4 out of every 5 themes to hit the site. The first four adjectives regularly show up in descriptions of frameworks. Where were all the heavy, slow, dirty, frameworks that these are in response to? If these are valid design tradeoffs, why don't people ever advertise as the opposite? Those four adjectives are basically meaningless - when an author of some software says it's "fast" and "lightweight" they just mean "I know what it does, so it doesn't surprise me when it's slow, and I can get things done quickly with it, because I wrote it." If it were measurably fast or small they would be quoting statistics at you about how many requests per second it can perform or how small the image can be compressed.
  3. Have an application. It looks like Mr. Allegra is, in fact, driving the requirements for his new-old framework from an application, and that's good. If his approach doesn't work well, he'll find that out, rather than assuming that it is fine.
  4. Finally, specific to asynchronous I/O frameworks: separate your transport from your protocols. Don't force your users to fetch data themselves from the OS; there is just no reason for that. It is a real pain to implement a low-level enough API in Twisted for compatibility, if your users expect to diddle the sockets themselves.
13 comments or Leave a comment
From: tarlano Date: February 2nd, 2006 03:39 pm (UTC) (Link)

Asyncchat, Asyncore, Medusa vs. Twisted

I have used both Twisted and Asyncchat, Asyncore, and Medusa for a variety of projects.

I agree that Twisted is overkill for 90% of the applications I write that use a socket.

I think Asyncchat and Asyncore are really two of the nicest standard modules and I think that if you read the code that they don't get in your way like twisted and are not really at any "lower" level..

Sorry to disagree with your point of view.

grimmtooth From: grimmtooth Date: February 2nd, 2006 05:55 pm (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

I don't get this "get in the way" thing on Twisted. As has been mentioned, a fully functional server is just a matter of a few lines of code. A web server - complete with virtual hosts - is less than twenty lines of code. That's seriously not getting in the way.

I'm also recalling my examination of the asyncore source code a while back (when I was trying to suss out why it seemed to be so flakey under load) and seem to recall that there wasn't much there to deal with the peculiarities between platforms. Maybe that's been improved since then, but I don't recall seeing anything in the changelogs for the last several versions of Python. As Glyf pointed out above (and via the links above) there are serious issues between W32 and posix environments where sockets are concerned.

My 2c on that.
From: tarlano Date: February 2nd, 2006 06:28 pm (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

I guess you are reading my response from a particular angle. I *do* think that twisted gives you "everything and the kitchen sink" and as Glyf said "The main problem with Medusa is that it does too little".

The point that I was making is that at least personally I need everything only about 10% of the time and at other times sometimes less is more.

When you need everything that twisted offers (reactor, interfaces, demonizer, configurator, adaptors, etc.) then twisted is the best choice for your requirements, but that is not always. I think that both have their merits and that both should be respected for their design decision. As apple used to say "sometime more is more and sometimes less is more"

grimmtooth From: grimmtooth Date: February 2nd, 2006 06:42 pm (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

I can understand that perspective. I personally use maybe 5% of Twisted's potential in even the most complex thing I do.

Thing is, I've found that for all its size and extras, I'm still happier with it for the simple stuff than asyncore. All that extra stuff doesn't end up in memory every time I write a handler subclassed off of the Protocol class, for example. The other stuff is available if I need it, but if I don't, fine. The same applies to most of Python's standard library. I certainly have no need of the various flavors of webserver that ships with Python every time, but so what?

Heck, I don't ever anticipate 1/100th the load that Twisted is capable of handling, but it's nice to know I have the headroom. And it's nice knowign that the designers of Twisted have taken the time to investigate the many differences across platforms and gotten most (not all) of it to work reliably no matter what.

How does asyncore deal with the differences between Win sockets and posix? Does it abstract these differences out to a consistent way to handle them, or do you have to account for the differences yourself? How well does it integrate with other event loops, such as GUIs like wx? These are all rather common concerns, nothing fancy. The effort of writing a simple RSS reader brought all of these issues into focus. That's not such an unusual thing, is it?
From: zooko Date: February 6th, 2006 06:05 pm (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

I developed and maintained an async I/O and event queue library on top of asyncore for a few years for the Mojo Nation nee Mnet project. I'm quite happy to be using Twisted instead nowadays.

If someone out there really wants to go that other way, you might want to grep for "asyncore" in the Mnet source and pick up half-a-dozen work-arounds that we had to layer atop asyncore in order to run on both Windows and Linux...

grimmtooth From: grimmtooth Date: February 6th, 2006 07:58 pm (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

... an async I/O and event queue library on top of asyncore for a few years for the Mojo Nation nee Mnet project.

Oh, yeah, that takes me back.
glyf From: glyf Date: February 3rd, 2006 01:19 am (UTC) (Link)

Re: Asyncchat, Asyncore, Medusa vs. Twisted

I don't mind that you disagree. I know there are lots of places that Twisted isn't appropriate, lots of audiences it doesn't cater to.

What I do mind is that (like pretty much everyone else with this set of opinions) you haven't explained yourself at all. If you have, as you say, used Twisted for a variety of projects, then you presumably have an example of some specific situation where Twisted's "overkill" has been a problem, then please mention them. I'm certainly convinced that Twisted is better, and I'm not going to stop working on it, so the only positive effect your comments might have would be to improve it. Comments like "it's overkill" or "it's too big" can't really elicit a more productive response than "no, it isn't".

Medusa is lower-level than Twisted though, that is not opinion, it's fact. While there is no formal definition for "lower" versus "higher" level, I think that it's easy to recognize that "here is some data, process it in your application" is higher-level than "a socket has become available for reading. Please call recv(2) and handle all of: EBADF EAGAIN ECONNREFUSED EFAULT EINTR EINVAL ENOMEM ENOTCONN ENOTSOCK, if you are on UNIX, or WSAEINTR WSAEACCES WSAEFAULT WSAEINVAL WSAEMFILE WSAEWOULDBLOCK WSAEINPROGRESS WSAEALREADY WSAENOTSOCK WSAEDISCON WSANO_RECOVERY (...) if you are on Windows."

Granted, Twisted doesn't handle all of those errors gracefully, it's not perfect. The ones it doesn't, though, you can handle separately from your application. I think that is pretty much the definition of low- versus high-level - Twisted has a discrete level for "handling sockets" whereas in asyncore your entire application is sitting at that same level.

So, maybe you could post some code? Give an example of something that is graceful and easy with Asyncore and difficult or overcomplex with Twisted?
grimmtooth From: grimmtooth Date: February 2nd, 2006 04:55 pm (UTC) (Link)
Well, he's got gumption, I'll give him that. But I'd be certain to have something to deliver before I went after the Big Dog.
inklesspen From: inklesspen Date: February 3rd, 2006 05:53 pm (UTC) (Link)

On Documentation

I looked at Twisted about half a year ago, and at the time, I couldn't find any usable documentation. There were the sort of little tutorials like you linked to, that get a simple four line server running, but won't actually do anything a person actually wants to do. Or, there's high-level API documentation that assumes you already have a handle on everything Twisted does. But there was nothing to bridge the gap.

I ended up using asyncore.

Now, maybe Twisted has gotten its documentation head out of its ass in the meantime, but it's gonna take a lot to make me go look, since the last time I tried it, I wasted two days before chucking it out.
glyf From: glyf Date: February 4th, 2006 12:57 am (UTC) (Link)

Re: On Documentation

This is usually where advocacy discussions like this end up, and here's the part where I encourage you to file doc bugs in the tracker, to describe what wasn't documented and help us improve the documentation.

However, prominently featured on the front page of twistedmatrix.com, there's an O'Reilly book out about Twisted now. It's not the Ultimate Reference that we really need, but it does provide a useful high-level introduction to a lot of different Twisted concepts, and lots of example code.
inklesspen From: inklesspen Date: February 4th, 2006 01:02 am (UTC) (Link)

Re: On Documentation

Congratulations on the book. I'm sure it's quite good. But at $30, I won't be buying it anytime soon. Not that I don't believe in paying money for software books; you won't find me stealing it either. I just don't have the money for a book on a framework I don't need to use for work.

(Incidentally, I'd be careful if I were you. Perhaps I'm being cynical, but it might become all too easy to let the volunteer work of writing documentation slide, since after all, the users can just buy the book. I'm sure you know where this can lead.)
glyf From: glyf Date: February 4th, 2006 01:40 am (UTC) (Link)

Re: On Documentation

Where can it lead? I'm not aware of any project where that particular kind of documentation lapse has happened, least of all Python; there are easily 100x as many words written about it commercially, and yet people are still writing entire free books about it.
From: zooko Date: February 6th, 2006 06:00 pm (UTC) (Link)

I'm a prophet.

On 2006-01-30, on #twisted, I wrote:

"Twisted is almost big and successful enough for it to become uncool, widely criticized, and beset by numerous upstart competitors."
13 comments or Leave a comment