You are viewing glyf

entries friends calendar profile Scribbles in the Dark Previous Previous Next Next
Please Visit http://glyph.twistedmatrix.com/ - This Blog Is Closed. - Functional Functions and the Python Singleton Unpattern
Sorry about the ads, I can't turn them off.
glyf
glyf
Add to Memories
Share
Functional Functions and the Python Singleton Unpattern
Have you ever written a module that looked like this?
subscribers = []

def addSubscriber(subscriber):
    subscribers.append(subscriber)

def publish(message):
    for subscriber in subscribers:
        subscriber.notify(message)
And then used it like this?
from publisher import publish

class worker:
    def work(self):
        publish(self)
I've done this many times myself.

I used to think that this was the "right" way to implement Singletons in Python.  Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.

Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have.  As Wikipedia points out, "It is also considered an anti-pattern since it is often used as a euphemism for global variable."

The module above is brittle, and as a result, unpleasant to test and extend.

It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.

It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish".  Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).

It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up.  If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.

Unfortunately, this probably doesn't seem particularly bad, because bad examples abound.  It's just the status quo.  Twisted's twisted.python.log module is used everywhere like this.  The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.

And there's a good reason that this keeps happening.  Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places.  It seems convenient to have it available, and it basically works.

Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.

First, make your "global" thing be a class.
class Publisher:
    def __init__(self):
        self.subscribers = []

    def addSubscriber(self, subscriber):
        self.subscribers.append(subscriber)

    def publish(self, message):
        for subscriber in self.subscribers:
            subscriber.notify(message)

thePublisher = Publisher()
Second, decide and document how "global" you mean.  Is it global to your process?  Global to a particular group of objects?  Global to a certain kind of class?  Document that, and make sure it is clear who should use the singleton you've created.  At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing,  Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.

Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly.  For example:
from publisher import thePublisher

class Worker:
    publisher = thePublisher

    def work(self):
        self.publisher.publish(self)
In this example, you now have a clean point of integration for testing and extending this code.  You can make a single Worker instance, and change its "publisher" attribute before calling "work".  Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker.  If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.

Does this mean you should make everything into objects, and never use free functions?  No.  Free functions are fine, but functions in Python are for functional programming.  The hint is right there in the name.  If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.

Once you've started adding mutable state into the mix, you're into object territory.  If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods.
Comments
crackmonkey From: crackmonkey Date: July 8th, 2007 03:30 am (UTC) (Link)
I'm curious, how did you test the Singleton classes before using this pattern? At work we actually ask people during interviews to describe their experiences with singletons, and people have consistently reported that testing is highly problematic, and control of scope even more so.

I use Spring (an IoC container on the Java side) for much of the configuration I use, since it encapsulates the scope (instance per VM/session/etc) and insertion into the appropriate objects. What do you do with the apps you build in python?
glyf From: glyf Date: July 8th, 2007 09:13 am (UTC) (Link)
I'm curious, how did you test the Singleton classes before using this pattern? At work we actually ask people during interviews to describe their experiences with singletons, and people have consistently reported that testing is highly problematic, and control of scope even more so.
It's Python. Some things are horrible, but everything is testable. For example, consider this little gem:
class Thing:
    stuffed = False
    def stuff(self):
        self.stuffed = True
globalThing = Thing()
def frob():
    globalThing.stuff()

# Oh no!  A global!  What ever will we do!

from types import FunctionType
def testable(thunk, **newGlobals):
    "This'll fix that global's wagon but good."
    globs = {}
    globs.update(thunk.func_globals)
    globs.update(newGlobals)
    testable = FunctionType(thunk.func_code, globs, thunk.func_name,
                            thunk.func_defaults, thunk.func_closure)
    return testable
def test_frob():
    localThing = Thing()
    frobPrime = testable(frob, globalThing=localThing)
    frobPrime()
    print localThing.stuffed
test_frob()

Coming up with crimes against nature like this takes a little creativity and a little knowledge of the Python runtime. The real trick is figuring out which one is appropriate in a particular nasty situation, and avoiding nasty situations (mostly by avoiding globals).
crackmonkey From: crackmonkey Date: July 8th, 2007 08:02 pm (UTC) (Link)
That is definitely... creative. I wouldn't have come up with that. But my python foo is weak. At least you can do that in Python. Java doesn't leave you many options short of byte code modification.
glyf From: glyf Date: July 8th, 2007 08:54 pm (UTC) (Link)
Oh, sure, it's tedious and unpleasant in Java, but hey - you are using Java, clearly you like tedious and unpleasant things :). But it's often possible even without bytecode hacking.

For example, libraries like JMock achieve similar, if not identical, results by using java.lang.reflect.Proxy and friends.

(And who says bytecode manipulation has to be hard.)
crackmonkey From: crackmonkey Date: July 9th, 2007 02:16 am (UTC) (Link)
Agreed, if you're using interfaces! Which don't matter for things like singletons, since static methods aren't inheritable in Java anyway.

But you're right about tedious and unpleasant. I still look at Python and Ruby with great envy, and I'm picking up Ruby in my spare time, so that's not too bad.

ASM *is* nice, as are many of the bytecode manipulation libraries, but frankly I try and avoid them when possible, in favor of an in-language/in-band option. Lately, I've been using Aspect/J more, and JRockit is cool too, although picking a VM because of a library that Sun patently won't support is a poor long term choice.
glyf From: glyf Date: July 8th, 2007 09:21 am (UTC) (Link)
I use Spring (an IoC container on the Java side) for much of the configuration I use, since it encapsulates the scope (instance per VM/session/etc) and insertion into the appropriate objects. What do you do with the apps you build in python?
I have to confess I don't really understand this question. I have heard it before, and I keep my ear to the ground in the Java-verse, so I hear periodic murmurs about something or other totally revolutionizing Java programming by providing an even newer, even improveder way to initialize configuration and state and manage the lightweight dependency injection of control configuration inversion rama rama ding dong.

I would say "I don't think it's that big of a problem" if I even understood what the problem was that these massive configuration and initialization "container" frameworks were trying to solve. I assume it's not that big of a problem because I seem to be able to solve something similar by spending 30 seconds every six months writing a tac file, or these days maybe a whole ten minutes writing tests and docs and a twistd plugin.

The other possible answer is that I use Axiom, since Axiom applications are completely self-contained universes that have all the application data and "configuration" information and initialization bootstrapping bundled together in a convenient database format.
crackmonkey From: crackmonkey Date: July 8th, 2007 08:14 pm (UTC) (Link)
I'll assume you know what IoC is and skip to the "Why I use it" part. Maybe these aren't issues in Python/Twisted for various reasons, which I'd love to hear about.

A common problem solved by IoC containers is "How do I get a DB connection?". Most of the time, applications rarely care which DB they're connected to, or the way the connection is managed, so long as they can get one and return one to a pool and it's a valid connection. Java has a plethora (too many?) ways to do this. Get it from your ORM, get it from your EJB container, get it from one of the many third party standalone libraries, or meld a combination of those to get exactly what you need.

*Why* this situation occurs is probably more interesting than the solution, but the IoC solution is to expose a setter (or ctor) to allow the container to assign the appropriate object to your target object and your class just uses it, without knowing where it came from. The lifecycle and any dependencies required to instantiate that object (creation of the pool, credentials, etc) are "externalities" of the target object.

So, I can write a simpler class that just "magically" has a DB connection, rather than figuring out how to get the connection in the class itself (usually a bit of work in Java, and inflexibly changed).

Other bonuses are lazy-loading (or not loading) objects and subsystems you don't need. These aren't strictly part of an IoC container, but they're highly useful (Spring provides them, as did many of it's predecessors).

Since I never got around to writing DB apps in Python, and I was new to the language, I ended up writing configuration *in Python* and sourcing it, which was generally fairly easy and elegant (and fairly impossible in Java).

Does that explain the need/want for those sorts of features, and what the original question was getting at?

As a side note, I've never thought of a db format as "convenient", given the knowledge of SQL required, lack of text file editing options, and difficulty of distributing and controlling standard configuration, but I don't know Axiom, and it may get around those issues.
glyf From: glyf Date: July 8th, 2007 09:07 pm (UTC) (Link)
I'll assume you know what IoC is and skip to the "Why I use it" part. Maybe these aren't issues in Python/Twisted (...)
I understand what the design pattern is, I've read fowler's paper, and it's a nice idea when it's appropriate, but it seems necessary so often in the Java universe that something appears distorted to me.

*Why* this situation occurs is probably more interesting than the solution
I believe that's the thing that I'm missing, so yes, I'm curious about it.

So, I can write a simpler class that just "magically" has a DB connection, rather than figuring out how to get the connection in the class itself (usually a bit of work in Java, and inflexibly changed).
Basically, I try to avoid this kind of magic. Pass parameters where you need 'em, when you are passing too many parameters at once, group them and turn them into objects. Nevow has a "magic" object, the "context", which you can get all kinds of juicy awfulness out of, but we are desperately trying to phase it out in favor of people just constructing objects that have all the things they need passed to their constructor. From what I glean, this isn't quite the same kind of magic though, because IoC containers seem to focus very much on the "top" object rather than random components throughout the page.
I ended up writing configuration *in Python* and sourcing it, which was generally fairly easy and elegant (and fairly impossible in Java).
The first Java program I wrote, ever, (I was 15) was a custom classloader which allowed you to load objects to serve as configuration. It might not be quite as spiffy as Python, but with judicious use of object array literal syntax, you can do a surprising number of things. I actually took this idea from java to python, but it sufficied in both areas.
Does that explain the need/want for those sorts of features, and what the original question was getting at?
Kind of. I mean, not really, I think I would need to feel the actual pain of building a real J2EE application to really understand this, and uh no thanks say no to drugs kids stay in school.

As a side note, I've never thought of a db format as "convenient", ...
SQLite changes the game just a little bit, but in many cases that tiny change is enough to make a big difference. Axiom adds quite a bit of magic on top though. Observe:
glyph@illidan:~$ axiomatic mantissa
Use database 'mantissa.axiom'? (Y/n) 
Enter Divmod™ Mantissa™ password for 'admin@localhost': 
Confirm Divmod™ Mantissa™ password for 'admin@localhost': 
glyph@illidan:~$ axiomatic offering install Quotient
Use database 'mantissa.axiom'? (Y/n) 
glyph@illidan:~$ axiomatic userbase list
Use database 'mantissa.axiom'? (Y/n) 
admin@localhost
mantissa-base
Quotient
glyph@illidan:~$ axiomatic web --list
Use database 'mantissa.axiom'? (Y/n) 
The hostname is not set.
Configured to use HTTP port 8080.
Configured to use HTTPS port 8443 with certificate /home/glyph/mantissa.axiom/files/server.pem
Sessionless plugins:
  /static/webadmin => item(DeveloperSite) (prio. -255)
Sessioned plugins:
  / => item(FrontPage) (prio. -257)
glyph@illidan:~$ axiomatic web --port 8880
Use database 'mantissa.axiom'? (Y/n) 
glyph@illidan:~$ axiomatic web --list
Use database 'mantissa.axiom'? (Y/n) 
The hostname is not set.
Configured to use HTTP port 8880.
Configured to use HTTPS port 8443 with certificate /home/glyph/mantissa.axiom/files/server.pem
Sessionless plugins:
  /static/webadmin => item(DeveloperSite) (prio. -255)
Sessioned plugins:
  / => item(FrontPage) (prio. -257)
glyph@illidan:~$ axiomatic start -n
Use database 'mantissa.axiom'? (Y/n) 
2007-07-08 17:05:20-0400 [-] Log opened.
2007-07-08 17:05:20-0400 [-] twistd 2.5.0+rUnknown (/usr/bin/python 2.5.1) starting up
2007-07-08 17:05:20-0400 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2007-07-08 17:05:20-0400 [-] xmantissa.website.AxiomSite starting on 8880
2007-07-08 17:05:20-0400 [-] Starting factory <xmantissa.website.AxiomSite instance at 0x8b48dcc>
2007-07-08 17:05:20-0400 [-] xmantissa.website.AxiomSite starting on 8443
...
crackmonkey From: crackmonkey Date: July 9th, 2007 02:39 am (UTC) (Link)
So, part of the problem with passing this number of parameters (or a magical and abstract and vague "Context" object (which plagues pretty much any framework it seems)) is that it represents a very large and many times ( especially with Java ) unmanageable state object. (If you're curious, Josh Bloch has a presentation with a pattern for a type-safe context object, which takes some of the pain out of the problem, but not much.) In the end, it looks like spaghetti, and like globals, doesn't work so well when mingling different libraries and third party code.

I should clarify that I haven't written a J2EE app in a number of years, and my usage of IoC is odd in that sense. At work, and home, I use Spring specifically for wiring up various configurations of components (so far, none to do with web pages). In Ruby on Rails and elsewhere, configuration is done in code (as you mention later, it can be done, but it's "harder" in Java than using Spring, but possible. Some folks at google created Guice because, well, XML configuration sucks, but it's the same idea). It seems like the composition of parts is being handled by frameworks with plugins and a built-in configuration mechanism, which works well, if you're in a framework.

I agree about avoiding j2ee applications. I was curious more about generalized application configuration, rather than solely web apps (or j2ee). I've seen people do this in lots of ways ( dot dirs in home directories, env vars, file naming patterns, make files (!) and so forth, skipping the horrors that are working on windows ), but I haven't heard of any patterns emerging from any of the projects I keep tabs on (that span projects). Specifically "How do you compose an applications made of parts into something that walks, talks, and does something useful".

Most frameworks seem to give you their way of configuring the pieces and make assumptions about the parts the users don't specify, which works well enough. I generally don't work in an application container or framework, so I'm left to find another means. Hence, why I asked about Python, and Twisted, since that's something you create applications with/in, and it looks like Axiom is what you created to solve the configuration problem. So I'll look into that a bit more.

The original intent of my question was to figure out how you write applications, either in python or Twisted, out of pieces. How do you fit them together (without a framework)?

I should split hairs a bit and say that I don't think of Spring as a framework in the normal sense, in that none of my objects inherit or reference any Spring code (except in my main method, for bootstrapping purposes). There's no coupling involved. I obviously need someway of putting all the pieces together, but the way I do that isn't obvious directly in the code (with the unfortunate side effect of being magical at times). So, I rely on Spring, but my code doesn't know anything about it.
9 comments or Leave a comment
profile
Glyph Lefkowitz
User: glyf
Name: Glyph Lefkowitz
calendar
Back December 2010
1234
567891011
12131415161718
19202122232425
262728293031
page summary
tags