Mathieu Fenniak's Weblog

Ian Bicking’s comment suggesting I wasn’t far away from getting the MutableNumber class to work was quite correct. I had never considered using a descriptor, which made it difficult to make this work with a metaclass. The following code, though, does work:

class Operator(object):
    def __init__(self, instance, op):
        self.instance = instance
        self.op = op
    def __call__(self, rhs):
        try:
            return self.op(self.instance.number, rhs.number)
        except AttributeError:
            return self.op(self.instance.number, rhs)

class OperatorDescriptor(object):
    def __init__(self, op):
        self.op = op
    def __get__(self, obj, type=None):
        return Operator(obj, self.op)
    def __set__(self, obj, value):
        raise TypeError
    def __delete__(self, obj):
        raise TypeError

class NumberDelegateMetaclass(type):
    def __init__(cls, name, bases, dct):
        for attr in cls.__operatormap__:
            setattr(cls, attr, OperatorDescriptor(getattr(operator, attr)))

class MutableNumber(object):
    number = None

    __metaclass__ = NumberDelegateMetaclass
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    def __init__(self, value=None):
        self.number = value

It’s a miracle.

Occasionally, even Python violates the principal of ‘least surprise’ – that is, software should do what you expect it to do. Overriding mathematics operators in new-style classes led me to find one such violation, and investigate the reason behind it.

I setup myself a make-work project a few days ago when I probably should have done a dozen other things. After reading a posting to python-dev entitled Python is faster than C, I was motivated to create a pure-python implementation of a matrix class. I was interested in building it primarily as a refresher of matrix mathematics, and also to experiment with psyco’s x86 optimization capabilities.

I ran into a quirk of new-style classes. I envisioned the Matrix class having two hidden data fields, one to hold a tuple of Vector objects representing each row, and a second one to hold a tuple of Vector objects representing each column. This would allow for easy access to the data based upon rows or columns, whichever was more convenient, with no speed penalty. Each tuple of Vectors ought to point to the same numbers, though. How does one do this with Python?

I’d like self.__cdata[0][0] and self.__rdata[0][0] to always be the same number, even if I only change one. I could manage this with accessor methods that make sure they’re the same, but I decided to wrap each number in an object called a MutableNumber. The class was setup like so:

class MutableNumber(object):
    def __init__(self, value = None):
        self.number = value

Every mathematical operator should have then been overridden like this:

...
    def __mul__(self, rhs):
        try:
            return MutableNumber(self.number * rhs.number)
        except AttributeError:
            return MutableNumber(self.number * rhs)
...

What a pain to type and maintain, though. What if I wanted to add another possibility in the future, or change the operator to not return a new MutableNumber object wrapping the number? There are twenty-two operators I wanted to override (eg. __mul__, __eq__, __div__, __floordiv__…), and I didn’t want to have to type them, even once.

I originally setup a meta-class to delegate every access to a property in the class’s __delegateattributes__ member to self.number. This had some issues:

>>> MutableNumber(2) * 3
6
>>> MutableNumber(2) * MutableNumber(3)
Traceback (most recent call last):
  ...
TypeError: unsupported operand types for *: 'int' and 'MutableNumber'

I had to go a more complex route, rather than just pure delegation. I eventually replaced all the operators with custom instances of an Operator class, which held an operator and a reference to the instance of the class:

class MutableNumber(object):
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    class Operator(object):
        def __init__(self, instance, op):
            self.instance = instance
            self.op = op
        def __call__(self, rhs):
            op = getattr(operator, self.op)
            try:
                return op(self.instance.number, rhs.number)
            except AttributeError:
                return op(self.instance.number, rhs)

    def __init__(self, value=None):
        for op in self.__operatormap__:
            setattr(self, op, MutableNumber.Operator(self, op))
        self.number = value

So, we finally reach the part where it gets interesting. What happens? Does this work properly? The code seems good, but it fails to function:

>>> MutableNumber(3) * 2
Traceback (most recent call last):
  ...
TypeError: unsupported operand type(s) for *: 'MutableNumber' and 'int'

Unsupported operand for types? It looks like it’s totally ignoring the fact that the MutableNumber instance has a __mul__ property set. Hm… stranger still:

>>> MutableNumber(3).__mul__(2)
6
>>> MutableNumber(3).__class__.__mul__(2)
Traceback (most recent call last):
  ...
AttributeError: type object 'MutableNumber' has no attribute '__mul__'

Well, it seems that A * B does not call A.__mul__(B). It searches for A.__class__.__mul__, but not A.__mul__. It works fine with an old-style class, but not a new-style class.

It turns out this behaviour has been discovered before. In fact, Jan Decaluwe was trying to do the same thing I am, create an instance object that acts like a mutable number. Alex Martelli wrote:

The idea is that operators should rely on special
methods defined by the TYPE (class) of the object they’re working
on, NOT on special methods defined just by the OBJECT itself and
not by its class. Doing otherwise (as old-style classes did, and
still do for compatibility) is untenable in the general case, for
example whenever you consider a class and its metaclass (given that
the metaclass IS the class object’s type, no more and no less).

Well, too bad. I guess maybe I’ll just backtrack all the way to using accessor methods to make sure the column and row data is both kept constant… or just store one of them.

I got a bit bored at my job yesterday. This happens often, and usually a new random piece of software emerges as a result. This time I ended up hacking on a piece of software that had previously emerged from boredom, which I called the "Difference Machine". I added support for plot datasets generated on the fly through Python code, and in the end learned how to distribute Python within my application.

The purpose of the Difference Machine is very simple: I dislike MS Excel, and wish to never have to use it again. As someone who works in a job involving a lot of engineering, Excel is pretty much standard fare for comparing datasets and stuff of that nature, though. The Difference Machine allows plots generated by the software we develop to be imported, and compared against each other. Additionally, text import is possible from a CSV file or from the clipboard.

The Difference Machine comparing two gas supercompressibility (z-factor) correlations.

Adding in the Python embedding was easy. I’ve done similar things before, and the Python C-API is pretty straight-forward (although necessarily verbose). When it came time to put this application up on the network for co-workers to access, the real fun began.

The Difference Machine editing Python code with a crude edit control.

My first attempt was simple. Copy the application onto the network, and include python23.dll in the application’s directory. Behold! It actually worked. But, there were a few minor glitches…

• No access to the standard library. import sys and import math worked, but not import csv or anything else.
• No standard library meant no traceback library, which I had depended upon to format tracebacks when things went poorly. The software did not react to the missing library well.

I needed the standard library. I tried copying it up into a Lib directory in my application’s directory, and this worked okay… except that it was a huge number of files, and it was still missing things like _sre.pyd, the dynamically linked libraries.

I consulted PEP-237, Import Modules from Zip Archives, to try to create a zip file of the entire standard library that would be much easier to manage than my Lib directory. By naming the zip file python23.zip and putting it in the same directory as my python23.dll, it should have been able to access the library. But it still failed because it couldn’t access any dynamically linked libraries, and zlib is necessary to open the zip file.

By putting all the .pyd files into my application’s directory, I finally got it to work. It loads zlib to read python23.zip, where it retrieves the rest of the standard library. Yippee! In the end, the directory contains the following files:

DifferenceMachine.exe
... bunch of my dlls ...
python23.dll
python23.zip
zlib.pyd
_csv.pyd
_sre.pyd
... bunch more pyd files ...

And all is good and happy. I’m one small step closer to being Excel-less.

I was browsing the Daily Python-URL this morning, when I came across a link to a Python module for reading OSX’s iCal files. The module’s web page states:

Apple doesn't document the API for interfacing with iCal, but it does save
it's files as industry standard iCalendar files.  I went looking recently
for a python module to interface with iCal, and couldn't find one.

It’s great to have a good Python module to read standard iCalendar files, but the statement that Apple doesn’t have a documented API for interfacing with iCal is bogus. iCal supports the Open Scripting Architecture (AppleScript’s APIs), the same as every other Apple application does. The Open Scripting Architecture can be used from Python much easier than one can read an iCalendar file, and the information is exposed in a very easy to access format. I wrote an earlier article on how to do similar actions with iTunes, so in my continuing effort to educate the world on how to use OSA from Python, here’s some more!

The first step in using iCal from a Python script through OSA is to generate an iCal module which wraps up the low-level OSA details. The gensuitemodule.py script is designed to do just that. It is located in your plat-mac directory inside your Python installation’s lib directory. On a stock installation of Panther, that makes it in /System/Library/Frameworks/Python.framework/ pant,pant Versions/Current/lib/python2.3/plat-mac/gensuitemodule.py. In order to interface with the system window manager and access OSA, it must be run with the pythonw executable, rather than python. The --output command line parameter can be used to specify a name for the output module. In the end, generating the module is extremely simple:

pythonw {...}/gensuitemodule.py --output iCal /Applications/iCal.app

Now that you have a beautiful iCal module, what do you do with it? The iCal.iCal class is the basic application, which is used for all OSA communications with the iCal application itself. So, to start things off, an instance of that class should be created so we can do some communicating:

import iCal
app = iCal.iCal()

How do we determine what we can do to this application through OSA? The easiest way is to use OSX’s Script Editor, which allows us to open up an application’s OSA dictionary and view the contents. The Script Editor is located in the /Applications/AppleScript folder, and has an option in the File menu to Open Dictionary....

Once you have iCal’s dictionary open, inside the iCal suite you can see all the classes and commands that iCal exposes. For example, the application class has elements like calendar, window, and document. Since we’re interested in getting data out, we know that calendar is where we want to start. An element is a collection of objects, implying that each application has a collection of calendars. Based upon what we know of how iCal works, this sounds very reasonable. Here’s how we can iterate over all the calendars that our iCal has:

numCalendars = app.count(app, each=iCal.calendar)
for i in range(1, numCalendars + 1):
    cal = iCal.calendar(i)

The first line sends an OSA command to iCal asking it to count the number of iCal.calendar objects it currently has. Since Python normally uses zero-based counting, and OSA uses one-based counting, we need to iterate from 1 to numCalendars + 1. We create an iCal.calendar object from each of those indicies. Creating the object doesn’t do any communication with iCal, it just creates an object which can dispatch calls to the appropriate instance of the calendar class.

What if we wanted to get some data out of the object, now? The Script Editor shows that the calendar objects support a number of properties like tint, title, and description. Let’s print out the title of every calender:

print "Calendar %s - %s" % (i, app.get(cal.title))

The magic here is in the app.get(cal.title). This tells the application to retrieve the title property of the cal instance, which is already bound to the proper calendar object in iCal. We simply print it out, which outputs something like:

Calendar 1 - Home
Calendar 2 - Work

Let’s put these chunks of code together, and include the ability to print out every event for each calendar:

import iCal

def printAllEvents(app, calendar):
    numEvents = app.count(calendar, each=iCal.event)
    for i in range(1, numEvents + 1):
        event = calendar.event(i)
        print "\tEvent %s - %s" % (i, app.get(event.summary))

def printAllCalendars(app, eventsToo = True):
    numCalendars = app.count(app, each=iCal.calendar)
    for i in range(1, numCalendars + 1):
        calendar = iCal.calendar(i)
        print "Calendar %s - %s" % (i, app.get(calendar.title))
        if eventsToo:
            printAllEvents(app, calendar)

app = iCal.iCal()
printAllCalendars(app)

Yay! We have access to every piece of information iCal stores, and we didn’t need to parse an iCalendar file. Some parts of iCal even begin to integrate with the OSX Address Book, and we can access that information as well.

One last bit of code could be useful, though. What if we wanted to change some of the data in iCal? It’s fairly simple, but not too obvious or well documented anywhere. Here’s a short snippet that will give all your calendars dumb, pointless names:

import iCal
app = iCal.iCal()

numCalendars = app.count(app, each=iCal.calendar)
for i in range(1, numCalendars + 1):
    calendar = iCal.calendar(i)
    app.set(calendar.title, to = "Calendar %s" % i)

In addition to the pingback library that I built earlier today, I decided to add Trackback support to my weblog software as well. To that end, I’ve created a new trackback.py library which handles the grunt work of being a Trackback client. It’s not exactly very difficult, anyways. The method of greatest concern for a client:

def ping(trackbackURI, sourceURI, title = None, excerpt = None, siteName = None):
    """Implements a trackback ping.  This method throws exceptions based upon
    the error returned from the trackback server, unless it is the successful
    error code '0'.

    The excerpt parameter should contain plaintext for maximum readability on
    trackback servers.  Providing HTML is not recommended.  This library
    provides a function ``detagHTML`` which will shred HTML text for you."""
    ...

This library does not offer support for being a Trackback server, nor autodiscovery in any way. I do not believe that the autodiscovery method suggested by the Trackback specification is appropriate. Embedding RDF data in an HTML comment? This is the kind of thing a perl author would suggest, since they believe they could could regex their way out of the hole they dug for themselves. I don’t like it.

An alternative implementation of trackback in Python is available called tblib. I chose to write my own library because:

1. I was bored at work,
2. I didn’t want to use a GPL-licensed library in my modified-BSD licensed weblog,
3. I didn’t like the way tblib used regular expressions to grab data from ping responses rather than using an XML parser,
4. I didn’t want to implement autodiscovery in a flappy regex perl way. (No disrespect meant towards the author of tblib, of course. I realize that there is no simpler or stronger way to do Trackback autodiscovery.)

I’ve created a simple and easy-to-use Python library to handle the client implementation of the Pingback 1.0 protocol. The pingback.py script handles all aspects of a client, including automatically parsing HTML and reStructredText for hyperlinks which should be checked.

Pingback can now be implemented by other Python projects by using one of the following functions:

def autoPingback(sourceURI, reST = None, HTML = None):
    """Scans the input text, which can be in either reStructuredText or HTML
    format, pings every linked website for auto-discovery-capable pingback
    servers, and does an appropriate pingback."""
    ...

def pingback(sourceURI, targetURI):
    """Attempts to notify the server of targetURI that sourceURI refers to
    it."""
    ...

The documentation at the top of the module describes how one might implement a Pingback server by using the SimpleXMLRPCServer module, as well.