Mathieu Fenniak's Weblog

I just returned to Calgary from the Vancouver Python Workshop. Cecil and I drove out there last week (Thursday evening/Friday morning). Catsy flew out from Toronto. We all stayed at the Hyatt Regency in downtown Vancouver, where much fun was had watching the Back to The Future trilogy, and making up entertaining stories about elves.

The workshop was reasonably well organized, had a nice venue, and was well attended. The talks varied in quality from okay to excellent – all the speakers were well informed folk with interesting topics, but very few programmers are excellent public speakers. That being said, I have a few specific suggestions for the workshop itself:

• Some talks could have benefited from the presence of a strict moderator. On a scale from polite to rude, the following things happened:

  • A speaker deferring a question to Guido. This is appropriate, as it is the prerogative of the speaker to defer during his own presentation, and Guido may be an excellent resource for an answer.
  • Guido interjecting a comment like "It’s not happening." while an attendee asks a question (referencing a PEP) is more questionable, but ultimately "polite enough" in the company of geeks.
  • Getting into an audience debate about wxWidgets during a PyObjC talk is not very polite.
  • Nitpicking the usefulness of a contrived optimization example is rude, pointless, and time consuming.

• An entire track on the second day ended up being dedicated to Plone. I think Plone is cool, but these talks unfortunately ended up being about Plone from the point of view of a user rather than a Python developer. I believe that the workshop did not have quite as many speaker submissions as they wanted, so these Plone talks weighed in based upon how many people were willing to do them. They were good, but badly targeted and abundant.

  I would suggest that the conference organizers be willing to say "no" to speakers if they have an abundance of talks on one such subject. I realize getting submissions for talks must be a difficult process, and the fear of having not enough content at a conference must be pretty big as an organizer.

I enjoyed the workshop greatly. I should have flown out rather than drive 12 hours, but it seemed like a good idea at the time. I picked up a lot of good ideas, started in writing some interesting PyObjC code (a game of interracial life, oooh), enjoyed using SubEthaEdit in public for the first time, and ate a lot of good food. Catsy seems to have had her own kind of fun, too.

For the past couple weeks, I’ve been thinking about mathematical model development. There are lots of great tools out there to help with such tasks, like Mathcad and Mathematica. But if you’re doing software development, once you’ve built and tested a model, what you really want is code. Your Mathcad files are great for documentation, testing, and development of your model, but they can’t be embedded in your Java or C++ application.

Additionally, it’s very easy to use functions from those pieces of software which can’t be easily replicated in your software. They have very optimized methods for root solving, matrix math, symbolic derivative calculations, and other such tasks that you can’t reproduce without years worth of effort. So, when you’re developing a model that’s going to be used in software, what’s the easiest way to do it?

Python, of course. Python code tends to be very legible and terse, while still being well suited for mathematical programming. In fact, the syntax and control structures even look pretty similar between Mathcad and Python:

Part of a root solver implemented in Python.

Part of a root solver implemented in Mathcad.

So, it seems natural to me to use Python to develop models. My code can easily be translated into other programming languages after the fact, and it tends to have a more proper programming structure if it’s not translated from Mathcad. Plus, translating a big chunk code from Mathcad can be a real pain in the ass to do manually – there’s just too much to miss, and it can be very time consuming to confirm it’s been done correctly.

Python is not quite a silver bullet in this case, though. Mathcad has a lot of tools for visualization which are very useful when developing a new model. Python’s wonderful abillity to interoperate makes it fairly easy to leverage plotting capabilities of an application like Microsoft Excel though. So, without any further leadup, here’s a quick process which will get you up and running with an Excel plotting Python program:

1. Get and install Python, and the PythonWin extensions (or install ActivePython, which contains all the necessary tools and more).

2. Run the PythonWin IDE program, and generate static COM wrappers for Microsoft Excel. This is as easy as selecting the COM Makepy Utility option from the Tools menu of PythonWin, then selecting the most recent version of the Microsoft Excel n.m Object Library available:

   

   Selecting COM Makepy utility from PythonWin’s menu.

   The static COM wrappers must be used in order to access the Excel constants (of which there are hundreds).

3. Import the COM Dispatch function and the constants namespace into your application:

   
   from win32com.client import Dispatch, constants
   

4. At this point, all that’s left is to go to town on automating Excel. There is a lot of documentation that comes along with Excel, as well as a big chunk of MSDN content that will also help. Let’s dive in, and I’ll just throw an XY scatter plot at you:

   def plot(x, y, xAxisLog=False, yAxisLog=False):
       # acquire application object, which may start application
       application = Dispatch("Excel.Application")
   
       # create new file ('Workbook' in Excel-vocabulary)
       workbook = application.Workbooks.Add()
   
       # store default worksheet object so we can delete it later
       defaultWorksheet = workbook.Worksheets(1)
   
       # build new chart (on seperate page in workbook)
       chart = workbook.Charts.Add()
       chart.ChartType = constants.xlXYScatter
       chart.Name = "Plot"
   
       # create data worksheet
       worksheet = workbook.Worksheets.Add()
       worksheet.Name = "Plot data"
   
       # install data
       xColumn = addDataColumn(worksheet, 0, x)
       yColumn = addDataColumn(worksheet, 1, y)
   
       # create series for chart
       series = chart.SeriesCollection().NewSeries()
       series.XValues = xColumn
       series.Values = yColumn
       series.Name = "Data"
       series.MarkerSize = 3
   
       # setup axises
       xAxis = chart.Axes()[0]
       yAxis = chart.Axes()[1]
       xAxis.HasMajorGridlines = True
       yAxis.HasMajorGridlines = True
       if xAxisLog:
           xAxis.ScaleType = constants.xlLogarithmic
       if yAxisLog:
           yAxis.ScaleType = constants.xlLogarithmic
   
       # remove default worksheet
       defaultWorksheet.Delete()
   
       # make stuff visible now.
       chart.Activate()
       application.Visible = True
   
   def genExcelName(row, col):
       """Translate (0,0) into "A1"."""
       if col < 26:
           colName = chr(col + ord('A'))
       else:
           colName = chr((col / 26)-1 + ord('A')) +\
               chr((col % 26) + ord('A'))
       return "%s%s" % (colName, row + 1)
   
   def addDataColumn(worksheet, columnIdx, data):
       range = worksheet.Range("%s:%s" % (
           genExcelName(0, columnIdx),
           genExcelName(len(data) - 1, columnIdx),
           ))
       for idx, cell in enumerate(range):
           cell.Value = data[idx]
       return range
   

   I suppose that at least an explanation of what this is doing would be a good thing. This program will cause Excel to be run, and a new file to be created. The new file will contain one worksheet with the contents of the x/y arrays in columns A and B. It will also contain one sheet-location chart, an XY scatter, with the x/y points plotted. The x and y arguments to plot are sequences of numbers, and the two flags indicate whether the X and Y axis should be logarithmic or linear.

5. Some examples of using plot:

   # A simple example:
   plot( (1,2,3,4,5), (6,7,8,9,10) )
   
   # Some more data:
   x, y = [], []
   for i in range(100):
       x.append(i)
       y.append(i ** 2)
   plot(x, y)
   
   # Using log axises:
   plot(x, y, True, True)
   

6. Remember, there are good tools available if you’re not in Windows too. gnuplot and gnuplot-py provide a pretty nice graphing environment which is probably quite a bit more capable than Excel.

Ian Bicking’s comment suggesting I wasn’t far away from getting the MutableNumber class to work was quite correct. I had never considered using a descriptor, which made it difficult to make this work with a metaclass. The following code, though, does work:

class Operator(object):
    def __init__(self, instance, op):
        self.instance = instance
        self.op = op
    def __call__(self, rhs):
        try:
            return self.op(self.instance.number, rhs.number)
        except AttributeError:
            return self.op(self.instance.number, rhs)

class OperatorDescriptor(object):
    def __init__(self, op):
        self.op = op
    def __get__(self, obj, type=None):
        return Operator(obj, self.op)
    def __set__(self, obj, value):
        raise TypeError
    def __delete__(self, obj):
        raise TypeError

class NumberDelegateMetaclass(type):
    def __init__(cls, name, bases, dct):
        for attr in cls.__operatormap__:
            setattr(cls, attr, OperatorDescriptor(getattr(operator, attr)))

class MutableNumber(object):
    number = None

    __metaclass__ = NumberDelegateMetaclass
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    def __init__(self, value=None):
        self.number = value

It’s a miracle.

Occasionally, even Python violates the principal of ‘least surprise’ – that is, software should do what you expect it to do. Overriding mathematics operators in new-style classes led me to find one such violation, and investigate the reason behind it.

I setup myself a make-work project a few days ago when I probably should have done a dozen other things. After reading a posting to python-dev entitled Python is faster than C, I was motivated to create a pure-python implementation of a matrix class. I was interested in building it primarily as a refresher of matrix mathematics, and also to experiment with psyco‘s x86 optimization capabilities.

I ran into a quirk of new-style classes. I envisioned the Matrix class having two hidden data fields, one to hold a tuple of Vector objects representing each row, and a second one to hold a tuple of Vector objects representing each column. This would allow for easy access to the data based upon rows or columns, whichever was more convenient, with no speed penalty. Each tuple of Vectors ought to point to the same numbers, though. How does one do this with Python?

I’d like self.__cdata[0][0] and self.__rdata[0][0] to always be the same number, even if I only change one. I could manage this with accessor methods that make sure they’re the same, but I decided to wrap each number in an object called a MutableNumber. The class was setup like so:

class MutableNumber(object):
    def __init__(self, value = None):
        self.number = value

Every mathematical operator should have then been overridden like this:

...
    def __mul__(self, rhs):
        try:
            return MutableNumber(self.number * rhs.number)
        except AttributeError:
            return MutableNumber(self.number * rhs)
...

What a pain to type and maintain, though. What if I wanted to add another possibility in the future, or change the operator to not return a new MutableNumber object wrapping the number? There are twenty-two operators I wanted to override (eg. __mul__, __eq__, __div__, __floordiv__…), and I didn’t want to have to type them, even once.

I originally setup a meta-class to delegate every access to a property in the class’s __delegateattributes__ member to self.number. This had some issues:

>>> MutableNumber(2) * 3
6
>>> MutableNumber(2) * MutableNumber(3)
Traceback (most recent call last):
  ...
TypeError: unsupported operand types for *: 'int' and 'MutableNumber'

I had to go a more complex route, rather than just pure delegation. I eventually replaced all the operators with custom instances of an Operator class, which held an operator and a reference to the instance of the class:

class MutableNumber(object):
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    class Operator(object):
        def __init__(self, instance, op):
            self.instance = instance
            self.op = op
        def __call__(self, rhs):
            op = getattr(operator, self.op)
            try:
                return op(self.instance.number, rhs.number)
            except AttributeError:
                return op(self.instance.number, rhs)

    def __init__(self, value=None):
        for op in self.__operatormap__:
            setattr(self, op, MutableNumber.Operator(self, op))
        self.number = value

So, we finally reach the part where it gets interesting. What happens? Does this work properly? The code seems good, but it fails to function:

>>> MutableNumber(3) * 2
Traceback (most recent call last):
  ...
TypeError: unsupported operand type(s) for *: 'MutableNumber' and 'int'

Unsupported operand for types? It looks like it’s totally ignoring the fact that the MutableNumber instance has a __mul__ property set. Hm… stranger still:

>>> MutableNumber(3).__mul__(2)
6
>>> MutableNumber(3).__class__.__mul__(2)
Traceback (most recent call last):
  ...
AttributeError: type object 'MutableNumber' has no attribute '__mul__'

Well, it seems that A * B does not call A.__mul__(B). It searches for A.__class__.__mul__, but not A.__mul__. It works fine with an old-style class, but not a new-style class.

It turns out this behaviour has been discovered before. In fact, Jan Decaluwe was trying to do the same thing I am, create an instance object that acts like a mutable number. Alex Martelli wrote:

The idea is that operators should rely on special
methods defined by the TYPE (class) of the object they’re working
on, NOT on special methods defined just by the OBJECT itself and
not by its class. Doing otherwise (as old-style classes did, and
still do for compatibility) is untenable in the general case, for
example whenever you consider a class and its metaclass (given that
the metaclass IS the class object’s type, no more and no less).

Well, too bad. I guess maybe I’ll just backtrack all the way to using accessor methods to make sure the column and row data is both kept constant… or just store one of them.