Mathieu Fenniak's Weblog

2004/10/01

Some Nifty Things

Filed under: java, programming, python — admin @ 1:08 am

Lately I seem to have pushed a large number of important projects to the side to make room for some smaller personal projects. Here’s what I’ve been thinking about lately that’s kinda nifty:

  • I gave notice of resignation to my employer earlier this week. October 8th is my final work day. I’ll be free for a few months to persue some contract work that I’ve got waiting in the wings, and then I’m into a new job in the new year. I’ll be working as the head of software development at a small start-up company. Exciting!

  • I built a new photo gallery system. This one builds on a number of features in flickr. It has tag based image catagorizing, EXIF photo information, and a small number of different image sizes that can be viewed. You can see it in action on my pictures page.

  • A new version of Java was released today. Java 1.5, Tiger, contains a number of boring features: autoboxing (saves some typing), a new for/in loop (saves some typing), and generics (saves some typing – may find some ClassCastExceptions waiting to happen at compile time). These seem to be the features everyone thinks are cool, but I think they’re pretty lame.

    That said, there are a couple of features that are cool. Java 1.5 adds the ability to change a function’s return type when it is being overridden. You can only change it to a subclass of the original type, but this makes a lot of sense. When you really think about it… it turns out that all it does is save you some typing. In any situation where you’d actually use this, you’d be doing some casting that you wouldn’t have to do anymore.

    Hmmm… so… what does it have that’s cool? Variable length argument lists… I’ve never missed these in Java before. Annotations look pretty cool, but if you think about the pretty static applications they have unless you recode your own compiler, they seem to be pretty much just a couple nice builtin features and a new way to add documentation. So… static imports? Yeah, that’s nice. Yay!

    I really wanted to be impressed with Java 1.5’s new features, since I’m doing a lot of Java development these days. But I can just type faster, and I’ll still retain backwards compatibility with people using Java 1.4.

  • I added HTTP Digest authentication into my Twisted based weblog aggregator. This allows me to view LiveJournal RSS feeds with a logged in user, and hence getting links to protected LiveJournal entries that m yuser can see. I submitted a small patch to urllib2 to make it work with those same LiveJournal feeds, and I may add real authentication support to twisted.web.client rather than the hacked support I’m currently using. Maybe this weekend, if I have the inclination.

2004/08/29

Roundup

Filed under: programming, python — admin @ 2:47 am

Roundup is some damn beautiful software. It’s a very nice and simple package for software bug tracking (oh, pardon me… issue tracking). It can be customized very easily, and in fact from a minimal ‘tracker’ just about any web-based database application could be built with a minimum of fuss. The mail gateway is a beautiful design too. Oh, and I love the fact that e-mailing the system creates a user “account” for that e-mail address (unless it’s associated with an existing account, of course). No fuss bug tracking.

I’d love if it supported some e-mail security, though. Digitally signed messages, for example. The current complete lack of e-mail security makes me irrationally scared – a bad person couldn’t do much, but they could do some.

Here’s a neat trick – for nice clean URLs, place the roundup.cgi script wherever you want it to be, renamed it to just roundup, and add a couple lines to your Apache configuration:

<Location /blah/roundup>
    SetHandler cgi-script
</Location>

And you’ll magically get the CGI interface of roundup working without the minor annoyance of having ’roundup.cgi’ in your URLs. Go Apache!

2004/08/26

DevEnv vs. the Programmer

Filed under: programming, python — admin @ 3:57 pm

How can you capture the console output of a program, when it buffers that output if you’re not using a console to view it? This was a problem run into when building an automation tool for MS Visual Studio .NET. In the end, the programmer subjugated his tool (as it should be) by beating it over the head with a pipe.

A few of us programmers with in the unfortunately unfriendly environment of MS Windows. It might look pretty and have lots of applications written for it, but it’s basically an unfriendly environment for a software developer. Even MS Visual Studio .NET can be unfriendly to a developer, which is unfortunate since it’s the one program you’d expect would be really friendly.

Visual Studio allows you to provide command-line options which start a software build. Running inside a command prompt, all you need to do is pass a solution file and a build configuration to the program, and you’re off. In fact, Visual Studio even gives you more command-line flexibility by providing two executables, devenv.com and devenv.exe – the former will tend towards printing console output all the time, while the latter will avoid it if a build log file is provided instead.

In the creation of a complete build tool, I wanted to run devenv.com and capture the output so I could display the progress to a user. That’s when it became tough. Running the executable through os.popen (or any other popen function) didn’t accomplish what I wanted – the output being printed to the console (and now being read through a pipe) was buffered inside the devenv process and only printed after the build was completed. Clearly this didn’t accomplish the goal of providing a progress display for the user.

devenv.com provides an option which I thought might have some promise: /out. This writes the build output to a specified file. Great! All I need to do is start it writing to a file, and read through the file at the same time. I wasn’t sure of the implementation details, but it seemed feasible. Unfortunately, the devenv process locks the output file exclusively. Python’s open() was unable to read it, and even trying to find obscure parameters to win32file’s functions failed to give me the necessary access to the file.

In the UNIX world, the solution would be obvious. Create a pipe, and write the build output into the pipe while reading the pipe. In the Windows world though, a pipe is not a filesystem object. It can’t be created in a specific location, and so devenv wouldn’t be able to open it like a normal file and write to it. I considered for a while that there are a bunch of standard reserved file names, like CON and PRN. Might one of them help me? Could one of them be used to connect to a pipe? Well, no. Not really. They’re ancient history, a relic from years gone past, and they don’t have any concept of a pipe.

I started digging around for more information about named pipes, which seemed to be the prefered mechanism for IPC in Windows software. Could a named pipe be referenced through a file location? Yes, it can! \\%(host)s\pipe\%(name)s refers to the named pipe name on the host host. And as a bonus, the host . refers to the local machine at all times. Now I finally have a plan of action: Create a named pipe, make devenv write to \\.\pipe\buildOutput, and read the output on the fly.

In the end, I wrapped the named pipe code into module, NamedPipe, and the code to read devenv output on the fly was easy:

from NamedPipe import AnonymousNamedPipeReader

pipe = AnonymousNamedPipeReader()

# Application command line...
# (build application cmd line, devenv.com x.sln /build Release, etc..
# {code omitted}
cmd = cmd + r' /out \\.\pipe\%s' % pipe.name

# Okay, one of us needs to loop and accept a pipe connection, read
# data, display it to the user, and so on.
# The other of us needs to run the build command.
class ExecThread(threading.Thread):
    def __init__(self, cmdLine):
        threading.Thread.__init__(self)
        self.cmdLine = cmdLine
    def run(self):
        self.retval = os.system(self.cmdLine)
thread = ExecThread(cmd)
thread.start()

buildLog = ""
line = ""

for data in pipe:
    buildLog += data
    # {code omitted - display output on the fly}

Now, obviously this code snippet has left out all the magic. It’s a bit long and boring, so I thought maybe you’d just like a link to NamedPipe.py instead. Through the magic of functions like CreateNamedPipe and ConnectNamedPipe, you can read data being written to a file on the fly. It even works when the writer is a jerk, locking the file.

2004/08/04

Vancouver Python Workshop 2004

Filed under: programming, python — admin @ 2:51 pm

I just returned to Calgary from the Vancouver Python Workshop. Cecil and I drove out there last week (Thursday evening/Friday morning). Catsy flew out from Toronto. We all stayed at the Hyatt Regency in downtown Vancouver, where much fun was had watching the Back to The Future trilogy, and making up entertaining stories about elves.

The workshop was reasonably well organized, had a nice venue, and was well attended. The talks varied in quality from okay to excellent – all the speakers were well informed folk with interesting topics, but very few programmers are excellent public speakers. That being said, I have a few specific suggestions for the workshop itself:

  • Some talks could have benefited from the presence of a strict moderator. On a scale from polite to rude, the following things happened:

    • A speaker deferring a question to Guido. This is appropriate, as it is the prerogative of the speaker to defer during his own presentation, and Guido may be an excellent resource for an answer.
    • Guido interjecting a comment like "It’s not happening." while an attendee asks a question (referencing a PEP) is more questionable, but ultimately "polite enough" in the company of geeks.
    • Getting into an audience debate about wxWidgets during a PyObjC talk is not very polite.
    • Nitpicking the usefulness of a contrived optimization example is rude, pointless, and time consuming.
  • An entire track on the second day ended up being dedicated to Plone. I think Plone is cool, but these talks unfortunately ended up being about Plone from the point of view of a user rather than a Python developer. I believe that the workshop did not have quite as many speaker submissions as they wanted, so these Plone talks weighed in based upon how many people were willing to do them. They were good, but badly targeted and abundant.

    I would suggest that the conference organizers be willing to say "no" to speakers if they have an abundance of talks on one such subject. I realize getting submissions for talks must be a difficult process, and the fear of having not enough content at a conference must be pretty big as an organizer.

I enjoyed the workshop greatly. I should have flown out rather than drive 12 hours, but it seemed like a good idea at the time. I picked up a lot of good ideas, started in writing some interesting PyObjC code (a game of interracial life, oooh), enjoyed using SubEthaEdit in public for the first time, and ate a lot of good food. Catsy seems to have had her own kind of fun, too.

2004/05/20

Plotting in Excel through Python/COM

Filed under: programming, python — admin @ 1:41 pm

For the past couple weeks, I’ve been thinking about mathematical model development. There are lots of great tools out there to help with such tasks, like Mathcad and Mathematica. But if you’re doing software development, once you’ve built and tested a model, what you really want is code. Your Mathcad files are great for documentation, testing, and development of your model, but they can’t be embedded in your Java or C++ application.

Additionally, it’s very easy to use functions from those pieces of software which can’t be easily replicated in your software. They have very optimized methods for root solving, matrix math, symbolic derivative calculations, and other such tasks that you can’t reproduce without years worth of effort. So, when you’re developing a model that’s going to be used in software, what’s the easiest way to do it?

Python, of course. Python code tends to be very legible and terse, while still being well suited for mathematical programming. In fact, the syntax and control structures even look pretty similar between Mathcad and Python:

/wsimages/python-riddler.png

Part of a root solver implemented in Python.

/wsimages/mathcad-riddler.png

Part of a root solver implemented in Mathcad.

So, it seems natural to me to use Python to develop models. My code can easily be translated into other programming languages after the fact, and it tends to have a more proper programming structure if it’s not translated from Mathcad. Plus, translating a big chunk code from Mathcad can be a real pain in the ass to do manually – there’s just too much to miss, and it can be very time consuming to confirm it’s been done correctly.

Python is not quite a silver bullet in this case, though. Mathcad has a lot of tools for visualization which are very useful when developing a new model. Python’s wonderful abillity to interoperate makes it fairly easy to leverage plotting capabilities of an application like Microsoft Excel though. So, without any further leadup, here’s a quick process which will get you up and running with an Excel plotting Python program:

  1. Get and install Python, and the PythonWin extensions (or install ActivePython, which contains all the necessary tools and more).

  2. Run the PythonWin IDE program, and generate static COM wrappers for Microsoft Excel. This is as easy as selecting the COM Makepy Utility option from the Tools menu of PythonWin, then selecting the most recent version of the Microsoft Excel n.m Object Library available:

    /wsimages/pythonwin-comwrapper.png

    Selecting COM Makepy utility from PythonWin’s menu.

    The static COM wrappers must be used in order to access the Excel constants (of which there are hundreds).

  3. Import the COM Dispatch function and the constants namespace into your application:

    
    from win32com.client import Dispatch, constants
    
  4. At this point, all that’s left is to go to town on automating Excel. There is a lot of documentation that comes along with Excel, as well as a big chunk of MSDN content that will also help. Let’s dive in, and I’ll just throw an XY scatter plot at you:

    def plot(x, y, xAxisLog=False, yAxisLog=False):
        # acquire application object, which may start application
        application = Dispatch("Excel.Application")
    
        # create new file ('Workbook' in Excel-vocabulary)
        workbook = application.Workbooks.Add()
    
        # store default worksheet object so we can delete it later
        defaultWorksheet = workbook.Worksheets(1)
    
        # build new chart (on seperate page in workbook)
        chart = workbook.Charts.Add()
        chart.ChartType = constants.xlXYScatter
        chart.Name = "Plot"
    
        # create data worksheet
        worksheet = workbook.Worksheets.Add()
        worksheet.Name = "Plot data"
    
        # install data
        xColumn = addDataColumn(worksheet, 0, x)
        yColumn = addDataColumn(worksheet, 1, y)
    
        # create series for chart
        series = chart.SeriesCollection().NewSeries()
        series.XValues = xColumn
        series.Values = yColumn
        series.Name = "Data"
        series.MarkerSize = 3
    
        # setup axises
        xAxis = chart.Axes()[0]
        yAxis = chart.Axes()[1]
        xAxis.HasMajorGridlines = True
        yAxis.HasMajorGridlines = True
        if xAxisLog:
            xAxis.ScaleType = constants.xlLogarithmic
        if yAxisLog:
            yAxis.ScaleType = constants.xlLogarithmic
    
        # remove default worksheet
        defaultWorksheet.Delete()
    
        # make stuff visible now.
        chart.Activate()
        application.Visible = True
    
    def genExcelName(row, col):
        """Translate (0,0) into "A1"."""
        if col < 26:
            colName = chr(col + ord('A'))
        else:
            colName = chr((col / 26)-1 + ord('A')) +\
                chr((col % 26) + ord('A'))
        return "%s%s" % (colName, row + 1)
    
    def addDataColumn(worksheet, columnIdx, data):
        range = worksheet.Range("%s:%s" % (
            genExcelName(0, columnIdx),
            genExcelName(len(data) - 1, columnIdx),
            ))
        for idx, cell in enumerate(range):
            cell.Value = data[idx]
        return range
    

    I suppose that at least an explanation of what this is doing would be a good thing. This program will cause Excel to be run, and a new file to be created. The new file will contain one worksheet with the contents of the x/y arrays in columns A and B. It will also contain one sheet-location chart, an XY scatter, with the x/y points plotted. The x and y arguments to plot are sequences of numbers, and the two flags indicate whether the X and Y axis should be logarithmic or linear.

  5. Some examples of using plot:

    # A simple example:
    plot( (1,2,3,4,5), (6,7,8,9,10) )
    
    # Some more data:
    x, y = [], []
    for i in range(100):
        x.append(i)
        y.append(i ** 2)
    plot(x, y)
    
    # Using log axises:
    plot(x, y, True, True)
    
  6. Remember, there are good tools available if you’re not in Windows too. gnuplot and gnuplot-py provide a pretty nice graphing environment which is probably quite a bit more capable than Excel.

2004/04/08

The MutableNumber Class Revisited

Filed under: programming, python — admin @ 5:04 pm

Ian Bicking’s comment suggesting I wasn’t far away from getting the MutableNumber class to work was quite correct. I had never considered using a descriptor, which made it difficult to make this work with a metaclass. The following code, though, does work:

class Operator(object):
    def __init__(self, instance, op):
        self.instance = instance
        self.op = op
    def __call__(self, rhs):
        try:
            return self.op(self.instance.number, rhs.number)
        except AttributeError:
            return self.op(self.instance.number, rhs)

class OperatorDescriptor(object):
    def __init__(self, op):
        self.op = op
    def __get__(self, obj, type=None):
        return Operator(obj, self.op)
    def __set__(self, obj, value):
        raise TypeError
    def __delete__(self, obj):
        raise TypeError

class NumberDelegateMetaclass(type):
    def __init__(cls, name, bases, dct):
        for attr in cls.__operatormap__:
            setattr(cls, attr, OperatorDescriptor(getattr(operator, attr)))

class MutableNumber(object):
    number = None

    __metaclass__ = NumberDelegateMetaclass
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    def __init__(self, value=None):
        self.number = value

It’s a miracle.

Overriding Operators in New-Style Classes

Filed under: programming, python — admin @ 3:38 pm

Occasionally, even Python violates the principal of ‘least surprise’ – that is, software should do what you expect it to do. Overriding mathematics operators in new-style classes led me to find one such violation, and investigate the reason behind it.

I setup myself a make-work project a few days ago when I probably should have done a dozen other things. After reading a posting to python-dev entitled Python is faster than C, I was motivated to create a pure-python implementation of a matrix class. I was interested in building it primarily as a refresher of matrix mathematics, and also to experiment with psyco’s x86 optimization capabilities.

I ran into a quirk of new-style classes. I envisioned the Matrix class having two hidden data fields, one to hold a tuple of Vector objects representing each row, and a second one to hold a tuple of Vector objects representing each column. This would allow for easy access to the data based upon rows or columns, whichever was more convenient, with no speed penalty. Each tuple of Vectors ought to point to the same numbers, though. How does one do this with Python?

I’d like self.__cdata[0][0] and self.__rdata[0][0] to always be the same number, even if I only change one. I could manage this with accessor methods that make sure they’re the same, but I decided to wrap each number in an object called a MutableNumber. The class was setup like so:

class MutableNumber(object):
    def __init__(self, value = None):
        self.number = value

Every mathematical operator should have then been overridden like this:

...
    def __mul__(self, rhs):
        try:
            return MutableNumber(self.number * rhs.number)
        except AttributeError:
            return MutableNumber(self.number * rhs)
...

What a pain to type and maintain, though. What if I wanted to add another possibility in the future, or change the operator to not return a new MutableNumber object wrapping the number? There are twenty-two operators I wanted to override (eg. __mul__, __eq__, __div__, __floordiv__…), and I didn’t want to have to type them, even once.

I originally setup a meta-class to delegate every access to a property in the class’s __delegateattributes__ member to self.number. This had some issues:

>>> MutableNumber(2) * 3
6
>>> MutableNumber(2) * MutableNumber(3)
Traceback (most recent call last):
  ...
TypeError: unsupported operand types for *: 'int' and 'MutableNumber'

I had to go a more complex route, rather than just pure delegation. I eventually replaced all the operators with custom instances of an Operator class, which held an operator and a reference to the instance of the class:

class MutableNumber(object):
    __operatormap__ = "__add__", "__div__", "__eq__", "__floordiv__", "__ge__", \
        "__gt__", "__inv__", "__invert__", "__le__", "__lshift__", "__lt__", \
        "__mod__", "__mul__", "__ne__", "__neg__", "__pos__", "__pow__", \
        "__rshift__", "__sub__", "__truediv__"
    class Operator(object):
        def __init__(self, instance, op):
            self.instance = instance
            self.op = op
        def __call__(self, rhs):
            op = getattr(operator, self.op)
            try:
                return op(self.instance.number, rhs.number)
            except AttributeError:
                return op(self.instance.number, rhs)

    def __init__(self, value=None):
        for op in self.__operatormap__:
            setattr(self, op, MutableNumber.Operator(self, op))
        self.number = value

So, we finally reach the part where it gets interesting. What happens? Does this work properly? The code seems good, but it fails to function:

>>> MutableNumber(3) * 2
Traceback (most recent call last):
  ...
TypeError: unsupported operand type(s) for *: 'MutableNumber' and 'int'

Unsupported operand for types? It looks like it’s totally ignoring the fact that the MutableNumber instance has a __mul__ property set. Hm… stranger still:

>>> MutableNumber(3).__mul__(2)
6
>>> MutableNumber(3).__class__.__mul__(2)
Traceback (most recent call last):
  ...
AttributeError: type object 'MutableNumber' has no attribute '__mul__'

Well, it seems that A * B does not call A.__mul__(B). It searches for A.__class__.__mul__, but not A.__mul__. It works fine with an old-style class, but not a new-style class.

It turns out this behaviour has been discovered before. In fact, Jan Decaluwe was trying to do the same thing I am, create an instance object that acts like a mutable number. Alex Martelli wrote:

The idea is that operators should rely on special
methods defined by the TYPE (class) of the object they’re working
on, NOT on special methods defined just by the OBJECT itself and
not by its class. Doing otherwise (as old-style classes did, and
still do for compatibility) is untenable in the general case, for
example whenever you consider a class and its metaclass (given that
the metaclass IS the class object’s type, no more and no less).

Well, too bad. I guess maybe I’ll just backtrack all the way to using accessor methods to make sure the column and row data is both kept constant… or just store one of them.

2004/03/30

Embedding Python Tips

Filed under: moo, programming, python — admin @ 2:46 am

Python is a beautiful programming language. One of it’s most wonderful features is a very clean and simple C API that allows Python to be extended with dynamically loadable C modules. That same C API also allows Python to be embedded in other pieces of software. This means that any program can allow the user to enter Python code interactively (or otherwise) to affect the program in whatever way they wish. This is a powerful capability, but using occasionally requires a few tricks to accomplish the embedder’s goals.

Today’s embedding exercise was allowing a MOO server to execute arbitrary Python code:

;py_runfunction({"import math", "return math.sqrt(2*2*2)"})
> 2.8284271247461903

Of course, a MOO server can already do square roots… that wasn’t the point. There was no point. Anyways, here are a few ideas that might help other people embed Python in a useful way.

Evaluating Statements

One of the first things most people try to do is evaluate an arbitrary statement and get its return value. This is not quite as easy as it sounds. Although Python’s eval builtin does this, it may be more limited than the embedding programmer desires. eval will only permit an expression to be evaluated, not a statement:

>>> eval("x = 2")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 1
    x = 2
      ^
SyntaxError: invalid syntax

I suggest that if you want the user to be able to evaluate an arbitrary block of code, wrap an artificial function around it and call the function itself:

def f():
    import math
    class Cylinder:
        def _calcVolume(self):
            return math.pi * \
                self.radius**2 * \
                self.height
        volume = property(_calcVolume)
    c = Cylinder()
    c.radius = 12.2
    c.height = 16.12
    return c.volume

This allows the user to input much more complex functions, like the above example which uses a class and an import statement. All that needed to be artificially added was the ‘def f():’ and an arbitrary but constant amount of whitespace in front of each line of code.

Compiling Code without a Module

So you’ve gotten some code from a user, and you want to compile it. Maybe you’re creating a function to wrap around the user’s code. Where does that function belong? Where do you evaluate your code?

The first instinct I had was to use PyImport_AddModule to get the __main__ module and start importing functions into its module dictionary. I had a block of code similar to this (error checking omitted):

Py_Initialize();
PyObject* module = PyImport_AddModule("__main__");
PyObject* moduleDict = PyModule_GetDict(module);
PyObject* compileRetval = PyRun_String(code, Py_file_input,
    moduleDict, moduleDict);
...
Py_Finalize();

This then allowed me to call functions on the module object and get some code back. The only real downside was the initialize and finalize around my code. I didn’t want code from one compile to mess with another, and since I was using the __main__ module, this caused problems. Eventually I decided to use random strings as the names for my modules so that I could use them all independently, but that sure was ugly.

The solution I stumbled upon was caused by my accidently deleting some lines of code. I eventually realized that I didn’t need the module object at all. I could create a new, empty dictionary, and compile the code ‘into’ that:

PyObject* dict = PyDict_New();
PyObject* compileRetval = PyRun_String(code, Py_file_input,
    dict, dict);

Everything continued to work as before, except now I had to PyDict_GetItem out of dict and use PyObject_CallObject rather than the PyObject_CallMethod that I could have used before. But nothing crashed, the world continued to run, and I no longer needed to initialize and finalize around my evaluation. Yay!

Settings __builtins__

There was one minor problem. Some functionality like builtin functions and classes (like Exception) was missing. Oops:

// Check for __builtins__...
if (PyDict_GetItemString(dict, "__builtins__") == NULL)
{
    // Hm... no __builtins__ eh?
    PyObject* builtinMod = PyImport_ImportModule("__builtin__");
    if (builtinMod == NULL ||
        PyDict_SetItemString(dict, "__builtins__", builtinMod) != 0)
    {
        Py_DECREF(dict);
        Py_XDECREF(dict);
        // error handling
        return;
    }
    Py_DECREF(builtinMod);
}

Hey, that fixed that right up.

I had this problem when I was using random names for modules, as well. It seems PyImport_AddModule does not set __builtins__ on a new module — but it is set up on __main__ always.

Getting Tracebacks using the traceback Module

What happened when things went wrong? Well, at first, a lot of crashing. And things were going wrong a lot, especially when I was trying to use modules that didn’t exist in the system. Heh heh.

Thankfully, Python will setup tracebacks that are useful even when you’re using the C API and screwing things up from the inside. How on earth do you get at those tracebacks, though? You can get a lot of information from the PyErr_* class of functions, but not a properly formatted Python traceback to display to the user. Eventually, I ended up using the traceback module itself to display an error:

char* getPythonTraceback()
{
    // Python equivilant:
    // import traceback, sys
    // return "".join(traceback.format_exception(sys.exc_type,
    //    sys.exc_value, sys.exc_traceback))

    PyObject *type, *value, *traceback;
    PyObject *tracebackModule;
    char *chrRetval;

    PyErr_Fetch(&type, &value, &traceback);

    tracebackModule = PyImport_ImportModule("traceback");
    if (tracebackModule != NULL)
    {
        PyObject *tbList, *emptyString, *strRetval;

        tbList = PyObject_CallMethod(
            tracebackModule,
            "format_exception",
            "OOO",
            type,
            value == NULL ? Py_None : value,
            traceback == NULL ? Py_None : traceback);

        emptyString = PyString_FromString("");
        strRetval = PyObject_CallMethod(emptyString, "join",
            "O", tbList);

        chrRetval = strdup(PyString_AsString(strRetval));

        Py_DECREF(tbList);
        Py_DECREF(emptyString);
        Py_DECREF(strRetval);
        Py_DECREF(tracebackModule);
    }
    else
    {
        chrRetval = strdup("Unable to import traceback module.");
    }

    Py_DECREF(type);
    Py_XDECREF(value);
    Py_XDECREF(traceback);

    return chrRetval;
}

Of course, when one can’t import the traceback module, one can’t generate a traceback explaining why not. :-)

2004/03/12

Distributing a Python Embedding Program

Filed under: programming, python — admin @ 8:39 pm

I got a bit bored at my job yesterday. This happens often, and usually a new random piece of software emerges as a result. This time I ended up hacking on a piece of software that had previously emerged from boredom, which I called the "Difference Machine". I added support for plot datasets generated on the fly through Python code, and in the end learned how to distribute Python within my application.

The purpose of the Difference Machine is very simple: I dislike MS Excel, and wish to never have to use it again. As someone who works in a job involving a lot of engineering, Excel is pretty much standard fare for comparing datasets and stuff of that nature, though. The Difference Machine allows plots generated by the software we develop to be imported, and compared against each other. Additionally, text import is possible from a CSV file or from the clipboard.

differencemachine-plot.png

The Difference Machine comparing two gas supercompressibility (z-factor) correlations.

Adding in the Python embedding was easy. I’ve done similar things before, and the Python C-API is pretty straight-forward (although necessarily verbose). When it came time to put this application up on the network for co-workers to access, the real fun began.

differencemachine-edit.png

The Difference Machine editing Python code with a crude edit control.

My first attempt was simple. Copy the application onto the network, and include python23.dll in the application’s directory. Behold! It actually worked. But, there were a few minor glitches…

  • No access to the standard library. import sys and import math worked, but not import csv or anything else.
  • No standard library meant no traceback library, which I had depended upon to format tracebacks when things went poorly. The software did not react to the missing library well.

I needed the standard library. I tried copying it up into a Lib directory in my application’s directory, and this worked okay… except that it was a huge number of files, and it was still missing things like _sre.pyd, the dynamically linked libraries.

I consulted PEP-237, Import Modules from Zip Archives, to try to create a zip file of the entire standard library that would be much easier to manage than my Lib directory. By naming the zip file python23.zip and putting it in the same directory as my python23.dll, it should have been able to access the library. But it still failed because it couldn’t access any dynamically linked libraries, and zlib is necessary to open the zip file.

By putting all the .pyd files into my application’s directory, I finally got it to work. It loads zlib to read python23.zip, where it retrieves the rest of the standard library. Yippee! In the end, the directory contains the following files:

DifferenceMachine.exe
... bunch of my dlls ...
python23.dll
python23.zip
zlib.pyd
_csv.pyd
_sre.pyd
... bunch more pyd files ...

And all is good and happy. I’m one small step closer to being Excel-less.

2004/02/05

Open Scripting Architecture and iCal

Filed under: programming, python — admin @ 2:22 pm

I was browsing the Daily Python-URL this morning, when I came across a link to a Python module for reading OSX’s iCal files. The module’s web page states:

Apple doesn't document the API for interfacing with iCal, but it does save
it's files as industry standard iCalendar files.  I went looking recently
for a python module to interface with iCal, and couldn't find one.

It’s great to have a good Python module to read standard iCalendar files, but the statement that Apple doesn’t have a documented API for interfacing with iCal is bogus. iCal supports the Open Scripting Architecture (AppleScript’s APIs), the same as every other Apple application does. The Open Scripting Architecture can be used from Python much easier than one can read an iCalendar file, and the information is exposed in a very easy to access format. I wrote an earlier article on how to do similar actions with iTunes, so in my continuing effort to educate the world on how to use OSA from Python, here’s some more!

The first step in using iCal from a Python script through OSA is to generate an iCal module which wraps up the low-level OSA details. The gensuitemodule.py script is designed to do just that. It is located in your plat-mac directory inside your Python installation’s lib directory. On a stock installation of Panther, that makes it in /System/Library/Frameworks/Python.framework/ pant,pant Versions/Current/lib/python2.3/plat-mac/gensuitemodule.py. In order to interface with the system window manager and access OSA, it must be run with the pythonw executable, rather than python. The --output command line parameter can be used to specify a name for the output module. In the end, generating the module is extremely simple:

pythonw {...}/gensuitemodule.py --output iCal /Applications/iCal.app

Now that you have a beautiful iCal module, what do you do with it? The iCal.iCal class is the basic application, which is used for all OSA communications with the iCal application itself. So, to start things off, an instance of that class should be created so we can do some communicating:

import iCal
app = iCal.iCal()

How do we determine what we can do to this application through OSA? The easiest way is to use OSX’s Script Editor, which allows us to open up an application’s OSA dictionary and view the contents. The Script Editor is located in the /Applications/AppleScript folder, and has an option in the File menu to Open Dictionary....

OS X's script editor

Once you have iCal’s dictionary open, inside the iCal suite you can see all the classes and commands that iCal exposes. For example, the application class has elements like calendar, window, and document. Since we’re interested in getting data out, we know that calendar is where we want to start. An element is a collection of objects, implying that each application has a collection of calendars. Based upon what we know of how iCal works, this sounds very reasonable. Here’s how we can iterate over all the calendars that our iCal has:

numCalendars = app.count(app, each=iCal.calendar)
for i in range(1, numCalendars + 1):
    cal = iCal.calendar(i)

The first line sends an OSA command to iCal asking it to count the number of iCal.calendar objects it currently has. Since Python normally uses zero-based counting, and OSA uses one-based counting, we need to iterate from 1 to numCalendars + 1. We create an iCal.calendar object from each of those indicies. Creating the object doesn’t do any communication with iCal, it just creates an object which can dispatch calls to the appropriate instance of the calendar class.

What if we wanted to get some data out of the object, now? The Script Editor shows that the calendar objects support a number of properties like tint, title, and description. Let’s print out the title of every calender:

print &quot;Calendar %s - %s&quot; % (i, app.get(cal.title))

The magic here is in the app.get(cal.title). This tells the application to retrieve the title property of the cal instance, which is already bound to the proper calendar object in iCal. We simply print it out, which outputs something like:

Calendar 1 - Home
Calendar 2 - Work

Let’s put these chunks of code together, and include the ability to print out every event for each calendar:

import iCal

def printAllEvents(app, calendar):
    numEvents = app.count(calendar, each=iCal.event)
    for i in range(1, numEvents + 1):
        event = calendar.event(i)
        print "\tEvent %s - %s" % (i, app.get(event.summary))

def printAllCalendars(app, eventsToo = True):
    numCalendars = app.count(app, each=iCal.calendar)
    for i in range(1, numCalendars + 1):
        calendar = iCal.calendar(i)
        print "Calendar %s - %s" % (i, app.get(calendar.title))
        if eventsToo:
            printAllEvents(app, calendar)

app = iCal.iCal()
printAllCalendars(app)

Yay! We have access to every piece of information iCal stores, and we didn’t need to parse an iCalendar file. Some parts of iCal even begin to integrate with the OSX Address Book, and we can access that information as well.

One last bit of code could be useful, though. What if we wanted to change some of the data in iCal? It’s fairly simple, but not too obvious or well documented anywhere. Here’s a short snippet that will give all your calendars dumb, pointless names:

import iCal
app = iCal.iCal()

numCalendars = app.count(app, each=iCal.calendar)
for i in range(1, numCalendars + 1):
    calendar = iCal.calendar(i)
    app.set(calendar.title, to = "Calendar %s" % i)
« Newer PostsOlder Posts »

Powered by WordPress