Posts Tagged ‘moo’

Embedding Python Tips

Python is a beautiful programming language. One of it’s most wonderful features is a very clean and simple C API that allows Python to be extended with dynamically loadable C modules. That same C API also allows Python to be embedded in other pieces of software. This means that any program can allow the user to enter Python code interactively (or otherwise) to affect the program in whatever way they wish. This is a powerful capability, but using occasionally requires a few tricks to accomplish the embedder’s goals.

Today’s embedding exercise was allowing a MOO server to execute arbitrary Python code:

;py_runfunction({"import math", "return math.sqrt(2*2*2)"})
> 2.8284271247461903

Of course, a MOO server can already do square roots… that wasn’t the point. There was no point. Anyways, here are a few ideas that might help other people embed Python in a useful way.

Evaluating Statements

One of the first things most people try to do is evaluate an arbitrary statement and get its return value. This is not quite as easy as it sounds. Although Python’s eval builtin does this, it may be more limited than the embedding programmer desires. eval will only permit an expression to be evaluated, not a statement:

>>> eval("x = 2")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 1
    x = 2
      ^
SyntaxError: invalid syntax

I suggest that if you want the user to be able to evaluate an arbitrary block of code, wrap an artificial function around it and call the function itself:


def f():
    import math
    class Cylinder:
        def _calcVolume(self):
            return math.pi * \
                self.radius**2 * \
                self.height
        volume = property(_calcVolume)
    c = Cylinder()
    c.radius = 12.2
    c.height = 16.12
    return c.volume

This allows the user to input much more complex functions, like the above example which uses a class and an import statement. All that needed to be artificially added was the ‘def f():’ and an arbitrary but constant amount of whitespace in front of each line of code.

Compiling Code without a Module

So you’ve gotten some code from a user, and you want to compile it. Maybe you’re creating a function to wrap around the user’s code. Where does that function belong? Where do you evaluate your code?

The first instinct I had was to use PyImport_AddModule to get the __main__ module and start importing functions into its module dictionary. I had a block of code similar to this (error checking omitted):


Py_Initialize();
PyObject* module = PyImport_AddModule("__main__");
PyObject* moduleDict = PyModule_GetDict(module);
PyObject* compileRetval = PyRun_String(code, Py_file_input,
    moduleDict, moduleDict);
...
Py_Finalize();

This then allowed me to call functions on the module object and get some code back. The only real downside was the initialize and finalize around my code. I didn’t want code from one compile to mess with another, and since I was using the __main__ module, this caused problems. Eventually I decided to use random strings as the names for my modules so that I could use them all independently, but that sure was ugly.

The solution I stumbled upon was caused by my accidently deleting some lines of code. I eventually realized that I didn’t need the module object at all. I could create a new, empty dictionary, and compile the code ‘into’ that:


PyObject* dict = PyDict_New();
PyObject* compileRetval = PyRun_String(code, Py_file_input,
    dict, dict);

Everything continued to work as before, except now I had to PyDict_GetItem out of dict and use PyObject_CallObject rather than the PyObject_CallMethod that I could have used before. But nothing crashed, the world continued to run, and I no longer needed to initialize and finalize around my evaluation. Yay!

Settings __builtins__

There was one minor problem. Some functionality like builtin functions and classes (like Exception) was missing. Oops:


// Check for __builtins__...
if (PyDict_GetItemString(dict, "__builtins__") == NULL)
{
    // Hm... no __builtins__ eh?
    PyObject* builtinMod = PyImport_ImportModule("__builtin__");
    if (builtinMod == NULL ||
        PyDict_SetItemString(dict, "__builtins__", builtinMod) != 0)
    {
        Py_DECREF(dict);
        Py_XDECREF(dict);
        // error handling
        return;
    }
    Py_DECREF(builtinMod);
}

Hey, that fixed that right up.

I had this problem when I was using random names for modules, as well. It seems PyImport_AddModule does not set __builtins__ on a new module — but it is set up on __main__ always.

Getting Tracebacks using the traceback Module

What happened when things went wrong? Well, at first, a lot of crashing. And things were going wrong a lot, especially when I was trying to use modules that didn’t exist in the system. Heh heh.

Thankfully, Python will setup tracebacks that are useful even when you’re using the C API and screwing things up from the inside. How on earth do you get at those tracebacks, though? You can get a lot of information from the PyErr_* class of functions, but not a properly formatted Python traceback to display to the user. Eventually, I ended up using the traceback module itself to display an error:


char* getPythonTraceback()
{
    // Python equivilant:
    // import traceback, sys
    // return "".join(traceback.format_exception(sys.exc_type,
    //    sys.exc_value, sys.exc_traceback))

    PyObject *type, *value, *traceback;
    PyObject *tracebackModule;
    char *chrRetval;

    PyErr_Fetch(&type, &value, &traceback);

    tracebackModule = PyImport_ImportModule("traceback");
    if (tracebackModule != NULL)
    {
        PyObject *tbList, *emptyString, *strRetval;

        tbList = PyObject_CallMethod(
            tracebackModule,
            "format_exception",
            "OOO",
            type,
            value == NULL ? Py_None : value,
            traceback == NULL ? Py_None : traceback);

        emptyString = PyString_FromString("");
        strRetval = PyObject_CallMethod(emptyString, "join",
            "O", tbList);

        chrRetval = strdup(PyString_AsString(strRetval));

        Py_DECREF(tbList);
        Py_DECREF(emptyString);
        Py_DECREF(strRetval);
        Py_DECREF(tracebackModule);
    }
    else
    {
        chrRetval = strdup("Unable to import traceback module.");
    }

    Py_DECREF(type);
    Py_XDECREF(value);
    Py_XDECREF(traceback);

    return chrRetval;
}

Of course, when one can’t import the traceback module, one can’t generate a traceback explaining why not. :-)

XPCOM Enabled MOO

On April 29th, 2002, I began to look at integrating the standalone XPCOM with the LambdaMOO server.

XPCOM is a lightweight cross-platform implementation of the concepts behind Microsoft’s COM. It has been implemented in the process of the development of Mozilla. XPCOM is essentially COM, but cross-platform, hence the XP.

The integration inside the MOO server is not necessarily for any specific purpose, but I have a few plans in mind for what this integration will permit:

  • XPCOM can be used to allow builtin functions to be compiled into shared modules and dropped into the MOO ‘components’. These builtin functions will be registered with the ‘regxpcom’ program, and be accessible from inside the MOO instantly. There are multiple advantages over doing this rather than the traditional way of modifying the MOO server to add builtin functions. The most major advantage is that XPCOM components can be written in languages other than C. It’s useful to program in an object-oriented language like C++ when you’re programming a builtin function that will be used in an object-oriented environment in the first place. Additionally, plug-in DLLs can be isolated in such a way that the crash of a builtin function will not affect the running of the MOO server.

    This has been implemented and is currently undergoing testing. The implementation involved the creation of an ‘IBuiltinFunction’ interface. The plug-in DLL exports an IBuiltinFunction derived interface. The MOO server allows access to its data to the XPCOM component via a set of ‘IMOOData’-derived interfaces.

    structure diagram of xpcom interfaces in the moo

  • Another possible use for XPCOM inside the MOO is to allow MOO-code to access XPCOM objects. I expect that originally this will be implemented by creating an XPCOM plug-in DLL that exports builtin functions providing xpcom_createinstance(), xpcom_queryinterface(), xpcom_callfunction(), xpcom_get_attribute(), xpcom_set_attribute(), xpcom_releaseinstance(), and xpcom_isvalidinstance().

    xpcom_createinstance() will create an instance of a given contract id, if possible, and return a hash string that represents that instance. Eg:

    ;xpcom_createinstance("@mozilla.org/file/directory_service;1")
    => "##XPCOM#bbf8cab0-d43a-11d3-8cc2-00609792278c#59e7e77a##"

    The string representing that interface is a concationation of the CID, and a pointer address to it to make it unique for each instance. A hash table of these strings will be stored in the server so that implementation of the other xpcom_*() functions is relatively efficient.

    Implementation of this idea is pending further thought.

  • The MOOzilla MOO client is written in JavaScript using Mozilla, and utilizes XPCOM heavily. It should be possible to create a plug-in architecture for MOOzilla, using some kind of common interfaces that the MOO server is aware of. Using an implementation of SOAP, (implemented as an XPCOM module, of course), it should be possible for the MOO server to ‘script’ MOOzilla, or vice-versa, using out-of-band binary SOAP communication streams. Woo! The purpose? … Well, why not create an XPCOM module that wraps Gtk+, and then an interface for it on the client side and an interface for message handling on the server side could allow a Gtk+ application to be MOO-server-driven. Of course, this doesn’t apply just to Gtk+, it could be used for any toolkit… theoretically.

My efforts of XPCOM integration have yield a MOO which can successfully use XPCOM component builtin functions. My work has been based upon the MOO Canada server source, which is a rather hacked up LambdaMOO 1.8.1. The source has some unfortunate usability problems. It is basically customised just for my building and use of it. It should be possible to adapt it to build on your machine. The Makefile-xpcom-lao file is my Makefile. The automake and autoconf sources are not up to date. With some modifications to the Makefile, it should be possible to get it to build for you. Currently, I don’t have any example modules that are fit for publication.

The XPCOM enabled MOO server source can also be accessed by Subversion clients, or browsed with a web browser, at http://stompstompstomp.com/svn/moo/mc/xpcom-enabled-branch.

Unicode, MOO, and MOOzilla

Spent the morning talking with Jason of Achieve MOO. He came over while I was at Catsy’s in Toronto. The original goal was to discuss things related to the Unicode based MOO, and we did that, as well as touching on other topics of mutual interest.

I’ll be publishing a new unicode patch shortly which allows non-us-ascii keywords and variable names in the MOO. This means that non-english words and characters can be used as variable, verb, property, and function names in MOO code without any unnecessary escaping. Collaborating with Jason showed that there are apparently duplicates of characters in the unicode codepage, such as the semicolon, period, quotation marks, and so on, which must be recognized by the MOO in the future as having the same functionality as the us-ascii versions of these characters. This should be interesting, and I’ll look into it soon for another unicode patch.

Once that was out of the way, it became apparent that the server handles Unicode great, the only problems left were in front end programs. Jason’s native OSX MUD client was able to send text to the MOO, but could not receive it and display it properly. The ‘telnet’ command in an OSX terminal was able to receive but not send, exactly the opposite. MOOzilla was able to do both. We didn’t have the resources available to easily test whether Mooca, the Achieve Java telnet client, could do it correctly and nicely.

Jason was interested in MOOzilla as a client itself. A number of ideas and problems came out of that discussion.

The idea of somehow making seperable windows out of MOOzilla which are MOO driven came up. This has been thought of before, but the way it was discussed here was as windows that could be separated, or could be left integrated in MOOzilla. This would allow a ‘power-user’ to keep everything in one complex window, while a beginning user would have a simpler interface that they could complicate if they chose to. I need to give some thought as to how this might be implemented… but I know that MOOzilla/Mozilla have the capabilities to do it.

A problem that arose is MOOzilla distribution and access. A Java client is easy to access because it requires no software installation, is just right there, and is cross-platform. MOOzilla does require software installation, two pieces of software actually, and the software varies from platform to platform. This would be streamlined a bit by the proposed ‘MOOzilla Package’ which was just MOOzilla, without Mozilla cruft, but would still not be usable in Jason’s teaching environment. So… what would? Hm… if only IE supported XUL/XPCOM. Now, if someone could make a plugin for IE that would allow the running of an XUL application… it would probably be a huge plugin. Could one be made that was MOOzilla-specific? … who knows how to make an IE plugin? Need some magic glue components here…

DOM2 in MOO

I’ve taken a big break in MOOzilla documentation writing to implement a DOM2 interface on MOO Canada. Wheeha! It took me hours of planning last night to come up with a way to do it that would not involve a server hack to allow circular waif references.

When completed, MC will have a nice clean DOM2 implementation that is actually reference counted, not garbage collected, though a series of magic levers and spells. It’s very slick. Basically, references are only kept going up the tree, and copies of WAIFs as text are kept going down the tree. It works surprisingly well.

MOOzilla 0.9.9

Spent most of my day working on the MOOZilla help documentation once again. It’s amazing how much time one can pour into writing and never get very far. Technical documentation takes a lot of work… I spend my time trying to imagine, ‘If I wanted help on the ‘Connect’ menu option… what would I want to know? It’s a pretty simple option… but if I looked in the help for it, what am I looking for? ‘.

The surprising thing is the massive amount of groth that MOOzilla will go through for it’s next release. This documentation will eventually include colourful images of MOOzilla being used in various scary ways by people. Currently I’m writing just the section on how to use MOOzilla as a MOO/MUD client, and I already have bumped the MOOzilla size up to about 290k, from its previous 75k. The documentation for how to build with MOOzilla will be very pictureful to be as useful as possible… or at least, I guess I’d like it to be.

Maybe I could make MOOzilla mock-ups using the same HTML rendering that MOOzilla uses itself. This oughta be smaller than using pictures of the rendered text…

… something about that makes me feel uncomfortable. Well, I’ll try it out when I get to the building section of the documentation.

My current feature list for 0.9.9:

  • Documentation (1-2 weeks)
  • Stand-Alone MOOzilla distributions for Windows and Linux. No Mozilla installation required, just gecko and other technologies bundled together. (1-2 weeks)
  • Monospaced input windows, single + multi line. (2-3 days, if possible at all w/ XUL+XBL…)
  • Save log as web page (1 hour)

The biggest problem with this plan for 0.9.9 is that the stand-alone distributions will be for platforms I don’t have access to. Teehee… this means MOOzilla 0.9.9 is probably planned for release around Christmas, after I do have access to an x86 computer to do some development with.

1 2