Mathieu Fenniak's Weblog

How can you capture the console output of a program, when it buffers that output if you’re not using a console to view it? This was a problem run into when building an automation tool for MS Visual Studio .NET. In the end, the programmer subjugated his tool (as it should be) by beating it over the head with a pipe.

A few of us programmers with in the unfortunately unfriendly environment of MS Windows. It might look pretty and have lots of applications written for it, but it’s basically an unfriendly environment for a software developer. Even MS Visual Studio .NET can be unfriendly to a developer, which is unfortunate since it’s the one program you’d expect would be really friendly.

Visual Studio allows you to provide command-line options which start a software build. Running inside a command prompt, all you need to do is pass a solution file and a build configuration to the program, and you’re off. In fact, Visual Studio even gives you more command-line flexibility by providing two executables, devenv.com and devenv.exe – the former will tend towards printing console output all the time, while the latter will avoid it if a build log file is provided instead.

In the creation of a complete build tool, I wanted to run devenv.com and capture the output so I could display the progress to a user. That’s when it became tough. Running the executable through os.popen (or any other popen function) didn’t accomplish what I wanted – the output being printed to the console (and now being read through a pipe) was buffered inside the devenv process and only printed after the build was completed. Clearly this didn’t accomplish the goal of providing a progress display for the user.

devenv.com provides an option which I thought might have some promise: /out. This writes the build output to a specified file. Great! All I need to do is start it writing to a file, and read through the file at the same time. I wasn’t sure of the implementation details, but it seemed feasible. Unfortunately, the devenv process locks the output file exclusively. Python’s open() was unable to read it, and even trying to find obscure parameters to win32file’s functions failed to give me the necessary access to the file.

In the UNIX world, the solution would be obvious. Create a pipe, and write the build output into the pipe while reading the pipe. In the Windows world though, a pipe is not a filesystem object. It can’t be created in a specific location, and so devenv wouldn’t be able to open it like a normal file and write to it. I considered for a while that there are a bunch of standard reserved file names, like CON and PRN. Might one of them help me? Could one of them be used to connect to a pipe? Well, no. Not really. They’re ancient history, a relic from years gone past, and they don’t have any concept of a pipe.

I started digging around for more information about named pipes, which seemed to be the prefered mechanism for IPC in Windows software. Could a named pipe be referenced through a file location? Yes, it can! \\%(host)s\pipe\%(name)s refers to the named pipe name on the host host. And as a bonus, the host . refers to the local machine at all times. Now I finally have a plan of action: Create a named pipe, make devenv write to \\.\pipe\buildOutput, and read the output on the fly.

In the end, I wrapped the named pipe code into module, NamedPipe, and the code to read devenv output on the fly was easy:

from NamedPipe import AnonymousNamedPipeReader

pipe = AnonymousNamedPipeReader()

# Application command line...
# (build application cmd line, devenv.com x.sln /build Release, etc..
# {code omitted}
cmd = cmd + r' /out \\.\pipe\%s' % pipe.name

# Okay, one of us needs to loop and accept a pipe connection, read
# data, display it to the user, and so on.
# The other of us needs to run the build command.
class ExecThread(threading.Thread):
    def __init__(self, cmdLine):
        threading.Thread.__init__(self)
        self.cmdLine = cmdLine
    def run(self):
        self.retval = os.system(self.cmdLine)
thread = ExecThread(cmd)
thread.start()

buildLog = ""
line = ""

for data in pipe:
    buildLog += data
    # {code omitted - display output on the fly}

Now, obviously this code snippet has left out all the magic. It’s a bit long and boring, so I thought maybe you’d just like a link to NamedPipe.py instead. Through the magic of functions like CreateNamedPipe and ConnectNamedPipe, you can read data being written to a file on the fly. It even works when the writer is a jerk, locking the file.