Combining stdout, stderr, and pipes on Windows

This post is as much for my own reference as anyone else's. I need to capture the output of a program that outputs to both stdout and stderr in a file, but I also want it to display on the console so I can track its progress. The trick, of course, is to use tee (in this case, from UnxUtils). The only issue is how combine the streams, which cmd.exe is perfectly capable of doing, if you get the incantation correct:

runtests.cmd 2>&1 | tee django-tests-201010301352.log

What "2>&1" does is redirect stderr (2) into stdout (1), which is then piped into tee, which will write its input to the specified file as well as the console. With this, I can monitor the Django test runs while still keeping a log to review later.

Running the Django Test Suite on IronPython

I know, I know, I've written this post before. However, it's a lot easier now than it was back then, and as a bonus it actually runs to completion! This is, by far, the best way to gauge the status of Django on IronPython. Once the test suite passes (except for stuff that cannot be supported, like GeoDjango), then this project will be basically complete.

There is one major hitch: doctest. Doctest is an interesting Python library that tests code by comparing it to expected output (by converting the result to a string). The problem is that certain constructs (especially dictionaries) will output differently in different implementation of Python. Thus, because Django relies on doctest, many tests would fail even though they were actually correct, just because the output was different. Thankfully, the Django project is working to get rid of doctest (I smile every time I see one of Alex Gaynor’s “we have always been at war with doctest” commits go in).

I covered the setup of the environment in my post on running the Django tutorial on IronPython, this may be familiar.

First, you'll need to install IronPython. Using the latest (IronPython 2.7 Beta 1 as of this writing) is recommended. The MSI installer is best, but you can use the zip as well. You'll just need to remember where it's unpacked, as you'll need that path later.

Next, you'll need to get Django/IronPython. Because Django requires some IronPython-specific patches, you'll need to download the entire source tree. To do this, you'll need Mercurial (I recommend TortoiseHG). Create an empty directory to work in, as well. I find Mercurial easier to work with from the command line, even with TortoiseHG installed, but you can do all of this from TortoiseHG as well. You’ll also need to enable MQ and, if you’re new to Mercurial, check out this quick tour of bibucket and this great Mercurial tutorial.

To get the Django/IronPython sources, from your empty directory:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches django-ironpython
> pushd django-ironpython && hg qpush --all && popd
> hg clone http://bitbucket.org/jdhardy/django-ironpython-tests
> cd django-ironpython-tests

The first line pulls the sources from bitbucket into the 'django-ironpython' folder, the second line applies all of the Django/IronPython patches, and – here’s where it’s different from the tutorial! – the  next-to-last line clones the django-ironpython-tests repository, which contains a bunch of helpers for running the Django tests. The last line just switches to the django-ironpython-tests folder.

Now you’ll need to open testenv.cmd in a text editor and possibly make some changes. In particular, change _ipy_root to point to your IronPython installation if you didn’t use the MSI installer.
Now you can run the runtests.cmd file, which will run the entire Django test suite. There will be errors and failures, but the results are promising, with about 66% of the tests passing:

Ran 2595 tests in 3022.324s

FAILED (failures=405, errors=468, skipped=16, expected failures=1)

The elephant in the room: source control for IronPython

Currently, IronPython is hosted in a TFS repository on CodePlex (http://ironpython.codeplex.com/), which was a copy of MS's internal TFS repository. CodePlex also provides Subversion access, which makes it much more bearable. CodePlex also hosts our issue tracking and wiki pages, which probably won't change any time soon.

IronRuby's source code is hosted on github (http://github.com/ironruby/ironruby). It's also a copy of MS's internal TFS repository, but in git.

The interesting part is that IronRuby, IronPython, and the DLR are hosted in the *same* repository, since they evolved together. Thus, both the IronPython CodePlex repo and the IronRuby github repo are basically the same.
</history-lesson>

What this is going to look like in the future is an open question, as is the timeline. Originally, I wanted to focus on the 2.7 release and deal with the source control question later. However, it's been raised in a few places, so I think it's better to get some more feedback on whether we should switch source control methods (and if so, to what?) or just stay on TFS/SVN for the time being. Also up for consideration is whether you consider being part of the same repo as IronRuby is valuable, or whether IronPython should split out on its own.

We could, for example, drop the source control from CodePlex and just use the IronRuby github repo - it's already set up and we could start developing tomorrow (although it would probably be renamed 'ironlanguages' or something like that). It's also probably the only option if IronPython and IronRuby are to share a repo, as, so far as I know, the IronRuby guys have no plans on leaving github, which makes sense for them - git is the de facto choice in the Ruby community.

In Python, however, it's not so clear-cut - Python itself will be moving to Mercurial soon, and there are plans afoot to eventually put the Python stdlib in a separate repo from Python itself, which will likely also be a Mercurial repository. Thus there are advantages (subrepos, in particular) to being on the same DVCS. On top of that, both Michael Foord and I strongly dislike git - I prefer Mercurial, and I imagine the coffee at Canonical will have Michael singing the praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, and thus everything could remain there if we so wish.

However, converting the repo to Mercurial could be a difficult task - the fate of the 1.1, 2.0, and 2.6 branches would have to be decided (include them in the repo, or not? Their structure is radically different from the Main branch). There are folders that could very well be stripped (WiX, Ruby, and *3* CPython installations, not to mention IronRuby) to save space, and with a DVCS once they're in the history everyone has to pay that cost in disk space, forever, even if we later remove them. The fate of the DLR would need to be decided - do we keep a local copy, pull from IronRuby's copy, or make it a third repo altogether?

My preference is to stick with TFS/SVN for the time being, get 2.7 out the door (manually syncing up the DLR sources with IronRuby in the meantime), and then look at converting to Mercurial. My second choice would be to work out of IronRuby's git repository, get 2.7 released, and then look at converting to Mercurial. Anything that doesn't eventually involve Mercurial is a lot further down my list :).

I would like to see the DLR become a separate project, of which IronRuby and IronPython are just clients, along with IronJS, Clojure-CLR, and any others. I don't think the DLR will change too drastically, but the MS guys who are more familiar might have other plans, and Tomas has said he would prefer them to be together for ease of testing.

While the coordinators have discussed this already, I think we need more feedback to get an idea of what we should do, so please share your thoughts. This has a direct bearing on how you will be contributing to IronPython.

The Road to 2.7 (Or: The Future of IronPython)

There have been a few people asking what they can contribute to IronPython. Right now, we need to identify what's not working in 2.7B1 that needs to be in 2.7 final. The best thing to do would be to identify any issues that are causing you pain and bring them up on the list. Then we can decide what meets the bar for a 2.7 release (with issues that have patches getting priority, of course!). Dino, are there any issues that you know are in 2.7B1 that must be fixed for 2.7 final?

Dino has provided some instructions on contributing to IronPython. We need people to run through that and if there's anything it doesn't cover (I do intend to add subversion instructions directly at some point), and run the test suite as well. Also, doing all of that on Mono, to see what work needs to be done there.

Besides knowing what needs to be done, we need a rough timeline. I would like to see a release before the end of the year, or at least a solid release candidate with a possible release early next year. The idea behind an aggressive schedule is to focus on getting the features we have solid and not worry about adding new features or libraries (with the possible exception of zlib). That said, this all just my desires, and I really want to get an idea of what every one else is thinking. Please send feedback to the IronPython list on what you want to see.

Running the Django Tutorial on IronPython

The Django tutorial is fantastic. It's a great way to get a feel for Django, and a great test for how it will work on IronPython. Getting it working will give you pretty good idea of how to setup Django/IronPython for most things.
First, you'll need to install IronPython. Using the latest (IronPython 2.7 Beta 1 as of this writing) is recommended. The MSI installer is best, but you can use the zip as well. You'll just need to remember where it's unpacked, as you'll need that path later.
Next, you'll need to get Django/IronPython. Because Django requires some IronPython-specific patches, you'll need to download the entire source tree. To do this, you'll need Mercurial (I recommend TortoiseHG). Create an empty directory to work in, as well. I find Mercurial easier to work with from the command line, even with TortoiseHG installed, but you can do all of this from TortoiseHG as well. You’ll also need to enable MQ and, if you’re new to Mercurial, check out this quick tour of bibucket and this great Mercurial tutorial.
To get the Django/IronPython sources, from your empty directory:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches django-ironpython
> pushd django-ironpython && hg qpush --all && popd
> md django-tutorial && cd django-tutorial

The first line pulls the sources from bitbucket into the 'django-ironpython' folder, the second line applies all of the Django/IronPython patches, and the last line creates a directory ('django-tutorial') to hold the tutorial files, and switches into it. If you're not already using the command line, open a command window and switch to the 'django-tutorial' directory.
Now we need to set up some environment variables, which are a simple way to configure some settings for IronPython and Django. I usually copy these into a file called 'tutenv.cmd', which is easier than typing them each time (the variables are lost when the command window is closed):

@echo off

rem Get the folder this file is in
set _root=%~dp0

rem Add the IronPython installation paths to PATH; change these if you used
rem the .zip version of IronPython
set PATH=C:\Program Files\IronPython 2.7;C:\Program Files (x86)\IronPython 2.7;%PATH%

rem Add the django path and current paths to IronPython's search list
set IRONPYTHONPATH=%_root%..\django-ironpython;%_root%;%_root%deps

rem Tell Django when the its settings are
set DJANGO_SETTINGS_MODULE=mysite.settings

If you created a file, run it to set up the tutorial environment. Now, do a quick sanity check:

> ipy
IronPython 2.7 Beta 1 (2.7.0.10) on .NET 4.0.30319.1
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> ^Z

If that doesn't work, you need to make sure that the PATH variable includes your IronPython installation, and that IRONPYTHONPATH includes the path to the 'django-ironpython' folder created back at the beginning of the post.
Now we need to add in some dependencies (namely, sqlite and zlib support). These are in another bitbucket repository, django-ipy-tutorial-deps:

> hg clone http://bitbucket.org/jdhardy/django-ipy-tutorial-deps deps

Now you can start the tutorial. The first step, creating the project requires using django-admin.py; you can run it with IronPython like so:

> ipy ..\django-ironpython\django\bin\django-admin.py startproject mysite

Besides that, you can follow the tutorial almost exactly as written – just remember to substitute 'ipy' for 'python'! If you have any issues, please file them in the issue tracker.

Contributing to Django/IronPython

Since the new Django/IronPython repository (django-ipy-patches) is based on MQ, I thought I'd give a quick intro into what MQ is and how to work with it on bitbucket. First off, MQ is short "Mercurial Queues". The queue, in this case, is a queue of patches to be applied to a repo that can be versioned separately from the repo itself. These patches live independent of the history and can be easily added and removed as changes are made. You can find a lot more detailed information in the Mercurial book.

Using Bitbucket

The initial setup is a bit weird, but it works quite well after that. First off, get Mercurial – I prefer TortoiseHG, myself. Next, get a bitbucket account. Finally, go to the django-ipy-patches page and click the "fork" button. You can name your fork whatever you want. Once it's created, make a note of the URL in the "hg clone" line. For this example, it's https://bitbucket.org/jdhardy/django-ipy-patches-test.

Next, clone the django-trunk repository, but name it after your fork, and then switch to that directory:

> hg clone http://bitbucket.org/jdhardy/django-ipy-patches-test
> cd django-ipy-patches-test

Now, clone the fork you created earlier and pull in all of the patches. Note that we're pulling it into a specific directory (.hg\patches):

> hg clone http://bitbucket.org/jdhardy/django-ipy-patches-test .hg\patches
> hg qpush --all

You can now use MQ to manage patches as usual.

> hg qnew fix-1 -m "Fix issue #1"
> hg ci --mq
> hg push --mq

Now you can go back to the bitbucket page for your fork, click "pull request", and send me a request to pull your changes back into django-ipy-patches. Please do a pull from django-ipy-patches first to sync things up as much as possible.

Without Bitbucket

If you don't want to use bitbucket, you can get the django-ipy-patches queue directly:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches

Make your changes using MQ as usual, but instead of doing a push, export the changes instead:

> hg log --mq
> hg export –mq –r ...

The trick here is figuring out which changes you need to export using hg log and then export them. You should then post the bundle to the issue tracker, and I'll apply it as soon as I can.

MQ Shortcut

You may have noticed that most of the normal hg commands take an --mq parameter that causes them to operate on the MQ repository instead of the actual repository. I use a simple batch file as a shortcut so that I can use `mq push` instead of `hg push --mq`:

@hg %1 --mq %2 %3 %4 %5 %6 %7 %8 %9

Put that line in a file named "mq.cmd" somewhere in your PATH and you can save yourself a little bit of typing.

The End of the Beginning

Well, the shoe has finally dropped: IronPython and IronRuby have been axed. Jimmy Schementi spilled the beans back in August, and despite my thoughts to the contrary, the fate of the DLR team was already sealed. As best I can gather, the team was officially kaput on September 1st, or thereabouts.  I found out in late September, and since then the former DLR team has been working to make the handoff as smooth as possible.

As of today, myself, Jimmy Schementi, Michael Foord, and Miguel de Icaza have become the coordinators of the IronPython, IronRuby, and DLR projects. Bill Chiles and Dino Viehland, who worked on IronPython, will continue to work in an unofficial (i.e. spare time) role as well. Some of the old IronRuby team will probably continue to run the show over there as well. But officially, Microsoft is completely hands-off.

Why?
Why end one of the few teams that was actually doing something new and different and interesting at Microsoft? The official word is that it's because of "resource constraints" and "a shift in priorities". Now, I realize that even Microsoft has to choose where to spend their dollars, but I have a hard time believing that the half-dozen staff on the DLR team were that big a deal compared to the 200+ working on WPF and SL, or the billion-dollar KIN debacle. Maybe that's why I only make the small-to-medium dollars instead of the big bucks.

It's also a bit mystifying that Microsoft would do this after promoting the dynamic keyword so heavily in .NET 4.0. That part of the DLR (the "inner ring") isn't going anywhere – it's in the .NET framework, which means it will be supported more or less forever. The hosting APIs, or the "outer ring", is now in the hands of the community. Hopefully the people working on projects like IronScheme, IronJS, and Clojure-CLR will contribute what they want to see out of the DLR, although getting changes into the "inner ring" is likely to be impossible.

The Future
I know there are companies and software using IronPython and (I believe) IronRuby in production: Resolver One is completely built around IronPython; it's built-in to Umbraco; Jimmy has said that's its in use at Lab49. I hope that these companies will step up by offering some of their employees' time to help a project that they use. Other than that, there's at least a few Python programmers like myself who have to work in .NET, and hopefully they will also help out. The future of IronPython and IronRuby is entirely in the hands of those who use it, which is a new experience for those used to Microsoft calling all the shots.

So what's next? As a group, we're still hashing that out. Now that the cat is out of the bag, we're going to involve the community as well. My goal is to get a production-ready IronPython 2.7 released before the end of the year. To do that, we'll need continuous integration infrastructure. After that, all sorts of decisions will need to be made, from boring stuff like managing infrastructure, to setting a roadmap for 3.2, to deciding what cool .NET stuff we want to include (such as LINQ support or static compilation).

I want to see Django running on IronPython. I want to see the DLR be the system for embedding scripting into .NET applications, supporting a multitude of languages. I want the Iron* languages to become so popular that Microsoft regrets ever cutting them off. But I and other coordinators can't do it alone. We need help. We need people to contribute code, libraries, documentation - anything. From this point on, IronPython and IronRuby will live or die by their communities.

Come join the mailing lists (IronPython, IronRuby) and help us decide where we are going to go from here. This isn't the beginning of the end. Far from it.

Restarting Django/IronPython

Once again, I'm trying to work on Django/IronPython. This time around, I've finally found a workflow I'm happy with (for now) using MQ. Using MQ is a bit tricky to understand, but it has the distinct advantage that Django/IronPython patches are distinct from normal development and thus much easier to submit back to the Django project – which I also plan to start doing.

Of course, the repo has changed again – it's now django-ipy-patches. It's a patch queue (which bitbucket handles quite nicely) instead of a fork because using a fork mixed the IronPython changes in with normal development and made them hard to find amongst all the merge changesets (and you can't rebase a public repository). Forks are better for very short-lived branches, but I think patch queues are the better option for long-lived projects like this. I'll move the issues from the old repository over and then shut it down fairly soon.

I'll post some instructions soon on how to contribute and use MQ, and an update running the tests.

Using Extensions with a Custom IronPython Build

I tripped over this today – if you're using a custom build of IronPython (such as one built in debug mode), then any extensions have to be rebuilt against that build of IronPython.

This occurs because the references in the extensions are against the official builds, which are signed (strong-named), but a custom IronPython build is not signed (or signed with a different key).

Another option would be to build the extensions with a non-strong-named reference to IronPython, but I don't if this is possible or how to do it. If anyone else does, please let me know!

Thoughts on Diversification

I recently watched Rob Conery's talk from NDC2010 on his concerns about Microsoft. It's a good talk, and if you work with Microsoft tech at all (especially it it's your main technology, as it is mine), then you owe it to yourself to watch it.

Done? Good.

Rod describes the talk as 'incendiary'; I disagree. Personally, I didn't think anything in there was off the mark (although he should have left out his Twitter fight over Azure pricing, but whatever). He's right – Microsoft, overall, is not nearly as interesting as they used to be. He implores 'Microsoft developers' to look outside of Microsoft at what else is out there, and wants Microsoft to start pushing boundaries again.

None of this is really a surprise to those of us who have always had a foot outside of the Microsoft world; hell, the only reason I got a foot in the Microsoft world is because it paid the bills. I've been a Python fan since the first time I saw it, 7 or 8 years ago; one of my co-op jobs in University was working for a (sadly, now defunct) company that integrated Jython into the management software for their hardware platform. Before that I taught myself C++ after cutting my teeth on QBasic. I used Java and Ruby in University courses; the OS of choice was mainly Linux.

Knowing only one tech stack is extremely limiting and, honestly, foolish. Focusing on one is often a necessity dictated by needing to eat, but you should be keeping an eye on what other communities are doing, because you never know when it might be useful. It's truly unfortunate that many programmers didn't know about the beauty of functional programming until Microsoft introduced LINQ and F#; I've been using those techniques in Python for years and missed them horribly when working in pre-3.5 C#.

This doesn't just apply to programming. Some people recommend learning one new programming language a year; I agree, almost. I think you should learn one new skill a year. Last year I built myself an office, doing all of the carpentry, drywall, and electrical myself (with a little help from friends and family, of course). This year I'm learning how to cook properly (and have developed an unhealthy obsession with Alton Brown). Next year I plan to finally learn how to play the guitar and understand music theory (we'll see how that goes). Someday I hope to rebuild a car.

Step outside your comfort zone. Learn something new. What you find just might surprise you.