Developing IronPython with Mercurial

While the informal poll of the IronPython community favoured Mercurial, Miguel de Icaza had some good arguments for using github to develop IronPython and IronRuby (and I figured it was best to heed the advice of someone who's been running OSS projects since I was in elementary school). In particular, there was already an up-to-date git repo available.

However! All is not lost, thanks to the wonderful hg-git plugin. With it, you can push and pull from a git repository, but still use the Mercurial commands you know and love. To save everyone the hassle, though, I've set up a read-only mirror on bitbucket that is synchronized with the github repo. You can find it at http://bitbucket.org/ironpython/ironlanguages.

I assume you have some familiarity with Mercurial. If not, take a look at Hg Init and then give the Mercurial book a quick read to get a handle on the basic concepts.

In this post I'll show how to work with the IronPython repository on bitbucket. In a later post, I'll cover how to work directly with the github repository as well as a more advanced use of bitbucket.

Initial Setup

Before doing anything else, you'll need the latest version of Mercurial. On Windows, I strongly recommend TortoiseHG because it already includes almost everything you'll need.

Now, make sure the following sections are in your Mercurial.ini file:

[extensions]
mq=
rebase=
bookmarks=

[diff]
git=1

While none of these are strictly required, the bookmarks extension is needed if you want to work with git directly, and rebase and mq are just really nice to have. You'll want them later anyway, so you might as well turn them all on now. Git diffs are more useful than the unified diffs Mercurial generates by default, so turn those on as well.

Working With a Bitbucket Fork

To use bitbucket forks, you'll need a bitbucket account. If you don't already have one, go to bitbucket.org and create an account. For all of the details on creating a fork, I'll just point at Bitbucket's own documentation, which is very good. In short, go to http://bitbucket.org/ironpython/ironlanguages, click on "fork", and give the fork a descriptive name and create it.

Now, create a clone of your fork:

> hg clone https://yourname@bitbucket.org/yourname/yourfork

With your own fork you can push to it so that others can see your changes. However, pulling will (by default) pull from your fork, which is not automatically kept in sync with the original repository. To get any changes from the original repository, you'll need to pull from it directly:

> hg pull --update http://bitbucket.org/ironpython/ironlanguages

To save a bit of typing, add an alias to the [paths] section of the .hg\hgrc file of your clone:

[paths]
ironlanguages=http://bitbucket.org/ironpython/ironlanguages

And then you can do a pull from the alias instead of the URL:

> hg pull --update ironlanguages

You'll probably need to do a merge at this point. If you haven't pushed any changes publically, you could use hg rebase as well, but in this case I highly recommend using hg merge because it's safer. You should merge from upstream only as often as you need to, as too many merge commits can clutter the history. However, always merge in the upstream changes just before you send a pull request. It will make my life much easier.

Now you can use Mercurial as you normally would to track whatever changes you are making.

Once your changes are done, you'll need to send a pull request. First, if you haven't already, commit and push your changes to your fork:

> hg commit
> hg push

Then, go to the original repository (http://bitbucket.org/ironpython/ironlanguages), click "pull request", and enter a quick explanatory message. One of the IronPython coordinators (most likely me) will pull the request into our local repo, review it, and then push it to the main repo on github; this will trigger a synchronization, which will then pull your changes into the bitbucket mirror. Phew.

Working Directly With the Bitbucket Mirror

While using your own fork is highly recommended (because pull requests make integrating the changes very easy), you may want to work with the mirror repository directly. In particular, you don't need a bitbucket account to work with the mirror directly. However, the mirror is read-only, so if you choose to work this way, you'll need to submit patches (described below). For minor changes -- basically, a single commit -- this is fine, but for larger changes I strongly recommend using a fork or MQ, or it will make creating the patch extremely difficult.

First, clone the mirror repository:

> hg clone http://bitbucket.org/ironpython/ironlanguages

You can now use hg as usual to track your changes. I recommend using rebase to stay up to date, to keep the history free of merges:

> hg pull --rebase

Rebase Warning

For the most part, the rebase extension is safe to use, with one exception: do not use rebase on a repo that you've made public unless you want to break every other clone of that repo. The only time I recommend using it is when you only have a local clone, because it will make the resulting patches cleaner than repeated merges.

If you want rebase-like functionality that is safe to push publically, you should use MQ.

When your changes are complete, you'll need to create a patch and submit it.

Submitting a Patch

First, either find the corresponding issue that your patch is resolving or create a new one. Next, do an hg pull --rebase to make sure that you're up–to-date with the latest changes, which will make the patch easier to apply.

Now you need to determine which revisions are needed for the patch using hg log. Ideally, this is only the most recent revision (tip), or the last few revisions. Once you know what revisions comprise your changes, you can use hg export to create a patch (which can then be applied using hg import – funny, that).

If your fix is only the most recent commit, it's fairly easy:

> hg export tip –-output 12345-fix-broken-foo.patch

When naming the patch file, please include the issue number and a very brief description in the name of the file.

If the patch has more than one commit, you'll need to specify the revisions to export. In this example, we take all of the commits from revision 123 and to the tip:

> hg export 123: –-output 12346-fix-broken-bar.patch

If there are other revisions (such as merge commits) intermixed with work commits, it is still possible to produce a clean patch, but you'll need to use the full revision set query language to do it. In that case you're probably better off creating a fork, merging it in, pushing to it, and sending a pull request.

Once the patch is created, attach it to the appropriate issue and someone should pick it up. If we miss it (sorry!), just send an email to the mailing list to remind us.

Combining stdout, stderr, and pipes on Windows

This post is as much for my own reference as anyone else's. I need to capture the output of a program that outputs to both stdout and stderr in a file, but I also want it to display on the console so I can track its progress. The trick, of course, is to use tee (in this case, from UnxUtils). The only issue is how combine the streams, which cmd.exe is perfectly capable of doing, if you get the incantation correct:

runtests.cmd 2>&1 | tee django-tests-201010301352.log

What "2>&1" does is redirect stderr (2) into stdout (1), which is then piped into tee, which will write its input to the specified file as well as the console. With this, I can monitor the Django test runs while still keeping a log to review later.

Running the Django Test Suite on IronPython

I know, I know, I've written this post before. However, it's a lot easier now than it was back then, and as a bonus it actually runs to completion! This is, by far, the best way to gauge the status of Django on IronPython. Once the test suite passes (except for stuff that cannot be supported, like GeoDjango), then this project will be basically complete.

There is one major hitch: doctest. Doctest is an interesting Python library that tests code by comparing it to expected output (by converting the result to a string). The problem is that certain constructs (especially dictionaries) will output differently in different implementation of Python. Thus, because Django relies on doctest, many tests would fail even though they were actually correct, just because the output was different. Thankfully, the Django project is working to get rid of doctest (I smile every time I see one of Alex Gaynor’s “we have always been at war with doctest” commits go in).

I covered the setup of the environment in my post on running the Django tutorial on IronPython, this may be familiar.

First, you'll need to install IronPython. Using the latest (IronPython 2.7 Beta 1 as of this writing) is recommended. The MSI installer is best, but you can use the zip as well. You'll just need to remember where it's unpacked, as you'll need that path later.

Next, you'll need to get Django/IronPython. Because Django requires some IronPython-specific patches, you'll need to download the entire source tree. To do this, you'll need Mercurial (I recommend TortoiseHG). Create an empty directory to work in, as well. I find Mercurial easier to work with from the command line, even with TortoiseHG installed, but you can do all of this from TortoiseHG as well. You’ll also need to enable MQ and, if you’re new to Mercurial, check out this quick tour of bibucket and this great Mercurial tutorial.

To get the Django/IronPython sources, from your empty directory:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches django-ironpython
> pushd django-ironpython && hg qpush --all && popd
> hg clone http://bitbucket.org/jdhardy/django-ironpython-tests
> cd django-ironpython-tests

The first line pulls the sources from bitbucket into the 'django-ironpython' folder, the second line applies all of the Django/IronPython patches, and – here’s where it’s different from the tutorial! – the  next-to-last line clones the django-ironpython-tests repository, which contains a bunch of helpers for running the Django tests. The last line just switches to the django-ironpython-tests folder.

Now you’ll need to open testenv.cmd in a text editor and possibly make some changes. In particular, change _ipy_root to point to your IronPython installation if you didn’t use the MSI installer.
Now you can run the runtests.cmd file, which will run the entire Django test suite. There will be errors and failures, but the results are promising, with about 66% of the tests passing:

Ran 2595 tests in 3022.324s

FAILED (failures=405, errors=468, skipped=16, expected failures=1)

The elephant in the room: source control for IronPython

Currently, IronPython is hosted in a TFS repository on CodePlex (http://ironpython.codeplex.com/), which was a copy of MS's internal TFS repository. CodePlex also provides Subversion access, which makes it much more bearable. CodePlex also hosts our issue tracking and wiki pages, which probably won't change any time soon.

IronRuby's source code is hosted on github (http://github.com/ironruby/ironruby). It's also a copy of MS's internal TFS repository, but in git.

The interesting part is that IronRuby, IronPython, and the DLR are hosted in the *same* repository, since they evolved together. Thus, both the IronPython CodePlex repo and the IronRuby github repo are basically the same.
</history-lesson>

What this is going to look like in the future is an open question, as is the timeline. Originally, I wanted to focus on the 2.7 release and deal with the source control question later. However, it's been raised in a few places, so I think it's better to get some more feedback on whether we should switch source control methods (and if so, to what?) or just stay on TFS/SVN for the time being. Also up for consideration is whether you consider being part of the same repo as IronRuby is valuable, or whether IronPython should split out on its own.

We could, for example, drop the source control from CodePlex and just use the IronRuby github repo - it's already set up and we could start developing tomorrow (although it would probably be renamed 'ironlanguages' or something like that). It's also probably the only option if IronPython and IronRuby are to share a repo, as, so far as I know, the IronRuby guys have no plans on leaving github, which makes sense for them - git is the de facto choice in the Ruby community.

In Python, however, it's not so clear-cut - Python itself will be moving to Mercurial soon, and there are plans afoot to eventually put the Python stdlib in a separate repo from Python itself, which will likely also be a Mercurial repository. Thus there are advantages (subrepos, in particular) to being on the same DVCS. On top of that, both Michael Foord and I strongly dislike git - I prefer Mercurial, and I imagine the coffee at Canonical will have Michael singing the praises of bzr fairly soon :). Finally, CodePlex supports Mercurial, and thus everything could remain there if we so wish.

However, converting the repo to Mercurial could be a difficult task - the fate of the 1.1, 2.0, and 2.6 branches would have to be decided (include them in the repo, or not? Their structure is radically different from the Main branch). There are folders that could very well be stripped (WiX, Ruby, and *3* CPython installations, not to mention IronRuby) to save space, and with a DVCS once they're in the history everyone has to pay that cost in disk space, forever, even if we later remove them. The fate of the DLR would need to be decided - do we keep a local copy, pull from IronRuby's copy, or make it a third repo altogether?

My preference is to stick with TFS/SVN for the time being, get 2.7 out the door (manually syncing up the DLR sources with IronRuby in the meantime), and then look at converting to Mercurial. My second choice would be to work out of IronRuby's git repository, get 2.7 released, and then look at converting to Mercurial. Anything that doesn't eventually involve Mercurial is a lot further down my list :).

I would like to see the DLR become a separate project, of which IronRuby and IronPython are just clients, along with IronJS, Clojure-CLR, and any others. I don't think the DLR will change too drastically, but the MS guys who are more familiar might have other plans, and Tomas has said he would prefer them to be together for ease of testing.

While the coordinators have discussed this already, I think we need more feedback to get an idea of what we should do, so please share your thoughts. This has a direct bearing on how you will be contributing to IronPython.

The Road to 2.7 (Or: The Future of IronPython)

There have been a few people asking what they can contribute to IronPython. Right now, we need to identify what's not working in 2.7B1 that needs to be in 2.7 final. The best thing to do would be to identify any issues that are causing you pain and bring them up on the list. Then we can decide what meets the bar for a 2.7 release (with issues that have patches getting priority, of course!). Dino, are there any issues that you know are in 2.7B1 that must be fixed for 2.7 final?

Dino has provided some instructions on contributing to IronPython. We need people to run through that and if there's anything it doesn't cover (I do intend to add subversion instructions directly at some point), and run the test suite as well. Also, doing all of that on Mono, to see what work needs to be done there.

Besides knowing what needs to be done, we need a rough timeline. I would like to see a release before the end of the year, or at least a solid release candidate with a possible release early next year. The idea behind an aggressive schedule is to focus on getting the features we have solid and not worry about adding new features or libraries (with the possible exception of zlib). That said, this all just my desires, and I really want to get an idea of what every one else is thinking. Please send feedback to the IronPython list on what you want to see.