Tuesday, November 23, 2010

Developing IronPython with Mercurial

While the informal poll of the IronPython community favoured Mercurial, Miguel de Icaza had some good arguments for using github to develop IronPython and IronRuby (and I figured it was best to heed the advice of someone who's been running OSS projects since I was in elementary school). In particular, there was already an up-to-date git repo available.

However! All is not lost, thanks to the wonderful hg-git plugin. With it, you can push and pull from a git repository, but still use the Mercurial commands you know and love. To save everyone the hassle, though, I've set up a read-only mirror on bitbucket that is synchronized with the github repo. You can find it at http://bitbucket.org/ironpython/ironlanguages.

I assume you have some familiarity with Mercurial. If not, take a look at Hg Init and then give the Mercurial book a quick read to get a handle on the basic concepts.

In this post I'll show how to work with the IronPython repository on bitbucket. In a later post, I'll cover how to work directly with the github repository as well as a more advanced use of bitbucket.

Initial Setup

Before doing anything else, you'll need the latest version of Mercurial. On Windows, I strongly recommend TortoiseHG because it already includes almost everything you'll need.

Now, make sure the following sections are in your Mercurial.ini file:

[extensions]
mq=
rebase=
bookmarks=

[diff]
git=1

While none of these are strictly required, the bookmarks extension is needed if you want to work with git directly, and rebase and mq are just really nice to have. You'll want them later anyway, so you might as well turn them all on now. Git diffs are more useful than the unified diffs Mercurial generates by default, so turn those on as well.

Working With a Bitbucket Fork

To use bitbucket forks, you'll need a bitbucket account. If you don't already have one, go to bitbucket.org and create an account. For all of the details on creating a fork, I'll just point at Bitbucket's own documentation, which is very good. In short, go to http://bitbucket.org/ironpython/ironlanguages, click on "fork", and give the fork a descriptive name and create it.

Now, create a clone of your fork:

> hg clone https://yourname@bitbucket.org/yourname/yourfork

With your own fork you can push to it so that others can see your changes. However, pulling will (by default) pull from your fork, which is not automatically kept in sync with the original repository. To get any changes from the original repository, you'll need to pull from it directly:

> hg pull --update http://bitbucket.org/ironpython/ironlanguages

To save a bit of typing, add an alias to the [paths] section of the .hg\hgrc file of your clone:

[paths]
ironlanguages=http://bitbucket.org/ironpython/ironlanguages

And then you can do a pull from the alias instead of the URL:

> hg pull --update ironlanguages

You'll probably need to do a merge at this point. If you haven't pushed any changes publically, you could use hg rebase as well, but in this case I highly recommend using hg merge because it's safer. You should merge from upstream only as often as you need to, as too many merge commits can clutter the history. However, always merge in the upstream changes just before you send a pull request. It will make my life much easier.

Now you can use Mercurial as you normally would to track whatever changes you are making.

Once your changes are done, you'll need to send a pull request. First, if you haven't already, commit and push your changes to your fork:

> hg commit
> hg push

Then, go to the original repository (http://bitbucket.org/ironpython/ironlanguages), click "pull request", and enter a quick explanatory message. One of the IronPython coordinators (most likely me) will pull the request into our local repo, review it, and then push it to the main repo on github; this will trigger a synchronization, which will then pull your changes into the bitbucket mirror. Phew.

Working Directly With the Bitbucket Mirror

While using your own fork is highly recommended (because pull requests make integrating the changes very easy), you may want to work with the mirror repository directly. In particular, you don't need a bitbucket account to work with the mirror directly. However, the mirror is read-only, so if you choose to work this way, you'll need to submit patches (described below). For minor changes -- basically, a single commit -- this is fine, but for larger changes I strongly recommend using a fork or MQ, or it will make creating the patch extremely difficult.

First, clone the mirror repository:

> hg clone http://bitbucket.org/ironpython/ironlanguages

You can now use hg as usual to track your changes. I recommend using rebase to stay up to date, to keep the history free of merges:

> hg pull --rebase

Rebase Warning

For the most part, the rebase extension is safe to use, with one exception: do not use rebase on a repo that you've made public unless you want to break every other clone of that repo. The only time I recommend using it is when you only have a local clone, because it will make the resulting patches cleaner than repeated merges.

If you want rebase-like functionality that is safe to push publically, you should use MQ.

When your changes are complete, you'll need to create a patch and submit it.

Submitting a Patch

First, either find the corresponding issue that your patch is resolving or create a new one. Next, do an hg pull --rebase to make sure that you're up–to-date with the latest changes, which will make the patch easier to apply.

Now you need to determine which revisions are needed for the patch using hg log. Ideally, this is only the most recent revision (tip), or the last few revisions. Once you know what revisions comprise your changes, you can use hg export to create a patch (which can then be applied using hg import – funny, that).

If your fix is only the most recent commit, it's fairly easy:

> hg export tip –-output 12345-fix-broken-foo.patch

When naming the patch file, please include the issue number and a very brief description in the name of the file.

If the patch has more than one commit, you'll need to specify the revisions to export. In this example, we take all of the commits from revision 123 and to the tip:

> hg export 123: –-output 12346-fix-broken-bar.patch

If there are other revisions (such as merge commits) intermixed with work commits, it is still possible to produce a clean patch, but you'll need to use the full revision set query language to do it. In that case you're probably better off creating a fork, merging it in, pushing to it, and sending a pull request.

Once the patch is created, attach it to the appropriate issue and someone should pick it up. If we miss it (sorry!), just send an email to the mailing list to remind us.