Running the Django Test Suite on IronPython

I know, I know, I've written this post before. However, it's a lot easier now than it was back then, and as a bonus it actually runs to completion! This is, by far, the best way to gauge the status of Django on IronPython. Once the test suite passes (except for stuff that cannot be supported, like GeoDjango), then this project will be basically complete.

There is one major hitch: doctest. Doctest is an interesting Python library that tests code by comparing it to expected output (by converting the result to a string). The problem is that certain constructs (especially dictionaries) will output differently in different implementation of Python. Thus, because Django relies on doctest, many tests would fail even though they were actually correct, just because the output was different. Thankfully, the Django project is working to get rid of doctest (I smile every time I see one of Alex Gaynor’s “we have always been at war with doctest” commits go in).

I covered the setup of the environment in my post on running the Django tutorial on IronPython, this may be familiar.

First, you'll need to install IronPython. Using the latest (IronPython 2.7 Beta 1 as of this writing) is recommended. The MSI installer is best, but you can use the zip as well. You'll just need to remember where it's unpacked, as you'll need that path later.

Next, you'll need to get Django/IronPython. Because Django requires some IronPython-specific patches, you'll need to download the entire source tree. To do this, you'll need Mercurial (I recommend TortoiseHG). Create an empty directory to work in, as well. I find Mercurial easier to work with from the command line, even with TortoiseHG installed, but you can do all of this from TortoiseHG as well. You’ll also need to enable MQ and, if you’re new to Mercurial, check out this quick tour of bibucket and this great Mercurial tutorial.

To get the Django/IronPython sources, from your empty directory:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches django-ironpython
> pushd django-ironpython && hg qpush --all && popd
> hg clone http://bitbucket.org/jdhardy/django-ironpython-tests
> cd django-ironpython-tests

The first line pulls the sources from bitbucket into the 'django-ironpython' folder, the second line applies all of the Django/IronPython patches, and – here’s where it’s different from the tutorial! – the  next-to-last line clones the django-ironpython-tests repository, which contains a bunch of helpers for running the Django tests. The last line just switches to the django-ironpython-tests folder.

Now you’ll need to open testenv.cmd in a text editor and possibly make some changes. In particular, change _ipy_root to point to your IronPython installation if you didn’t use the MSI installer.
Now you can run the runtests.cmd file, which will run the entire Django test suite. There will be errors and failures, but the results are promising, with about 66% of the tests passing:

Ran 2595 tests in 3022.324s

FAILED (failures=405, errors=468, skipped=16, expected failures=1)

Running the Django Tutorial on IronPython

The Django tutorial is fantastic. It's a great way to get a feel for Django, and a great test for how it will work on IronPython. Getting it working will give you pretty good idea of how to setup Django/IronPython for most things.
First, you'll need to install IronPython. Using the latest (IronPython 2.7 Beta 1 as of this writing) is recommended. The MSI installer is best, but you can use the zip as well. You'll just need to remember where it's unpacked, as you'll need that path later.
Next, you'll need to get Django/IronPython. Because Django requires some IronPython-specific patches, you'll need to download the entire source tree. To do this, you'll need Mercurial (I recommend TortoiseHG). Create an empty directory to work in, as well. I find Mercurial easier to work with from the command line, even with TortoiseHG installed, but you can do all of this from TortoiseHG as well. You’ll also need to enable MQ and, if you’re new to Mercurial, check out this quick tour of bibucket and this great Mercurial tutorial.
To get the Django/IronPython sources, from your empty directory:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches django-ironpython
> pushd django-ironpython && hg qpush --all && popd
> md django-tutorial && cd django-tutorial

The first line pulls the sources from bitbucket into the 'django-ironpython' folder, the second line applies all of the Django/IronPython patches, and the last line creates a directory ('django-tutorial') to hold the tutorial files, and switches into it. If you're not already using the command line, open a command window and switch to the 'django-tutorial' directory.
Now we need to set up some environment variables, which are a simple way to configure some settings for IronPython and Django. I usually copy these into a file called 'tutenv.cmd', which is easier than typing them each time (the variables are lost when the command window is closed):

@echo off

rem Get the folder this file is in
set _root=%~dp0

rem Add the IronPython installation paths to PATH; change these if you used
rem the .zip version of IronPython
set PATH=C:\Program Files\IronPython 2.7;C:\Program Files (x86)\IronPython 2.7;%PATH%

rem Add the django path and current paths to IronPython's search list
set IRONPYTHONPATH=%_root%..\django-ironpython;%_root%;%_root%deps

rem Tell Django when the its settings are
set DJANGO_SETTINGS_MODULE=mysite.settings

If you created a file, run it to set up the tutorial environment. Now, do a quick sanity check:

> ipy
IronPython 2.7 Beta 1 (2.7.0.10) on .NET 4.0.30319.1
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> ^Z

If that doesn't work, you need to make sure that the PATH variable includes your IronPython installation, and that IRONPYTHONPATH includes the path to the 'django-ironpython' folder created back at the beginning of the post.
Now we need to add in some dependencies (namely, sqlite and zlib support). These are in another bitbucket repository, django-ipy-tutorial-deps:

> hg clone http://bitbucket.org/jdhardy/django-ipy-tutorial-deps deps

Now you can start the tutorial. The first step, creating the project requires using django-admin.py; you can run it with IronPython like so:

> ipy ..\django-ironpython\django\bin\django-admin.py startproject mysite

Besides that, you can follow the tutorial almost exactly as written – just remember to substitute 'ipy' for 'python'! If you have any issues, please file them in the issue tracker.

Contributing to Django/IronPython

Since the new Django/IronPython repository (django-ipy-patches) is based on MQ, I thought I'd give a quick intro into what MQ is and how to work with it on bitbucket. First off, MQ is short "Mercurial Queues". The queue, in this case, is a queue of patches to be applied to a repo that can be versioned separately from the repo itself. These patches live independent of the history and can be easily added and removed as changes are made. You can find a lot more detailed information in the Mercurial book.

Using Bitbucket

The initial setup is a bit weird, but it works quite well after that. First off, get Mercurial – I prefer TortoiseHG, myself. Next, get a bitbucket account. Finally, go to the django-ipy-patches page and click the "fork" button. You can name your fork whatever you want. Once it's created, make a note of the URL in the "hg clone" line. For this example, it's https://bitbucket.org/jdhardy/django-ipy-patches-test.

Next, clone the django-trunk repository, but name it after your fork, and then switch to that directory:

> hg clone http://bitbucket.org/jdhardy/django-ipy-patches-test
> cd django-ipy-patches-test

Now, clone the fork you created earlier and pull in all of the patches. Note that we're pulling it into a specific directory (.hg\patches):

> hg clone http://bitbucket.org/jdhardy/django-ipy-patches-test .hg\patches
> hg qpush --all

You can now use MQ to manage patches as usual.

> hg qnew fix-1 -m "Fix issue #1"
> hg ci --mq
> hg push --mq

Now you can go back to the bitbucket page for your fork, click "pull request", and send me a request to pull your changes back into django-ipy-patches. Please do a pull from django-ipy-patches first to sync things up as much as possible.

Without Bitbucket

If you don't want to use bitbucket, you can get the django-ipy-patches queue directly:

> hg qclone http://bitbucket.org/jdhardy/django-ipy-patches

Make your changes using MQ as usual, but instead of doing a push, export the changes instead:

> hg log --mq
> hg export –mq –r ...

The trick here is figuring out which changes you need to export using hg log and then export them. You should then post the bundle to the issue tracker, and I'll apply it as soon as I can.

MQ Shortcut

You may have noticed that most of the normal hg commands take an --mq parameter that causes them to operate on the MQ repository instead of the actual repository. I use a simple batch file as a shortcut so that I can use `mq push` instead of `hg push --mq`:

@hg %1 --mq %2 %3 %4 %5 %6 %7 %8 %9

Put that line in a file named "mq.cmd" somewhere in your PATH and you can save yourself a little bit of typing.

Restarting Django/IronPython

Once again, I'm trying to work on Django/IronPython. This time around, I've finally found a workflow I'm happy with (for now) using MQ. Using MQ is a bit tricky to understand, but it has the distinct advantage that Django/IronPython patches are distinct from normal development and thus much easier to submit back to the Django project – which I also plan to start doing.

Of course, the repo has changed again – it's now django-ipy-patches. It's a patch queue (which bitbucket handles quite nicely) instead of a fork because using a fork mixed the IronPython changes in with normal development and made them hard to find amongst all the merge changesets (and you can't rebase a public repository). Forks are better for very short-lived branches, but I think patch queues are the better option for long-lived projects like this. I'll move the issues from the old repository over and then shut it down fairly soon.

I'll post some instructions soon on how to contribute and use MQ, and an update running the tests.

Running the Django Test Suite On IronPython

UPDATE: A simpler version of these instructions is available on the django-ironpython Bitbucket page.

This guide will explain how to setup and attempt to run the Django test suite on IronPython. Once the test suite runs, it should be much easier to fill in the parts of Django that don't work properly.

What You'll Need

It's not terribly difficult to set this up, but there are quite a few pieces.

• IronPython 2.6 – use the installer so that you get the standard library as well.
• Django trunk – an SVN checkout for now; if you have Mercurial, you can get it from django-ironpython.  Following the hg repo will get you my IronPython fixes.
• adonet-dbapi – For the sqlite3 module implementation – MS SQL is a future target, but not right now (use the "get source" link if you don't have hg installed).
• System.Data.SQLite – used by the sqlite3 module (just download the binary zip; no need to install)
• 1 cup flour…

Getting Started

First up, install IronPython and checkout the Django trunk and adonet-dbapi somewhere. I'll use %USERPROFILE%\Documents\Repositories\django\ and %USERPROFILE%\Documents\Repositories\adonet-dbapi\ in these examples. Next, create a "DLLs" folder in the IronPython install folder and drop System.Data.SQLite.dll into it (this way IronPython will automatically reference it).

The next step is to prepare the Django test suite. This requires you to create a small Django app that contains things like database settings. The full instructions can be found on Alex Gaynor's blog, or you can download it and unzip it (I'll assume %USERPROFILE%\Documents\Repositories\django-test\).

Running the Tests

OK, command prompt time – if you're not comfortable with the command prompt, this won't be for you.

set PATH=%PATH%;C:\Program Files (x86)\IronPython 2.6
set IRONPYTHONPATH=%USERPROFILE%\Documents\Repositories\django\;%USERPROFILE%\Documents\Repositories\adonet-dbapi\abapi;%USERPROFILE%\Documents\Repositories\django-test\
set DJANGO_SETTINGS_MODULE=django_test.settings

Now, make sure you're in the django directory and run the test cases:

cd %USERPROFILE%\Documents\Repositories\django\
ipy tests\runtests.py -v 1

So far, so good.

The Problems

It'll bomb immediately on an assertion failure. Django does not like the fact that on IronPython str == unicode and thus their lazy evaluation doesn't work immediately (see issue #1 for details). Comment out that assertion, and it fails again – and I haven't fixed this one yet. Stay tuned.

Django + IronPython

Update: See my newer post on how to run the Django test suite on IronPython 2.6.

I've had a couple of people ask about a Django + IronPython "Getting Started". This has been on my TODO list for a while; I just haven't gotten around to it. I'll try to do something over the next couple of weeks.

A couple of tips to anyone trying in the meantime:

• Get the SQLite provider from FePy SVN; it's the only one that I know works
• If manage.py has problems, file (or vote on) bugs, and use Python for manage.py instead
• Set the session backend to use the caching system, and set your cache backed to be in-memory (you can try to use the ASP.NET backends I posted earlier, but no guarantees)
• It probably won't work 100%, but I did work my way through the tutorial – except the admin system

Integrating Django and ASP.NET

Using NWSGI to run Django on IronPython opens up some interesting possibilities, such as using ASP.NET's caching, session, or authentication systems for Django. I imagine any ASP.NET shop spent time configuring their ASP.NET providers for those systems, and I thought it would be interesting to see if that effort could be used for Django apps as well, and it certainly looks like they can.

A demo version of django-aspnet is now available. I'll expand on the details later, but for now: set the *_BACKEND variables in settings.py to point to these modules (they must be on the Python path). All other configuration settings for the Django apps are ignored; you must use Web.config to configure them, like any ASP.NET app.

Some parts are still missing, but the basic functionality seems to work.

Django on IronPython: On Databases

Django pretty well requires a database to do anything useful. It includes backends for a number of databases, such as mysql, portgres, and sqlite. Seeing as sqlite is by far the easiest to install, I started there. Using the dbapi module form FePy as a starting point, I extended the sqlite module to make it work for django (and made some tweaks to dbapi.py in the process).

There were a few hurdles to jump through, but for the most part it worked. I was able to run though the Django tutorial and build the complete 'poll' application. Except for some double-escaping on the admin page and a stupid bug in NWSGI, it works.

From there I started trying to get MS SQL Server working. This one is what most people would be using anywhere, and it doesn't require an extra assembly. For the most part it works, as long as you insure that all of the cursors are closed. If not, it doesn't like to work. The SQL Server classes really don't like having a DataReader open when committing a transaction. Unfortunately, manually closing cursors isn't going to scale to all of Django, so another approach will be necessary.

Using the sqlite3 driver requires System.Data.SQLite. Make sure you use the correct architecture; on Vista 64, IronPython requires the x64 driver.

Using the mssql driver requires code from the django-mssql project, with a minor change to base.py to load the mssql driver instead of the internal one.

Download sqlite3 and mssql drivers.