Debugging Techniques for IronPython: Breakpoints

Currently, programs IronPython is not particularly easy to debug – they Python code gets mixed in with the code for IronPython itself and can be hard to follow. This is the first of a few tricks to make debugging easier.

Trigger a Breakpoint

This is my favourite and most used technique – I sometimes wish Python had the equivalent of JavaScript's debugger keyword to make it easier to use. The trick is to use the System.Diagnostics.Debugger.Break() function to insert a breakpoint in your code – when the breakpoint is hit, Windows will offer to launch a debugger for you. Also make sure that you launch IronPython with the –X:Debug switch, or it will be next to impossible to debug.

Insert the following line where you want to trigger the breakpoint:

import System.Diagnostics; System.Diagnostics.Debugger.Break()

Windows will then open the following dialog (if you have Visual Studio installed; I don't have a machine without VS to see what it looks like otherwise):

I prefer to use VS for debugging, so I'll open a  new instance of Visual Studio 2008.

Browsing the Call Stack

Now, we'll need to go looking for the actual Python code in the call stack. Open the call stack browser, and you'll see something like this:

There are a few things to note here, especially if you're not familiar with the VS debugger:

• Each line represents a call stack frame (a function call, basically).
• The yellow arrow is the current execution location; this is our breakpoint.
• Faded-out lines are frames that do not have any debug information available. They can be ignored.
• Lines beginning with "Snippets.debug" are the actual Python stack frames we are looking for. Double click on those lines to open up the source for those files in the Visual Studio editor.

Once the Python source is loaded in the editor, you can inspect variable values by hovering over them with the mouse, just like in any other Visual Studio language. To find the calling stack frame, you'll have to skip over several IronPython frames to find the next Python frame.

A better experience would elide the IronPython frames; perhaps Harry Pierson's debugger work could be applied to improve it. That, and other Visual Studio improvements for IronPython, are another project entirely.

Running the Django Test Suite On IronPython

UPDATE: A simpler version of these instructions is available on the django-ironpython Bitbucket page.

This guide will explain how to setup and attempt to run the Django test suite on IronPython. Once the test suite runs, it should be much easier to fill in the parts of Django that don't work properly.

What You'll Need

It's not terribly difficult to set this up, but there are quite a few pieces.

• IronPython 2.6 – use the installer so that you get the standard library as well.
• Django trunk – an SVN checkout for now; if you have Mercurial, you can get it from django-ironpython.  Following the hg repo will get you my IronPython fixes.
• adonet-dbapi – For the sqlite3 module implementation – MS SQL is a future target, but not right now (use the "get source" link if you don't have hg installed).
• System.Data.SQLite – used by the sqlite3 module (just download the binary zip; no need to install)
• 1 cup flour…

Getting Started

First up, install IronPython and checkout the Django trunk and adonet-dbapi somewhere. I'll use %USERPROFILE%\Documents\Repositories\django\ and %USERPROFILE%\Documents\Repositories\adonet-dbapi\ in these examples. Next, create a "DLLs" folder in the IronPython install folder and drop System.Data.SQLite.dll into it (this way IronPython will automatically reference it).

The next step is to prepare the Django test suite. This requires you to create a small Django app that contains things like database settings. The full instructions can be found on Alex Gaynor's blog, or you can download it and unzip it (I'll assume %USERPROFILE%\Documents\Repositories\django-test\).

Running the Tests

OK, command prompt time – if you're not comfortable with the command prompt, this won't be for you.

set PATH=%PATH%;C:\Program Files (x86)\IronPython 2.6
set IRONPYTHONPATH=%USERPROFILE%\Documents\Repositories\django\;%USERPROFILE%\Documents\Repositories\adonet-dbapi\abapi;%USERPROFILE%\Documents\Repositories\django-test\
set DJANGO_SETTINGS_MODULE=django_test.settings

Now, make sure you're in the django directory and run the test cases:

cd %USERPROFILE%\Documents\Repositories\django\
ipy tests\runtests.py -v 1

So far, so good.

The Problems

It'll bomb immediately on an assertion failure. Django does not like the fact that on IronPython str == unicode and thus their lazy evaluation doesn't work immediately (see issue #1 for details). Comment out that assertion, and it fails again – and I haven't fixed this one yet. Stay tuned.

LINQ Support for IronPython

This is a collection of thoughts and a summary of a mailing list conversation about LINQ support in IronPython. The IronPython team (which seems to be just Dino & Dave at this point) seemed receptive to the idea; it's just a matter of resources. Therefore, please vote on the linked issues so that they can get their priorities straight.

There are two key parts to LINQ support: extension methods and expression trees. Each is useful on its own, but both are required to really take advantage of LINQ.

Extensions Methods

An extension method is a way of adding a function to a class without editing the class, or providing a default implementation for a class implementing an interface. LINQ is almost entirely composed of extension methods on the IEnumerable<T> and IQueryable<T>interfaces, so supporting them in IronPython is critical to supporting LINQ and many other interfaces. In this case I'm only talking about IronPython being able to c0nsume extension methods, not create them.

In C#, an extension method is made available by a using directive for the namespace containing a static class that contains the method. For example, the Enumerable class is in the namespace System.Linq and contains about half of LINQ's extension methods. To use this class from C# requires only:

using System.Linq;

The Python equivalent of this would be:

from System.Linq import *

Now, this style is frowned upon in Python circles because it pollutes the namespace unnecessarily. A more Pythonic equivalent would be:

from System.Linq import Enumerable

When doing an import, IronPython would have to check if Enumerable contains any static methods marked with ExtensionAttribute and add them to the list of possible methods to resolve for the applicable type. I actually tried to implement this at one point, but haven't had the time to finish it up.

The issue for this is CodePlex #17250 - Support for LINQ extension methods.

Expression Trees

An expression tree is an abstract, language-independent representation of a piece of code that can be more easily parsed and transformed than the raw code it was generated from. From the expression tree, the LINQ provider (such as LINQ to SQL) determines how to convert it into a query in its target language (such as SQL). The DLR actually uses "expression" trees as well (a superset of the LINQ classes that support statements as well as expressions), which are compiled into IL code and then executed.

In C# (and VB), lambda expressions are convertible to expression trees. Python also has lambda expressions, and these should also be convertible to expression trees – in particular, expression trees that exactly match what would be created by the C# compiler for an equivalent lambda, which is what every existing LINQ provider would expect.

The issue for this is CodePlex #26044 - lambda should be convertible to Expression<...>.

IronPython Web Roles for Windows Azure

Following up to my previous post on IronPython worker roles, I'm going to discuss how to implement web roles using IronPython. The code discussed here is on bitbucket, as usual.

There are currently two ways to implement IronPython web roles on Azure: using the ASP.NET Dynamic Language Support or using NWSGI. Someday I hope the IronRubyMVC project matures to be an option as well.

Using ASP.NET Dynamic Language Support

This example can be found in the PyWebRole folder of the project.

First, you'll need to download the ASP.NET Dynamic Language Support (ASP.NET DLS) files. Next, create a standard C# web role and delete all of the .cs files except WebRole.cs. After that, there's really not much to it – drop the assemblies from the ASP.NET DLS package into the project's bin folder and add the following bits to the web.config file (alternatively, just copy the Web.config from the project):

<configSections>
    <!-- Add this to the existing <configSections> element -->
    <section name="microsoft.scripting" type="Microsoft.Scripting.Hosting.Configuration.Section, Microsoft.Scripting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="false"/>
</configSections>

<system.web>
    <!-- Replace the existing <pages> element (if any) -->
    <pages compilationMode="Auto" pageParserFilterType="Microsoft.Web.Scripting.UI.NoCompileCodePageParserFilter" pageBaseType="Microsoft.Web.Scripting.UI.ScriptPage" userControlBaseType="Microsoft.Web.Scripting.UI.ScriptUserControl">
        <controls>
            <add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
            <add tagPrefix="asp" namespace="System.Web.UI.WebControls" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"/>
        </controls>
    </pages>
</system.web>

<system.webserver>
    <modules>
        <!-- Add this to the existing <modules> element -->
        <add name="DynamicLanguageHttpModule" preCondition="integratedMode" type="Microsoft.Web.Scripting.DynamicLanguageHttpModule"/>
    </modules>
</system.webserver>

<!-- Add this whole section -->
<microsoft.scripting debugMode="true">
    <languages>
        <language names="IronPython;Python;py" extensions=".py" displayName="IronPython 2.6" type="IronPython.Runtime.PythonContext, IronPython, Version=2.6.10920.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </languages>
</microsoft.scripting>

From here, just follow the directions on the ASP.NET DLS page and in the package to create an ASP.NET Dynamic Language site.

NWSGI

First, download NWSGI 2.0. Next, deploying NWSGI to Azure is exactly the same as deploying it to any other server using xcopy. That's about it. You can see this example in the PyHelloWorld folder of the project.

Now What?

Unfortunately, now you're pretty much on your own. So far, none of the big Python web frameworks work 100% on IronPython, there is no Python library to access Azure's storage tables, blobs, or queues, and the .NET storage client library relies on LINQ support (which IronPython doesn't have, yet).

Personally, I plan to implement the data access in C# and call that from IronPython, running my own web framework. We'll see how it goes.

IronPython Worker Roles for Windows Azure

Curiosity has finally got the better of me and I've started looking into Windows Azure again. It's matured quite a bit since I looked at it last year and now looks like a pretty solid platform to work with.

Aside from using NWSGI to write a web role (which I'll show later), I wanted to see if it was possible to write a worker role in Python. Happily, it is, and it's not that complicated. In fact, it's pretty similar to how NWSGI works – load up a Python file and run some functions.

Worker Role Requirements

A standard C# worker roles requires three functions: OnStart, OnStop, and Run:

public class PyWorkerRole : RoleEntryPoint
{
    public override bool OnStart() { /* ... */ }
    public override void OnStop() { /* ... */ }
    public override void Run() { /* ... */ }
}

This can be mapped to a Python module in a fairly straightforward fashion:

def start():
    return True

def run():
    pass

def stop():
    pass

This has some advantages and disadvantages compared to using a class, but I like it for its simplicity.

The Implementation

Azure requires an actual .NET class to implement a worker role, so we create one that hosts the IronPython engine. This is a good example of how to embed IronPython to run very simple scripts. The core IronPython hosting function is shown here; for the rest, see the files linked below.

private void InitScripting(string scriptName)
{
    this.engine = Python.CreateEngine();
    this.engine.Runtime.LoadAssembly(typeof(string).Assembly);
    this.engine.Runtime.LoadAssembly(typeof(DiagnosticMonitor).Assembly);
    this.engine.Runtime.LoadAssembly(typeof(RoleEnvironment).Assembly);
    this.engine.Runtime.LoadAssembly(typeof(Microsoft.WindowsAzure.CloudStorageAccount).Assembly);                 
    
    this.scope = this.engine.CreateScope();
    engine.CreateScriptSourceFromFile(scriptName).Execute(scope);             
    
    if(scope.ContainsVariable("start"))
        this.start = scope.GetVariable<Func<bool>>("start");
    
    this.run = scope.GetVariable<Action>("run");
    
    if(scope.ContainsVariable("stop"))
        this.stop = scope.GetVariable<Action>("stop");
}

First, we create a ScriptEngine and add some useful assemblies;  then we create a Scope to execute in; then we actually execute the script. Finally, we try to pull out the functions and convert them to C# delegates; run is required but start and stop are optional. Those delegates are called from the C# wrapper (from Run, OnStart, and OnStop, as appropriate).

The rest of the file is pretty much taken from the worker role template, so I'll leave it out.

Doing Actual Work

Now, a worker role needs some actual work to do – usually, reading items from a queue and processing them. Happily, the Azure StorageClient library is perfectly usable from IronPython.

from Microsoft.WindowsAzure import CloudStorageAccount
from Microsoft.WindowsAzure.StorageClient import CloudQueueMessage, CloudStorageAccountStorageClientExtensions

def run():
    account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString")
    queueClient = CloudStorageAccountStorageClientExtensions.CreateCloudQueueClient(account)
    queue = queueClient.GetQueueReference("messagequeue")

    while True:
        Thread.Sleep(10000)

        if queue.Exists():
            msg = queue.GetMessage()
            if msg:
                Trace.TraceInformation("Message '%s' processed." % msg.AsString)
                queue.DeleteMessage(msg)

The only catch is that CreateCloudQueueClient is an extension method, so it must be called as a static method on the CloudStorageAccountStorageClientExtensions class.

Using the Code

To actually use the code, create a C# worker role as per usual, but replace the generated class file with PythonWorkerRole.cs (see below). Next, add the IronPython assemblies as references to the project. Then, create a string setting for the role (under the Cloud project's Roles folder) called ScriptName and set it to the name of the script file. Finally, add a .py file to the worker role and ensure that (under 'Properties') its 'Build Action' is 'Content' and 'Copy to Output' is 'Copy if Newer'.

The code can be downloaded from my PyAzureExamples repository, including zip archives of it. It includes the PyWorkerRole project and the Cloud Service project.