Biting Off More Than I Can Chew

I was going through the “Repositories” folder on my machine, looking at all of the stuff I’ve downloaded since, well, the last time I went through this exercise. This time around, a lot of it is stuff I’ve tried to get working on IronPython. Here’s a sampling:

• Django
• setuptools
• Trac
• Genshi
• Mercurial
• SCons
• CherryPy
• docutils
• moin
• pygments
• pymarkdown
• nose
• sqlalchemy
• IronRubyMVC

On top of that, there’s the stuff I wrote: NWSGI, IronPython.Zlib, adonet-dbapi, and more.

Unfortunately, I just don’t have time to track all (or even some) of those any more. Debugging things is also pretty difficult – IronPython is getting good enough that the only bugs are really obscure. It probably takes me ten times longer to find a bug and isolate it into a testcase than it does for Dino to fix it. There are other things in real life that have changed as well, drastically reducing the time I can spend on this stuff.

Somewhere along the line I developed a weird sense of duty to make that list of stuff work on IronPython, even though I didn’t have a use for most of it. Realistically, getting all of that to work, and finding all of the little corner cases of Python that they exploit but IronPython doesn’t implement, would be a full time job.

So, I’m cutting back. Way back. IronPython-wise, I’m only going to focus on the stuff I’ve written (and yes, I will be getting Beta 2 of NWSGI 2.0 out soon), and Iron[Ruby|Python]MVC . For me, those are going to be the most immediately rewarding, and I can also stop worrying about the others.

Now, with all of that said, I have an offer: if anyone wants to work on the top part of that list (up to and including SCons), I’ll help get you up to speed on some of the issues with those programs, and some of the tricks involved in debugging them (hint: Debugger.Break). This will be  limited-time offer, valid until I forget the details of the program in question. I really want to see these things compatible with IronPython, but I just no longer have the time to do it myself.

NWSGI 2.0: Removing URL Warts

The normal procedure for configuring NWSGI leaves an unsightly URL wart: the URL must contain the .wsgi file. There are two ways to get rid of that wart: URL rewriting and wildcards.

URL Rewriting

When using URL rewriting, the web server changes ("rewrites") the incoming URL into something NWSGI can understand. IIS 7 has a URL rewriting extension to do just that. It's quite easy to use, too:

<rewrite> 
    <rules> 
        <clear /> 
        <rule name="Redirect Warty Requests" stopProcessing="true"> 
            <match url="simple.wsgi/(.*)" /> 
            <conditions logicalGrouping="MatchAll" /> 
            <action type="Redirect" url="{R:1}" redirectType="Permanent" /> 
        </rule> 
        <rule name="Remove WSGI Wart"> 
            <match url="^(.*)$" /> 
            <conditions logicalGrouping="MatchAll" /> 
            <action type="Rewrite" url="simple.wsgi/{R:1}" appendQueryString="true" /> 
        </rule> 
    </rules> 
</rewrite>

Replace simple.wsgi with the name of the .wsgi file you would normally use. Now, http://example.com/simple/simple.wsgi/Products/Widgets can be accessed as http://example.com/simple/Products/Widgets. The first rule ("Redirect Warty Requests") will redirect any uses of http://example.com/simple/simple.wsgi/Products/Widgets to http://example.com/simple/Products/Widgets to ensure that only one set of URLs is used (which is important for search engines). However, most modern applications expect to have their URLs rewritten and can be configured to generate the "clean" URLs by default.

If you are using IIS 6, you'll need to use something like ISAPI_Rewrite to achieve the same effect. The following should have the same effect as the above rules:

RewriteEngine On
RewriteRule ^hello.wsgi/(.*) $1 [NC, R=301]
RewriteRule ^(.*)$ hello.wsgi/$1 [NC]

Wildcards

Normally, IIS dispatches requests by extension; this is why we have the .wsgi file in the first place. However, it can be configured to pass all requests to a specific handler by configuring a wildcard extension. Unlike URL rewriting, which is transparent to NWSGI, NWSGI needs to know that it has been configured as a wildcard handler. To do this, add a <wildcard> element to the configuration:

<wsgi>
    <wildcard physicalPath="C:\simple\simple.wsgi" callable="simple_app" />
</wsgi>

<system.web>
    <httpHandlers>
        <add verb="*" path="*" type="NWSGI.WsgiHandler, NWSGI, Version=2.0.0.0, Culture=neutral, PublicKeyToken=41e64ddc1bf1fc86" />
    </httpHandlers>
</system.web>
<system.webServer>
    <handlers>
        <add name="WsgiHandler" path="*" verb="*" type="NWSGI.WsgiHandler, NWSGI, Version=2.0.0.0, Culture=neutral, PublicKeyToken=41e64ddc1bf1fc86" resourceType="Unspecified" />
    </handlers>
    <validation validateIntegratedModeConfiguration="false" />
</system.webServer>

The physicalPath and callable attributes have the same meaning as on the <scriptMapping> element. If the configuration includes a <wildcard> element, any <scriptMapping> elements are ignored. Also, make sure that the path attribute of the handler mappings in set to *.

Ultimately, the effect is the same as URL rewriting: http://example.com/simple/simple.wsgi/Products/Widgets can be accessed as http://example.com/simple/Products/Widgets.

Which to Choose

If possible, you should prefer URL rewriting, as it should be faster than using wildcard mappings; however, it requires support from the application. If you're on IIS 6 or IIS 7 without a URL Rewrite extension installed, or your application doesn't support rewritten URLs, then wildcard mappings are available.

NWSGI 2.0: Advanced Dispatching

Function Callables

Let’s say we’re dealing with a much larger Python application, coolapp, that is installed at C:\coolapp-1.1\. Unfortunately, coolapp doesn’t provide a ready-made .wsgi file for us, but it does provide a function(coolapp.dispatchers.wsgi_application) that we can use by writing a little wrapper that looks something like:

import sys
sys.path.append("C:\coolapp-1.1")
import coolapp.dispatchers.wsgi_application
application = coolapp.dispatchers.wsgi_application

We could save this as coolapp.wsgi and follow either of two methods from last time, but since we’re not doing anything fancy in the wrapper, NWSGI provides a shortcut:

<wsgi>
    <pythonPaths>
        <path path="C:\coolapp-1.1" />
    </pythonPaths>
    <scriptMappings>
        <scriptMapping scriptName="coolapp.wsgi" callable="coolapp.dispatchers.wsgi_application" />
    </scriptMappings>
</wsgi>

First, we add C:\coolapp-1.1 to Python’s path so that we can import it. Next, we tell it that any requests for coolapp.wsgi should be dispatched to the function coolapp.dispatchers.wsgi_application. This is completely equivalent to the wrapper file, but saves us from creating it. Visiting http://example.com/coolapp.wsgi/ will run coolapp.

Class Callables

Functions aren’t the only things in Python that are callable; classes are also callable (it creates an instance of the class). Let’s say we upgrade coolapp to version 2.0 (installed in C:\coolapp-2.0). The team has made some changes, and their WSGI application is now a class, WsgiApplication. This class has a __call__ method (making its instances callable as well!), so our WSGI application is an instance of the WsgiApplication class. We could use a coolapp.wsgi wrapper again:

import sys
sys.path.append("C:\coolapp-2.1")
import coolapp.dispatchers.WsgiApplication
application = coolapp.dispatchers.WsgiApplication()

Again, our little wrapper doesn’t do much, so NWSGI provides a shortcut:

<wsgi>
    <pythonPaths>
        <path path="C:\coolapp-2.0" />
    </pythonPaths>
    <scriptMappings>
        <scriptMapping scriptName="coolapp.wsgi" callable="coolapp.dispatchers.WsgiApplication()" />
    </scriptMappings>
</wsgi>

This time, NWSGI will create an instance of WsgiApplication, and then call the instance and return the result to IIS. This shortcut will only work if the class doesn’t require any arguments; if it does, it needs to be wrapped in a .wsgi file (when in doubt, you can always use a wrapper file). Again, visit http://example.com/coolapp.wsgi/ to run the application.

There’s actually one more case: Python classes that implement the iterator protocol, but I don’t think two many applications are implemented that, so I’ll skip it. PEP 333 gives an example if you’re really interested.

NWSGI 2.0: Dispatching

NWSGI has a fairly simple task: given a URL, call some Python code that produces some output. The devil, as always, is in the details.

What is WSGI?

NWSGI is an implementation of the Python WSGI specification (PEP 333). WSGI is the Web Server Gateway Interface, “a simple and universal interface between web servers and web applications or frameworks”, as defined in PEP 333.

OK, so what does that mean?

The purpose of WSGI is to define how web servers (i.e. Apache, IIS, etc.) talk to Python web application or frameworks (i.e. Trac, Django, etc.). In theory, a web server with a WSGI implementation (such as mod_wsgi for Apache, or NWSGI for IIS) should be able to run any web application that is implemented as a WSGI application (prior to WSGI, many applications/frameworks were server-specific). Of course, that doesn’t always pan out in practice, but it’s a start.

What is a WSGI application?

A WSGI application is surprisingly simple:

def simple_app(environ, start_response):
    """Simplest possible application object"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

The simple_app function is the entire application! A WSGI application is a callable (a function, mostly) that takes two arguments and returns an iterable (a list or generator, mostly). There’s some extra goo in there as well (start_response, for example) that’s not really relevant as far as dispatching is concerned. Writing WSGI applications is probably a book’s worth of content, so I’ll leave it at that for now.

Finding Callables in .wsgi Files

Local Files

Now that we have a callable for our application, NWSGI needs to know how to find it. The simplest way is to simply drop the file (let’s call it simple.wsgi) into the application root (i.e. C:\inetpub\wwwroot\, next to web.config). If you make a request to http://example.com/simple.wsgi/, IIS will see that .wsgi is associated with NWSGI and pass the request to it. NWSGI will then open simple.wsgi, try to find a variable called application (which is, by convention, the name of WSGI callables), call it, and return the result to IIS.

In this case, however, our callable is named simple_app, not application. To be able to run our app, NWSGI needs to know this! To do this, there needs to be a script mapping telling NWSGI how to run simple.wsgi.

<wsgi>
    <scriptMappings>
        <scriptMapping scriptName="simple.wsgi" callable="simple_app" />
    </scriptMappings>
</wsgi>

This configuration tells NWSGI that for the script simple.wsgi, it should use simple_app as the callable instead of application. (NOTE: This exact syntax requires 2.0b2 or later).

Other Files

Best practices for WSGI applications say that you should not put your application files in the web-exposed directory. This means that simple.wsgi should really live somewhere else; let’s say C:\simple\simple.wsgi. Of course, now NWSGI doesn’t have the slightest clue where it is, so we have to tell it:

<wsgi>
    <scriptMappings>
        <scriptMapping scriptName="simple.wsgi" physicalPath="C:\simple\simple.wsgi" callable="simple_app" />
    </scriptMappings>
</wsgi>

There is no longer a .wsgi in our application folder, but IIS doesn’t care. If you visit http://example.com/simple.wsgi/ with this configuration, IIS still happily passes the simple.wsgi script onto NWSGI. NWSGI looks at the script mappings and finds that simple.wsgi maps to C:\simple\simple.wsgi, so it loads that file instead (without looking for simple.wsgi in the application directory), looks up simple_app, and calls it.

Using .wsgi files is the simplest way to deploy an application, but NWSGI has a couple of shortcuts to make things easier for certain applications.

NWSGI 2.0: Configuration Details

For the most part, NWSGI can be used without any configuration. All you have to do is create a .wsgi file, throw it in a directory, let IIS know what’s going on, and you’re good to go! Of course, if it was always that easy, configuration wouldn’t exist. To really work with NWSGI, you need to understand its configuration. It helps if you’re already familiar with web.config files; if not, check this out (but hurry back!).

First Things First

NWSGI uses .NET’s built-in configuration system (System.Configuration), so the first thing that needs to be done is to let the .NET Framework know about the custom NWSGI configuration section:

<configSections>
    <section name="wsgi" type="NWSGI.WsgiSection" />
</configSections>

This must be the first entry in web.config, directly under the <configuration> element.

The <wsgi> element

All of the configuration options for NWSGI are under the <wsgi> element. The first ones we will look at are specified as attributes of the <wsgi> element: enableExtensions, adaptiveCompilation and frames.

enableExtensions

Enable extensions to NWSGI. Relying on these will make your application nonportable to other WSGI implementations. Required to use ASP.NET's built-in Session support, for example. Choices: true, false.

compilation

Control how IronPython compiles code. Choices: Adaptive, Compiled.

frames

Control sys._getframe. Enabling sys._getframe causes a performance hit. Choices: Off, On, Full.

Example:

<wsgi adaptiveCompilation="false" frames="Full" />

Adding Python Search Paths

If your application needs to access Python modules that are not in the default NWSGI search paths (~/, ~/Bin/ and ~/Bin/Lib/), you can use the <pythonPaths> element to add them. The <pythonPaths> element contains a set of <path> elements that are added (in order) to sys.path before the application is run. The <path> element has one required attribute – path – that contains the full path to a folder on disk containing the modules you want to use.

Example:

<wsgi>
    <pythonPaths>
        <path path="C:\django" />
    </pythonPaths>
</wsgi>

Adding WSGI Environment Variables

Most applications will require some configuration of their own; these options are passed in the environ parameter of the WSGI application. You can add entries to this list by using the <wsgiEnviron> element, which contains a set of <variable> elements. Each <variable> element has two required attributes, name and value.

Example:

<wsgi>
    <wsgiEnviron>
        <variable name="trac.env_path" value="C:\trac\projects" />
    </wsgiEnviron>
</wsgi>

Advanced Script Usage

There are cases where you may need to configure how NWSGI handles a particular script file. In these cases you need a virtual script mapping; NWSGI will always check script mappings before looking for a file on disk. Not surprisingly, these are stored in the <scriptMappings> element, which contains a set of <scriptMapping> elements. There are three parts to a script mapping:

scriptName – required

The name of the virtual script; overrides any scripts on disk.

physicalPath

The path to the actual file to execute. This can be a full path (i.e. C:\myapp\myapp.wsgi) or an app-relative path (i.e. ~/scripts/myapp.wsgi).

callable

If physicalPath is set, the name of the callable object in that file; if it is not set, an entry point for the application.

Entry points are used to specify a particular function in a Python module on sys.path (see the <pythonPaths> element above for how to control sys.path). For example, for Django the entry point would be "django.core.handlers.wsgi.WSGIHandler()". This mimics the standard Python syntax for creating an object; it is this object that is used as the application. If, instead, the application provides a function (like trac), the entry point would be "trac.web.main.dispatch_request".

Example:

<scriptMappings>
    <scriptMapping scriptName="myapp.wsgi" physicalPath="C:\myapp\myapp.wsgi" callable="myapp" />
</scriptMappings>

Wildcard Settings

When IIS is configured in wildcard mode, all requests for an application are passed to NWSGI, instead of only requests for .wsgi files. You can enable this by setting path="*" in the handler/httpHandler definition. Because NWSGI normally expects there to be a .wsgi file in the URL, but in wildcard mode there isn't one, the <wildcard> element is required to tell NWSGI what script to execute. The available settings – physicalPath and callable – have the same meaning as the same settings for script mappings (above). If the <wildcard> element is present, the <scriptMappings> element is ignored.

Example:

<wildcard physicalPath="C:\HelloWorld\hello.wsgi" callable="hello" />

Conclusion

NWSGI is extremely configurable; in many cases, you don't even need a .wsgi file. That said, the configuration can be complex, so please ask if you run into any problems. A later post will go into more detail on the mechanics of callables.