NWSGI 1.0

I'm very pleased to announce the release of NWSGI 1.0. NWSGI is an implementation of the WSGI specification (PEP 333) for IronPython. It is implemented as an ASP.NET HttpHandler and is supported on IIS 6 and IIS 7 (and should work on any other ASP.NET-capable web server). NWSGI allows the integration of Python applications with existing ASP.NET facilities such as caching, sessions, and user management.

This release does not mean that any Python application will work. The underlying IronPython 2.0 engine is not yet complete. It is a nearly-complete implementation of the NWSGI specification, and any app that takes IronPython's limitations into account should work just fine.

NWSGI fully supports xcopy deployment and medium trust environments (with the caveat that parts of the Python standard library are not supported). There is also an installer available that provides a managment UI for IIS 7.

NWSGI requires IIS (6 or 7), IronPython 2.0, and .NET Framework 2.0 SP1. All downloads include the NWSGI assembly, basic documentation, and a HelloWorld sample. The installers also include the IIS management UI.

Documentation is currently a weak spot, but I hope to improve that in the near future. In the meantime there is a README included, as well as some basic documentation at the NWSGI website.

easy_install on IronPython, Part Deux

Getting easy_install working for IronPython will be a big win for the IronPython ecosystem, and as I mentioned last time, the lack of zlib is really the only thing holding it back. Well, IronPython.Zlib solves that problem, so what else is holding up easy_install?

The big thing that's missing for setuptools is zipimport support, which is used to implement the Egg packages. To get things working, a stub zipimport module is needed with the following contents:

zipimporter = None

Next, make sure you are using at least Python 2.5.2 standard library (I'm not sure what version comes with IronPython 2.0, but it should be at least that new). The version of tarfile shipped with 2.5.0 doesn't seem to work at all.

Now some changes need to be made to the Python standard library. First up, since _winreg.error is not defined, distutils/msvccompiler.py must be patched to use WindowsError instead. We can't build C extensions anyway (or rather, there would be no point), but setuptools will still try. Then, py_compile.py needs to be changed so that the compile() funtion, well, doesn't – put "return" as the first statement, as IronPython can't create .pyc files (yet). Finally (and this is an IronPython.Zlib problem), do not throw if the CRC-32 check fails in gzip.py. A patch for all of these is available.

The next big changes is to implement binascii.crc32, which is used by the zipfile module. easy_install always tries to build an Egg, which requires zipfile (an Egg is just a .zip with some special files). This requires recompiling IronPython.Modules. I used the same (slightly questionable) CRC-32 implementation I used for IronPython.Zlib.

Once you get this far, easy_install can download, extract, and build a Python library (I used simplejson for my tests).  Installation is still a problem because pkg_resources is looking for an Egg, which require zipimport, which was blanked out in the first step (it's almost poetic, really). Getting zipimport working should be the last major hurdle.

Solving the Zlib Problem – IronPython.Zlib

IronPython 2.0 does not include an implementation of the zlib module, which handles data compression. In CPython the zlib module is a C module, which it can't be used from IronPython. It is also required by several standard Python modules, include tarfile, gzip, and zipfile, all of which handle compressed archives. Thus, for setuptools - particularly, easy_install and zipimport (Eggs) - to work, there must be a fully working zlib module.

FePy includes a partial implementation built on .NET's DeflateStream, but it is missing support for the decompressobj API that is used by gzip and zipfile. Supporting this API using DeflateStream appears to be impossible, as it expects a few implementation details of the zlib implementation.

That left two options for implementing zlib: P/Invoke, and ComponentAce's zlib.net. P/Invoke had a couple of issues: first, native-managed interop is hard; second, handling 32-bit and 64-bit systems seamlessly adds a lot of extra code (I use Vista x64, so I ran into this almost immediately). Zlib.net is an open source (BSD-style license), pure .NET implementation of zlib; this avoids both native-managed interop and 32/64-bit issues. It has an interface nearly identical to zlib, which means that it could be used to mimic the Python C library, which is exactly what I've done.

IronPython.Zlib is an implementation of the zlib module for IronPython using zlib.net. Both binary and source are available under the MS-PL license. To use it create a "DLLs" directory under your IronPython implementation and drop IronPython.Zlib.dll into it.

It passes most of the Python zlib tests and all of the gzip tests. There are a few issues with the CRC-32 implementation that the tests didn't catch but real-world usage did, but I haven't figured out how to fix them yet.

There are still some issues with setuptools (the zipfile module uses binascii.crc32, which is not yet implemented) that I'll detail in a later post, but it almost works.

Update June 27, 2009: Uploaded a new version that fixes crc32 bugs by forwarding to the now-implemented but 2.6-only binascii.crc32.

Django + IronPython

Update: See my newer post on how to run the Django test suite on IronPython 2.6.

I've had a couple of people ask about a Django + IronPython "Getting Started". This has been on my TODO list for a while; I just haven't gotten around to it. I'll try to do something over the next couple of weeks.

A couple of tips to anyone trying in the meantime:

• Get the SQLite provider from FePy SVN; it's the only one that I know works
• If manage.py has problems, file (or vote on) bugs, and use Python for manage.py instead
• Set the session backend to use the caching system, and set your cache backed to be in-memory (you can try to use the ASP.NET backends I posted earlier, but no guarantees)
• It probably won't work 100%, but I did work my way through the tutorial – except the admin system

NWSGI 0.7 Released

NWSGI 0.7 is now available on Codeplex. This release mainly deals with cleaning up some rough edges and incompatibilities with PEP 333, although there are some new features. If you used the installer for NWSGO 0.6, you must manually uninstall it before installing NWSGI 0.7.

Session Support

ASP.NET HttpHandlers must inherit from IRequiresSessionState¹ in order to have access to the Session state. This has been added to NwsgiHandler. Now, HttpContext.Session can be used to access the Session state. However, because the Session module adds a cookie, the entire response must be buffered, which is against the WSGI spec. See the section on Extensions for more details.

¹ Why did they have to add the first "s"? IRequireSessionState is too clever to pass up.

Proper Unicode Handling

IronPython does not distinguish between str and unicode (and neither does Jython). WSGI requires apps to return str objects, but allows Unicode if there is no other option and only the bottom 8 bits are used. Previously, NWSGI used the incorrect encoding (and not even consistently; I had both ASCII and UTF-8 in different places). PEP 333 specifies ISO 8859-1 (aka Latin1) for all Unicode conversions, as is maps precisely to the first 256 Unicode characters and thus preserves raw bytes (ASCII and UTF-8 mappings are required to move things around).

As a result, raw files can now be passed through NWSGI without the use of wsgi.file_wrapper, such is when generating images on the fly.

Wildcard Mappings

All HttpHandlers must have a target extension so that they can be used; NWSGI uses *.wsgi. This results in URLs such as http://www.example.com/hello/hello.wsgi/Jeff. There are two ways to handle this: URL rewriting, which is the preferred way, and wildcard mapping, which is more of a last resort as it can hurt performance for static files.²

By setting the handler path to "*", all file requests will be sent through NWSGI. To tell NWSGI that this is what you want, set the wildcardModule attribute like so:

<wsgi wildcardModule="~/hello.wsgi">

The equivalent URL would be http://www.example.com/hello/Jeff. NWSGI will handle all requests to that app by remapping SCRIPT_NAME to the application (/hello) and PATH_INFO to the remainder (/Jeff).

The wildcardModule attribute can be either absolute (C:\…) or app-relative (~/…). Using wildcardModule will bypass any script mappings, as they are unnecessary.

² One option is to move static files to a separate virtual directory or even separate server that does not use NWSGI.

WSGI Extensions

NWSGI implements a couple of non-conforming extensions to WSGI. In the interest of compatibility, these are off by default. To enable them, use the enableExtensions attribute:

<wsgi enableExtensions="true">

The current extensions are:

• nwsgi.context – provide access to the HttpContext via environ["nwsgi.context"]. This is somewhat moot as it can always be reached from HttpContext.Current.
• Responses are not flushed – PEP 333 requires the response to be flushed after every iteration of the WSGI callable. This breaks the use of ASP.NET Sessions, and possibly other IIS modules.

IronPython Version

NESGI 0.7 is linked against IronPython 2.0RC2, but RC1 should work as well – the assembly versions are the same. This hasn't been tested, however.

NWSGI on Window Azure

I have had a couple of people ask about NWSGI on Windows Azure. Architecturally (as far as NWSGI is concerned), Azure is just IIS7 running in medium trust. You have to use xcopy deployment, which makes me glad I went to the effort of supporting it.

NWSGI 0.6 actually supports medium trust. I only discovered after the release that the error I was hitting was a bug in Cassini; it works fine in medium trust on IIS7. With that hurdle out of the way, there's no reason NWSGI shouldn't work on Azure.

And, what do you know, it works just fine: NWSGI "Hello, World" for Windows Azure. You'll need the Azure SDK and the Visual Studio tools to run it.

The big downside is that every NWSGI app would have to include its entire framework and the Python standard library (or whatever subset that it uses).

Integrating Django and ASP.NET

Using NWSGI to run Django on IronPython opens up some interesting possibilities, such as using ASP.NET's caching, session, or authentication systems for Django. I imagine any ASP.NET shop spent time configuring their ASP.NET providers for those systems, and I thought it would be interesting to see if that effort could be used for Django apps as well, and it certainly looks like they can.

A demo version of django-aspnet is now available. I'll expand on the details later, but for now: set the *_BACKEND variables in settings.py to point to these modules (they must be on the Python path). All other configuration settings for the Django apps are ignored; you must use Web.config to configure them, like any ASP.NET app.

Some parts are still missing, but the basic functionality seems to work.

NWSGI 0.6 Released

NWSGI 0.6 has been released on Codeplex. This release has some possibly breaking changes and a whole host of new features. There is also an installer available along with the zip archive. The HelloWorld example has been rolled into the zip file.

There is one minor feature addition: wsgi.file_wrapper is now supported for faster file transfers. It uses HttpResponse.TransmitFile to send the file directly.

The major new feature is IIS7 management integration, including a UI editor. This is a huge addition and one I'm quite proud of. It makes configuring NWSGI a breeze from the command line (appcmd) or from the UI. The IIS7 integration is only available through the installer.

The changes are in the configuration system to support the IIS7 management system. Unfortunately, IIS7 doesn't use System.Configuration; instead, it has its own assembly, Microsoft.Web.Administration. This caused some complications for configuration. There are now three possible usage scenarios (IIS6, IIS7 xcopy, and IIS7 installed) that require slightly different web.config settings; I will detail these in an upcoming post.

Installing IIS7 UI Extensions With WiX

IIS7 includes a new set of APIs for building extensions for its UI configuration manager, inetmgr. Following the Understanding UI Extension Authoring guide on iis.net results in a pair of assemblies to be added to the GAC, a configuration schema, and some entries to be added to applicationHost.config and administration.config. This is basically a WiX-ification of the deployment guide available here.

All of these files are in %systemroot%\system32\inetsrv\config.

WiX XmlConfig

The XmlConfig element is what is used to edit existing XML files (XmlFile is used to edit files that are installed with your program). It's ugly and a pain to use correctly, but it does the trick. The reference that comes with WiX is decent, but it doesn't really show you how to use the thing.

The use of XmlConfig follows a basic pattern - create the element, add any attributes or sub-elements, and finally an XmlConfig to handle uninstallation.

applicationHost.config

The applicationHost.config file needs to be edited so that IIS7 can find the custom config section described by the schema file. This amounts to adding a <section name="wsgi" overrideMode="Deny" /> element to the configSections element.

<util:XmlConfig Id="appHostEntry"
                File="[IIS7LOCATION]\config\applicationHost.config"
                Action="create"
                ElementPath="//configuration/configSections/sectionGroup[\[]@name='system.webServer'[\]]"
                Name="section"
                Node="element"
                Sequence="1"
                On="install" />

<util:XmlConfig Id="appHostEntryName"
                File="[IIS7LOCATION]\config\applicationHost.config"
                ElementPath="appHostEntry"
                Name="name"
                Value="wsgi"
                Sequence="2" />

<util:XmlConfig Id="appHostEntryOverrideMode"
                File="[IIS7LOCATION]\config\applicationHost.config"
                ElementPath="appHostEntry"
                Name="overrideModeDefault"
                Value="Deny"
                Sequence="3" />

<util:XmlConfig Id ="removeAppHostEntry"
                File="[IIS7LOCATION]\config\applicationHost.config"
                Action="delete"
                ElementPath="//configuration/configSections/sectionGroup[\[]@name='system.webServer'[\]]"
                Node="element"
                VerifyPath="section[\[]@name='wsgi'[\]]"
                On="uninstall"
                Sequence="1" />

The first XmlConfig entry creates the section element under configSections. The next two add the attributes that I want. Note that the latter entries have their ElementPath pointing to the Id of the first entry. The final XmlConfig is used on uninstallation to remove the element.

administration.config

The entries in administration.config are used by inetmgr to find the admin modules. These follow the same basic pattern as the previous XmlConfig entries. These need to be added in two different spots, so the elements look similar for both cases.

Under moduleProviders:

<util:XmlConfig Id="adminModuleProvider" 
                File="[IIS7LOCATION]\config\administration.config" 
                Action="create" 
                ElementPath="//configuration/moduleProviders" 
                Name="add" 
                Node="element" 
                Sequence="1" 
                On="install" />

<util:XmlConfig Id="adminModuleProviderName" 
                File="[IIS7LOCATION]\config\administration.config" 
                ElementPath="adminModuleProvider" 
                Name="name" 
                Value="NWSGI" 
                Sequence="2" />

<util:XmlConfig Id="adminModuleProviderType" 
                File="[IIS7LOCATION]\config\administration.config" 
                ElementPath="adminModuleProvider" 
                Name="type" 
                Value="NWSGI.Management.NwsgiModuleProvider, NWSGI.Management, Version=0.6.0.0, Culture=neutral, PublicKeyToken=41e64ddc1bf1fc86" 
                Sequence="3" />

<util:XmlConfig Id ="removeAdminModuleProvider" 
                File="[IIS7LOCATION]\config\administration.config" 
                Action="delete" 
                ElementPath="//configuration/moduleProviders" 
                Node="element" 
                VerifyPath="add[\[]@name='NWSGI'[\]]" 
                On="uninstall" 
                Sequence="1" />

Under location:

<util:XmlConfig Id="adminModule"
                File="[IIS7LOCATION]\config\administration.config"
                Action="create"
                ElementPath="//configuration/location[\[]@path='.'[\]]/modules"
                Name="add"
                Node="element"
                Sequence="1"
                On="install" />

<util:XmlConfig Id="adminModuleName"
                File="[IIS7LOCATION]\config\administration.config"
                ElementPath="adminModule"
                Name="name"
                Value="NWSGI"
                Sequence="2" />

<util:XmlConfig Id ="removeAdminModule"
                File="[IIS7LOCATION]\config\administration.config"
                Action="delete"
                ElementPath="//configuration/location[\[]@path='.'[\]]/modules"
                Node="element"
                VerifyPath="add[\[]@name='NWSGI'[\]]"
                On="uninstall"
                Sequence="1" />

schema.xml

Installing the schema.xml file is done with a File element, just like any other file.

NWSGI 0.5 Released

The latest version of NWSGI, built for IronPython 2.0 Beta 5, has been posted. There are a couple of changes in this version: one breaking change and one new feature.

Configuraton Changes

To fit with convention, the configuration elements now start with lowercase letters - i.e. Wsgi => wsgi, OSEnviron => osEnviron, etc. This has a consequence because XML is case-sensitive, so the old configuration elements will not be picked up and the defaults will be used instead.

Script Mappings

Script mappings can be used to create a virtual .wsgi file that points to a file elsewhere on the filesystem. For example:

<scriptmappings>
    <scriptmapping scriptname="dispatch.wsgi" physicalpath="C:\myapp\dispatch.wsgi">
</scriptmappings>

Rather than copy the dispatch.wsgi file into the web aplication directory, any request to dispatch.wsgi will be mapped to c:\myapp\dispatch.wsgi instead. This example is a bit trivial, as an IIS virtual directory could probably be used, but it could be useful for any application that ships a .wsgi file.

Future Directions

I'm working on building an IIS7 admin interface for all of this; it's a bit of a mess right now because the advanced UI tricks aren't really documented that well. I'm hoping to have the IIS7 admin and the installer ready by the time IronPython goes 2.0.

easy_install on IronPython

setuptools is a critical piece of the Python ecosystem: it extends distutils, but more importantly it provides easy_install, which is used by almost every Python library and application to handle installation. Making easy_install work in IronPython would make it far easier to download and install Python libraries.

The good news is that it's not too hard to get going; there are only a couple of bugs that need to be worked around. The bad news is that it can't work with any of the files that are downloaded, as the zlib module is shot.

This is all on IronPython 2.0b4.

Starting Off

First off, get the trunk version of setuptools: http://svn.python.org/projects/sandbox/trunk/setuptools. Under the checkout, run "python setup.py develop" (that will hopefully use ipy soon as well). Then, apply the following small patch:

==========================================================--- pkg_resources.py    (revision 65738)
+++ pkg_resources.py    (working copy)
@@ -2066,9 +2066,10 @@
             return str(self)

     def __str__(self):
-        try: version = getattr(self,'version',None)
-        except ValueError: version = None
-        version = version or "[unknown version]"
+        version = "[unknown version]"
         return "%s %s" % (self.project_name,version)

     def __getattr__(self,attr):

This is to work around #17810.

From your Python standard library, edit urllib2.py at line 1096 as follows:

+        fp = socket._fileobject(r, close=True)
-        fp = socket._fileobject(r)

The is work around #17894.

Under your IronPython\Lib directory, add a site-packages directory.

The zlib problem

This will allow you to run "ipy easy_install.py <package>". It will download the file, but fail (with a misleading error; there's an exception being swallowed somewhere) because there is no zlib module, and the one FePy is incomplete. Specifically, it is missing zlib.decompressobj.

I looked at implementing zlib.decompressobj, but it doesn't look like it will be easy with the .NET DeflateStream API.

Summary

easy_install can be made to work, but you can't do anything useful with it until the zlib module is fully implemented.

NWSGI 0.4 Released

To coincide with the release of IronPython 2.0b4, NWSGI(http://codeplex.com/NWSGI) has been updated to version 0.4. This version also brings improvements to configuration and performance.

Download NWSGI 0.4.

Configuration

This version brings major improvements to configuration. Previously, the only way to configure NWSGI was to use some undocumented AppSettings parameters. Now there is a full ConfigSection that changing NWSGI's behaviour a breeze. More information on the new configuration options is available here.

Performance

Older versions on NWSGI rebuilt the ScriptRuntime and ScriptEngine on each request; this is extremely slow. Now, each the ScriptRuntime and ScriptEngine and created once and each request is executed in its own ScriptScope. This saves some work on each request, but it also means that imported modules are cached instead of reloaded on each request.

This has two consequences: requests are processed much quicker, but the server must be manually recycled/restarted if the Python code changes. As IronPython's Achilles heel is still the slow import of Python modules, this is a big win for performance, but it does slow down development and debugging.

Conclusion

I'll be doing some Django testing (as well as some other apps) on 2.0b4, and I hope to be able to post a workable patch for Django soon.

Django on IronPython: On Databases

Django pretty well requires a database to do anything useful. It includes backends for a number of databases, such as mysql, portgres, and sqlite. Seeing as sqlite is by far the easiest to install, I started there. Using the dbapi module form FePy as a starting point, I extended the sqlite module to make it work for django (and made some tweaks to dbapi.py in the process).

There were a few hurdles to jump through, but for the most part it worked. I was able to run though the Django tutorial and build the complete 'poll' application. Except for some double-escaping on the admin page and a stupid bug in NWSGI, it works.

From there I started trying to get MS SQL Server working. This one is what most people would be using anywhere, and it doesn't require an extra assembly. For the most part it works, as long as you insure that all of the cursors are closed. If not, it doesn't like to work. The SQL Server classes really don't like having a DataReader open when committing a transaction. Unfortunately, manually closing cursors isn't going to scale to all of Django, so another approach will be necessary.

Using the sqlite3 driver requires System.Data.SQLite. Make sure you use the correct architecture; on Vista 64, IronPython requires the x64 driver.

Using the mssql driver requires code from the django-mssql project, with a minor change to base.py to load the mssql driver instead of the internal one.

Download sqlite3 and mssql drivers.

Django on IronPython, Part II

Progress! The admin module (mostly) works. I've cobbled together an SQLite module that works with the existing backend, and made quite a few workarounds. This is encouraging.

Django on IronPython

The very basics of Django will work on IronPython. Now I need to hook up a database and see how that works. There are a few minor changes required to work around some IronPython bugs, but nothing major. I'll post them later. Note that this is using NWSGI 0.3 + Cassini; I don't know if the Django webserver works. EDIT: Here is the subversion patch:

Index: http/__init__.py
===================================================================
--- http/__init__.py (revision 8069)
+++ http/__init__.py (working copy)
@@ -14,7 +14,7 @@
 from django.http.multipartparser import MultiPartParser
 from django.conf import settings
 from django.core.files import uploadhandler
-from utils import *
+from django.http.utils import *

 RESERVED_CHARS="!*'();:@&=+$,/?%#[]"

Index: utils/functional.py
===================================================================
--- utils/functional.py (revision 8069)
+++ utils/functional.py (working copy)
@@ -160,10 +160,11 @@
             for resultclass in resultclasses:
                 self.__dispatch[resultclass] = {}
                 for (k, v) in resultclass.__dict__.items():
-                    setattr(self, k, self.__promise__(resultclass, k, v))
+                    if k != "__doc__":
+                        setattr(self, k, self.__promise__(resultclass, k, v))
             self._delegate_str = str in resultclasses
             self._delegate_unicode = unicode in resultclasses
-            assert not (self._delegate_str and self._delegate_unicode), "Cannot call lazy() with both str and unicode return types."
+            #~ assert not (self._delegate_str and self._delegate_unicode), "Cannot call lazy() with both str and unicode return types."
             if self._delegate_unicode:
                 # Each call to lazy() makes a new __proxy__ object, so this
                 # doesn't interfere with any other lazy() results.
Index: utils/translation/__init__.py
===================================================================
--- utils/translation/__init__.py (revision 8069)
+++ utils/translation/__init__.py (working copy)
@@ -42,9 +42,11 @@
     # Make the originally requested function call on the way out the door.
     return g['real_%s' % caller](*args, **kwargs)

+import trans_null
 g = globals()
 for name in __all__:
-    g['real_%s' % name] = delayed_loader
+    if hasattr(trans_null, name):
+        g['real_%s' % name] = getattr(trans_null, name)
 del g, delayed_loader

 def gettext_noop(message):