easy_install on IronPython, Part Deux

Getting easy_install working for IronPython will be a big win for the IronPython ecosystem, and as I mentioned last time, the lack of zlib is really the only thing holding it back. Well, IronPython.Zlib solves that problem, so what else is holding up easy_install?

The big thing that's missing for setuptools is zipimport support, which is used to implement the Egg packages. To get things working, a stub zipimport module is needed with the following contents:

zipimporter = None

Next, make sure you are using at least Python 2.5.2 standard library (I'm not sure what version comes with IronPython 2.0, but it should be at least that new). The version of tarfile shipped with 2.5.0 doesn't seem to work at all.

Now some changes need to be made to the Python standard library. First up, since _winreg.error is not defined, distutils/msvccompiler.py must be patched to use WindowsError instead. We can't build C extensions anyway (or rather, there would be no point), but setuptools will still try. Then, py_compile.py needs to be changed so that the compile() funtion, well, doesn't – put "return" as the first statement, as IronPython can't create .pyc files (yet). Finally (and this is an IronPython.Zlib problem), do not throw if the CRC-32 check fails in gzip.py. A patch for all of these is available.

The next big changes is to implement binascii.crc32, which is used by the zipfile module. easy_install always tries to build an Egg, which requires zipfile (an Egg is just a .zip with some special files). This requires recompiling IronPython.Modules. I used the same (slightly questionable) CRC-32 implementation I used for IronPython.Zlib.

Once you get this far, easy_install can download, extract, and build a Python library (I used simplejson for my tests).  Installation is still a problem because pkg_resources is looking for an Egg, which require zipimport, which was blanked out in the first step (it's almost poetic, really). Getting zipimport working should be the last major hurdle.

easy_install on IronPython

setuptools is a critical piece of the Python ecosystem: it extends distutils, but more importantly it provides easy_install, which is used by almost every Python library and application to handle installation. Making easy_install work in IronPython would make it far easier to download and install Python libraries.

The good news is that it's not too hard to get going; there are only a couple of bugs that need to be worked around. The bad news is that it can't work with any of the files that are downloaded, as the zlib module is shot.

This is all on IronPython 2.0b4.

Starting Off

First off, get the trunk version of setuptools: http://svn.python.org/projects/sandbox/trunk/setuptools. Under the checkout, run "python setup.py develop" (that will hopefully use ipy soon as well). Then, apply the following small patch:

==========================================================--- pkg_resources.py    (revision 65738)
+++ pkg_resources.py    (working copy)
@@ -2066,9 +2066,10 @@
             return str(self)

     def __str__(self):
-        try: version = getattr(self,'version',None)
-        except ValueError: version = None
-        version = version or "[unknown version]"
+        version = "[unknown version]"
         return "%s %s" % (self.project_name,version)

     def __getattr__(self,attr):

This is to work around #17810.

From your Python standard library, edit urllib2.py at line 1096 as follows:

+        fp = socket._fileobject(r, close=True)
-        fp = socket._fileobject(r)

The is work around #17894.

Under your IronPython\Lib directory, add a site-packages directory.

The zlib problem

This will allow you to run "ipy easy_install.py <package>". It will download the file, but fail (with a misleading error; there's an exception being swallowed somewhere) because there is no zlib module, and the one FePy is incomplete. Specifically, it is missing zlib.decompressobj.

I looked at implementing zlib.decompressobj, but it doesn't look like it will be easy with the .NET DeflateStream API.

Summary

easy_install can be made to work, but you can't do anything useful with it until the zlib module is fully implemented.