Getting ISAPI-WSGI from Subversion Repository

Had an anonymous email asking how to check-out isapi-wsgi from subversion. The following command will do it:

svn co http://svn.isapi-wsgi.python-hosting.com/trunk

Hex Dump Tools written in Python

In reviewing my blog access stats, it would appear that most used search engine keyword combination that sends visitors to my site are "hex" and "dump". So to not disappoint these 20% of visitors, I have tracked down a number of hex dumping tools written in my favorite programming language.

READBIN by Tony Dycks is a Text Console-based program which reads a single Input File specified on the command line one character at a time and prints out a formatted hex "dump" representation of the files contents 16 characters per display line. A prompt for continuation is issued after displaying 20 lines (320 characters of information). An entry of "X" or "x" followed by the key terminates the program execution. Any other entry followed by continues the display of the formatted hex and character information. A "." character is used for any non-displayable hex character

Hex Dumper by Sébastien Keim is a function which will display to stdout, the classic 3 column hex dump of a string passed to it.

Hexdump by Ned Batchelder prints a 3 column hex dump to stdout of a list of files or stdin.

What's been happening in the world of WSGI

Good to see some activity in the development of WSGI component based microframeworks.

I have been following Ben Bangert and James Gardners' work on Pylons which is based on Myghty, with a custom Resolver, full Paste and WSGI integration. Haven't had a play with it yet, but have an idea for a project where Pylons may be the perfect fit.

Also Julian Krause has just released RhubarbTart, a light object publishing web framework built on WSGI and Paste. It's object publishing model is similar to CherryPy. Since it built on Paste, RhubardTart makes the most of Paste's selection of WSGI middleware components to provide the other functionality expected of a web framework. So if have been using CherryPy because of it's ease in exposing object methods, but want to customise the component stack you use for a web project, have a look at RhubarbTart.

Then again, if you want to create your own WSGI/Paste framework, Ian Bickings has created a great tutorial to get you started. If you have a problem with understanding what WSGI/Paste is all about, read the tutorial, you will be enlightened.

And even Guido this week has nice things to say about WSGI - 'Maybe the current crop of Python web frameworks (as well as Rails BTW) have it all wrong. Maybe the WSGI folks are the only ones who are "getting" it.'

One of the joys of programming in Python is "batteries included" and the fact I get to pick which batteries to use to help me solve the programming task at hand. With WSGI I get the freedom to pick what components I want use in the HTTP request/response area for a web project, and get to focus on solving the actual programming problem or challenge.

The Year in Review

Tomorrow my blog is one year old. So thought it was time to review it's value. In the last year I have made 44 posts, not quite once a week based on the math. But truthfully, I appear to blog in bursts directly related to when I am doing open source stuff. And since any open source work can only be done in my free time, which is hard to find as I try to balance life between work and family, there are large gaps of time between these bursts. I am in awe of the open source developers that seem able to hold down a day job, produce heaps of amazing code, and also update their blogs most days. Have these people discovered a secret source of free time, or can survive with 2 hours sleep a day, or maybe they are younger than me :-)

In reviewing what I have blogged about, it is great to see that the focus in the WSGI world has moved from server gateway implementations to frameworks and components that use it. In the future, I expect to write more posts on using WSGI and Paste as I implement systems at work using them.

Having a blog has hopefully improved my writing skills. When I started a year ago, my first posts were based on my thoughts, saved as drafts, then tweaked until they became coherent, then published. Now I find myself able to get these thoughts in something readable in the first attempt most of the time. Also having a blog means, other people leave comments and links to their blogs. This has meant greater exposure to others thoughts and experiences.

So this blog is of great value to me and hopefully the odd thought I have posted has been of value to someone else.

Why is the PC running so slow?

At work we created a new VMWare image of the latest release of our Financial Software. Since it was required for a demo at our KL office, we needed to compress it prior to sending over the Internet to them. Since it was over 4GB, we used WinRAR to compress each virtual disk. It took forever and transferring via scp was just as bad. The cpu was the bottleneck, not because of the compression but the virus scanner scanning the VMWare image :-(. Morale of the story disable scanning for your VM image directories.

I guess a sign of times, were there are more background tasks running than "real" work being done in the foreground. And please, no comments about moving our work desktops to Linux or buying Mac's. Our server components are developed under Linux and run under most versions of Unix and Linux, but all our customers use M$ operating systems for the clientside. So we need to as well.

Merry Christmas

Heading off on computer free Christmas holidays tomorrow. Back to my homeland of New Zealand and looking forward to crayfish and beef on a spit for Christmas dinner. So just wanted to post a greeting now. Thanks to all the people who have read my blog, the members of the open source community who have helped me and have provided feedback. To those who celebrate Christmas, have a great one with family and friends.

Recharged after OSDC

After attending OSDC last week, I came back energised and wanting to get some more open source stuff done.

So apart from getting Roundup to work with WSGI, I have been busy integrating a newly released version of informixdb into the Zope Informix DA and SQLObject. Informix is our development database at work and for any *ix Python DBI access we have been using mxODCBC as the previous release of informixdb had suspect threading support. The new version can be compiled with or without thread support and uses distutils to do the compile. The use of distutils alone makes it worthwhile upgrading as anyone who has tried to get the old informixdb to link has a horror story to tell. It also supports the newer data types of IDS 9 and 10.

The Zope Informix DA work is done and is currently being tested prior to release.

The experience of creating an Informix data connection for SQLObject has been interesting. I was surprised to discover that SQLObject creates insert and update SQL statements as complete strings including the data values rather than using placeholders. This created a problem for me with TEXT BLOBS as the informixdb did not translate from a string but required a buffer. The various solutions I tried can be viewed here. As none of the solutions were really perfect, I decided to request an enhancement to informixdb so it would do the string to BLOB translation automatically and the Open Source community delivered yet again. In less than 24 hours, Carsten Haese had provided a patch that did exactly what I wanted.

It had been my intention to get all these changes tested and checked in prior to going on Christmas holidays but as usual my free time as been limited with other important things like family. So looks like it will have to wait until January.

OSDC 2005 - My WSGI Talk Errata

I have updated the copy of the paper to include more info on using paste.deploy pipelines. This info was presented at the conference but wasn't part of my submitted paper due to time constraints.

OSDC 2005 - My WSGI talk

Due to lack of an Internet connection during the conference (my issue not OSDC's), I have not had a chance to post anything about the conference until now. On Monday 5 December I gave my talk on WSGI. Not really sure how it went across as no one asked any questions at the end or really talked to me about it during the rest of the conference. Of course maybe it was so good, everyone now fully understands WSGI, but that is very unlikely. Anyway the paper can be found here and the only photographic evidence of my talk can be seen here.

In the closing keynote "Tools for Freedom", Jeff Waugh made an interesting observation about how a user of our software doesn't really give a damm about the technical aspects, they just want it to make their life easier. That got me thinking about how to improve the uptake of WSGI. As much as TurboGears can use WSGI, most developers will just run the standard CherryPY server. Maybe when they start thinking about production deployment, then WSGI may be used as the "making life easier" solution. So for the next couple of months I hope to focus on making some existing python apps WSGI enabled so the users of these apps will find deployment easier. So during breaks in the conference, I wrote an adaptor to run Roundup as a WSGI app. Fixed the last of the problems on the plane to Sydney this morning and once I have fully tested it and tried it at work for one of our existing Roundup tracker sites, I will look at checking it in to the Roundup CVS.

Not sure, about the next Python app, any suggestions?

Deploying ViewCVS with WSGI

In my continuing expermentation with Python Paste I came across the cgiapp module. The module provides a class that acts as a proxy to an existing CGI script, which means you can take an existing CGI script and run it using a WSGI enabled server. Rather than testing it's functionality with a simple cgi script, decided to see if I could run ViewCVS with it. To get it to run was very simple.

Create a app factory wrapper script called viewcvs_wsgi.py

from paste import cgiapp

def app_factory(global_config, **local_config):
 
 viewcvs_app = cgiapp.CGIApplication(global_config, script, path=None, include_os_environ=True, query_string=None)
 return viewcvs_app

Create a Paste Deploy configuration file called viewcvs.ini

[app:main]
paste.app_factory = viewcvs_wsgi:app_factory
script = "/usr/local/viewcvs-0.9.4/cgi/viewcvs.cgi"

[server:main]
use = egg:PasteScript#wsgiutils
host = 192.168.1.1
port = 8082

Start the WSGIUtils webserver using paster

paster server viewcvs.ini

Point a browser to the URL and browse the CVS repository.

Apart from images not displaying everything else worked as normal. Now why would I want to do this? Well it means I can now run viewcvs under IIS as an ISAPI extension using my isapi_wsgi adaptor. But thats a future blog entry.

SyPy Meetup Last Night

Last night was the last meeting of the year for the Sydney Python group. Andy Todd did a test run of his OSDC presentation on PythonCard. Apart from having to cope with using a Windows laptop instead of his beloved iBook, it was very interesting introduction in how to use PythonCard. I gave a presentation on Python Paste and hopefully, a few more know now what it is. The discussion on Paste.Deploy prompted Graham Dumpleton to give an impromptu presentation on an extension for mod_python he is creating called lamia that as one of it's features uses Python as it's configuration language.

Confirming that Python programmers are happy to look at other languages Alan Green gave us a look at Groovy, the new Java scripting language.

As with all the meetups this year, lots of other interesting conversations and looking forward to next year. Thanks to Alan Green for organising all this years meetings.

Deploying a WSGI app with Python Paste

As part of my research of the current state of WSGI for my paper for OSDC2005 I revisited PythonPaste. Ian has added lots more functionality since I last looked and I wanted to use Python Paste as the "glue" for a demo of creating a web app from WSGI middleware components. As with most stuff Ian does, there was enough documentation to get me going or so I thought. In reality, it took me a lot longer to get things working than I originally planned. This is no reflection on Ian's docs, but more on me not "getting it", and trying to make the solution complex. So what follows is a little tutorial on how to deploy a very simple WSGI app using Paste.Deploy and Paste.Script under Ubuntu 5.10 Linux and Python 2.4. Also this tutorial only uses a very small subset of Paste functionality.

0. If you have not installed any Python packages using easy_install before, you will need to download it and install it following the instructions at this site.

1. Install paste, paste.deploy and paste.script using easy install

sudo easy_install http://cheeseshop.python.org/packages/2.4/P/Paste/Paste-0.3-py2.4.egg
sudo easy_install http://cheeseshop.python.org/packages/2.4/P/PasteDeploy/PasteDeploy-0.3-py2.4.egg
sudo easy_install http://cheeseshop.python.org/packages/2.4/P/PasteScript/PasteScript-0.3-py2.4.egg

Also install WSGIUtils so we have a WSGI server to run our WSGI app under.

sudo easy_install http://pylons.groovie.org/files/WSGIUtils-0.6-py2.4.egg

2. Create a simple WSGI app in a file called hello_wsgi.py

def application(environ, start_response):
    """Simple WSGI application"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

def app_factory(global_config, **local_config):
    """This function wraps our simple WSGI app so it
    can be used with paste.deploy"""
    return application

3. Create a paste deploy configuration file called config.ini

[app:main]
paste.app_factory = hello_wsgi:app_factory

[server:main]
use = egg:PasteScript#wsgiutils
host = localhost
port = 80

The first section app:main is where we define our application, or more simply tell paste.deploy where the code for our WSGI app can be found, in our case in the file hello_wsgi.py and by calling function app_factory.

The second section server:main tells paste.script what WSGI server to use. In this case we will be using the simple WSGI server that comes with WSGIUtils.

4. To serve the app we use the paster script that was installed as part of Paste.Script. So enter the following at the commandline.

export PYTHONPATH=.   # Ensure our hello_wsgi.py module can be imported by Python
sudo paster server ./config.ini

On the console, a message will be displayed similar to:

Starting server in PID 9582.

5. Enter the URL for the web app in a browser and hopefully you see hello world displayed.

6. Replace the boring hello world app with a more useful WSGI app you are now inspired to create.

OSDC Proposal Outline

It would appear that you need a login to view my proposal. So here is a copy.

WSGI - Gateway or Glue?

Short Description: Present a review of where WSGI currently is and describe/show how it is being used to create a framework-neutral set of components.

Stream: Python

30 minute Paper Presentation in English


Abstract
The Web Service Gateway Interface is an API defined in PEP 333 that allows web servers to talk to Python web applications. At the time of OSDC 2005, the PEP will be 2 years old and the presentation will review the state of WSGI implementations for the various web servers including Apache, IIS, Twisted etc. It will also show how the API is used to implement a web server interface and create an application. In the last 6 months, the main WSGI community focus has been on creating web applications by composing them from components that communicate via WSGI. The remainder of the presentation will discuss this and show what can be done.

OSDC Proposal Accepted

"Your proposal has now been reviewed, and I am pleased to be able to inform you that it has been accepted for inclusion in the conference program" so it is now time for me to gather together all the ideas that have flowing around in my head and create a coherent paper from them. The proposal description can be found at http://osdc2005.cgpublisher.com/proposals/24

After looking at the other accepted proposals, I know this is going to be an even better OSDC than last year. Also as mentioned in a previous post there is a much better balance in the "P" scripting languages than last year ;-)

So if you want to spend an interesting 3 days in Melbourne in early December, why not goto the Open Source Developers Conference