Merry Christmas

Heading off on computer free Christmas holidays tomorrow. Back to my homeland of New Zealand and looking forward to crayfish and beef on a spit for Christmas dinner. So just wanted to post a greeting now. Thanks to all the people who have read my blog, the members of the open source community who have helped me and have provided feedback. To those who celebrate Christmas, have a great one with family and friends.

Recharged after OSDC

After attending OSDC last week, I came back energised and wanting to get some more open source stuff done.

So apart from getting Roundup to work with WSGI, I have been busy integrating a newly released version of informixdb into the Zope Informix DA and SQLObject. Informix is our development database at work and for any *ix Python DBI access we have been using mxODCBC as the previous release of informixdb had suspect threading support. The new version can be compiled with or without thread support and uses distutils to do the compile. The use of distutils alone makes it worthwhile upgrading as anyone who has tried to get the old informixdb to link has a horror story to tell. It also supports the newer data types of IDS 9 and 10.

The Zope Informix DA work is done and is currently being tested prior to release.

The experience of creating an Informix data connection for SQLObject has been interesting. I was surprised to discover that SQLObject creates insert and update SQL statements as complete strings including the data values rather than using placeholders. This created a problem for me with TEXT BLOBS as the informixdb did not translate from a string but required a buffer. The various solutions I tried can be viewed here. As none of the solutions were really perfect, I decided to request an enhancement to informixdb so it would do the string to BLOB translation automatically and the Open Source community delivered yet again. In less than 24 hours, Carsten Haese had provided a patch that did exactly what I wanted.

It had been my intention to get all these changes tested and checked in prior to going on Christmas holidays but as usual my free time as been limited with other important things like family. So looks like it will have to wait until January.

OSDC 2005 - My WSGI Talk Errata

I have updated the copy of the paper to include more info on using paste.deploy pipelines. This info was presented at the conference but wasn't part of my submitted paper due to time constraints.

OSDC 2005 - My WSGI talk

Due to lack of an Internet connection during the conference (my issue not OSDC's), I have not had a chance to post anything about the conference until now. On Monday 5 December I gave my talk on WSGI. Not really sure how it went across as no one asked any questions at the end or really talked to me about it during the rest of the conference. Of course maybe it was so good, everyone now fully understands WSGI, but that is very unlikely. Anyway the paper can be found here and the only photographic evidence of my talk can be seen here.

In the closing keynote "Tools for Freedom", Jeff Waugh made an interesting observation about how a user of our software doesn't really give a damm about the technical aspects, they just want it to make their life easier. That got me thinking about how to improve the uptake of WSGI. As much as TurboGears can use WSGI, most developers will just run the standard CherryPY server. Maybe when they start thinking about production deployment, then WSGI may be used as the "making life easier" solution. So for the next couple of months I hope to focus on making some existing python apps WSGI enabled so the users of these apps will find deployment easier. So during breaks in the conference, I wrote an adaptor to run Roundup as a WSGI app. Fixed the last of the problems on the plane to Sydney this morning and once I have fully tested it and tried it at work for one of our existing Roundup tracker sites, I will look at checking it in to the Roundup CVS.

Not sure, about the next Python app, any suggestions?

Deploying ViewCVS with WSGI

In my continuing expermentation with Python Paste I came across the cgiapp module. The module provides a class that acts as a proxy to an existing CGI script, which means you can take an existing CGI script and run it using a WSGI enabled server. Rather than testing it's functionality with a simple cgi script, decided to see if I could run ViewCVS with it. To get it to run was very simple.

Create a app factory wrapper script called viewcvs_wsgi.py

from paste import cgiapp

def app_factory(global_config, **local_config):
 
 viewcvs_app = cgiapp.CGIApplication(global_config, script, path=None, include_os_environ=True, query_string=None)
 return viewcvs_app

Create a Paste Deploy configuration file called viewcvs.ini

[app:main]
paste.app_factory = viewcvs_wsgi:app_factory
script = "/usr/local/viewcvs-0.9.4/cgi/viewcvs.cgi"

[server:main]
use = egg:PasteScript#wsgiutils
host = 192.168.1.1
port = 8082

Start the WSGIUtils webserver using paster

paster server viewcvs.ini

Point a browser to the URL and browse the CVS repository.

Apart from images not displaying everything else worked as normal. Now why would I want to do this? Well it means I can now run viewcvs under IIS as an ISAPI extension using my isapi_wsgi adaptor. But thats a future blog entry.

SyPy Meetup Last Night

Last night was the last meeting of the year for the Sydney Python group. Andy Todd did a test run of his OSDC presentation on PythonCard. Apart from having to cope with using a Windows laptop instead of his beloved iBook, it was very interesting introduction in how to use PythonCard. I gave a presentation on Python Paste and hopefully, a few more know now what it is. The discussion on Paste.Deploy prompted Graham Dumpleton to give an impromptu presentation on an extension for mod_python he is creating called lamia that as one of it's features uses Python as it's configuration language.

Confirming that Python programmers are happy to look at other languages Alan Green gave us a look at Groovy, the new Java scripting language.

As with all the meetups this year, lots of other interesting conversations and looking forward to next year. Thanks to Alan Green for organising all this years meetings.

Deploying a WSGI app with Python Paste

As part of my research of the current state of WSGI for my paper for OSDC2005 I revisited PythonPaste. Ian has added lots more functionality since I last looked and I wanted to use Python Paste as the "glue" for a demo of creating a web app from WSGI middleware components. As with most stuff Ian does, there was enough documentation to get me going or so I thought. In reality, it took me a lot longer to get things working than I originally planned. This is no reflection on Ian's docs, but more on me not "getting it", and trying to make the solution complex. So what follows is a little tutorial on how to deploy a very simple WSGI app using Paste.Deploy and Paste.Script under Ubuntu 5.10 Linux and Python 2.4. Also this tutorial only uses a very small subset of Paste functionality.

0. If you have not installed any Python packages using easy_install before, you will need to download it and install it following the instructions at this site.

1. Install paste, paste.deploy and paste.script using easy install

sudo easy_install http://cheeseshop.python.org/packages/2.4/P/Paste/Paste-0.3-py2.4.egg
sudo easy_install http://cheeseshop.python.org/packages/2.4/P/PasteDeploy/PasteDeploy-0.3-py2.4.egg
sudo easy_install http://cheeseshop.python.org/packages/2.4/P/PasteScript/PasteScript-0.3-py2.4.egg

Also install WSGIUtils so we have a WSGI server to run our WSGI app under.

sudo easy_install http://pylons.groovie.org/files/WSGIUtils-0.6-py2.4.egg

2. Create a simple WSGI app in a file called hello_wsgi.py

def application(environ, start_response):
    """Simple WSGI application"""
    status = '200 OK'
    response_headers = [('Content-type','text/plain')]
    start_response(status, response_headers)
    return ['Hello world!\n']

def app_factory(global_config, **local_config):
    """This function wraps our simple WSGI app so it
    can be used with paste.deploy"""
    return application

3. Create a paste deploy configuration file called config.ini

[app:main]
paste.app_factory = hello_wsgi:app_factory

[server:main]
use = egg:PasteScript#wsgiutils
host = localhost
port = 80

The first section app:main is where we define our application, or more simply tell paste.deploy where the code for our WSGI app can be found, in our case in the file hello_wsgi.py and by calling function app_factory.

The second section server:main tells paste.script what WSGI server to use. In this case we will be using the simple WSGI server that comes with WSGIUtils.

4. To serve the app we use the paster script that was installed as part of Paste.Script. So enter the following at the commandline.

export PYTHONPATH=.   # Ensure our hello_wsgi.py module can be imported by Python
sudo paster server ./config.ini

On the console, a message will be displayed similar to:

Starting server in PID 9582.

5. Enter the URL for the web app in a browser and hopefully you see hello world displayed.

6. Replace the boring hello world app with a more useful WSGI app you are now inspired to create.

OSDC Proposal Outline

It would appear that you need a login to view my proposal. So here is a copy.

WSGI - Gateway or Glue?

Short Description: Present a review of where WSGI currently is and describe/show how it is being used to create a framework-neutral set of components.

Stream: Python

30 minute Paper Presentation in English


Abstract
The Web Service Gateway Interface is an API defined in PEP 333 that allows web servers to talk to Python web applications. At the time of OSDC 2005, the PEP will be 2 years old and the presentation will review the state of WSGI implementations for the various web servers including Apache, IIS, Twisted etc. It will also show how the API is used to implement a web server interface and create an application. In the last 6 months, the main WSGI community focus has been on creating web applications by composing them from components that communicate via WSGI. The remainder of the presentation will discuss this and show what can be done.

OSDC Proposal Accepted

"Your proposal has now been reviewed, and I am pleased to be able to inform you that it has been accepted for inclusion in the conference program" so it is now time for me to gather together all the ideas that have flowing around in my head and create a coherent paper from them. The proposal description can be found at http://osdc2005.cgpublisher.com/proposals/24

After looking at the other accepted proposals, I know this is going to be an even better OSDC than last year. Also as mentioned in a previous post there is a much better balance in the "P" scripting languages than last year ;-)

So if you want to spend an interesting 3 days in Melbourne in early December, why not goto the Open Source Developers Conference

Argh, having to learn java again

After managing to keep myself employed for almost the last decade without having to program in Java, a project at work means that the end is nigh. Pulled the "Teach yourself Java in 21 days" out of storage and noted that it was published in 1996. So Java and I have kept our distance for a long time. Also means I need to get myself up to speed with what has happened to Java over the period. In the end, I just need to learn enough to embed Jython 2.1 within an existing Java front-end so I can access it's API, well that's the theory anyway. Have spent the weekend consuming lot's of articles from the web and playing in eclipse. Have lot's of respect for what "real" Java programmers are doing out there, but hard to get back into writing lot's of code to achieve something. Python has spoilt me.

OSDC Proposal Submitted

Due to the "massive" response from my readers (thanks Richard) I went with WSGI and submitted a proposal. As of last night, it looks like they have 55 proposals of a wide and interesting range of topics. It would also appear there are more Python than Perl proposals, but I had my rose coloured glasses on, so I may be mistaken.

Goodbye Windows, Welcome Linux to my home

At work I use Windows XP as my workstation because that's what our clients use to run our frontend. Most of our development work is done on servers running Linux and truthfully the most used program on my XP box is ssh. But at home, I have always run a Microsoft OS. Of late, I have been questioning why. The home machine is used for paying the bills via the Internet, reading email, browsing the web, updating this blog and the odd bit of wordprocessing. In the past, it was also used to play games, but the XBox has taken over that role. Based on the last couple of months, the home machine seems to spend most of it's time downloading service packs or the lastest anti-virius update. So I decided to give Linux a chance to prove that it could do my home tasks. Using a "Live" CD version of Ubuntu 5.04, I have survived two weeks without having to see a Microsoft logo. Another interesting thing I have discovered is thanks to GMail etc, that I haven't needed any local hard disk storage.

Database design for newbies

Found this whitepaper on database design by Ian Bicking & Brian Moloney. A good explanation of Dr Codd's rules but in a language a newbie could understand.

How fast a year goes by, OSDC decisions again...

It doesn't seem that long ago that I was considering presenting a paper at OSDC 2004. But I did, and here we are a year later in the same situation. I really enjoyed last year's conference, both giving my presentation, meeting many interesting people, and deep discussions over many beers each night. I want to submit a proposal for a paper but still haven't decided what I want to talk about. Currently I have narrowed it down to 3 topics:

• WSGI - Present a review of where it currently is and describe/show how it is being used to create a framework-neutral set of components.
  
• A Smart Client using Python - Microsoft is pushing 2 types of user interface for an n-tier application, a web client and what they refer to as a "smart" client. In M$'s case, it is written using WinForms and other .NET technologies. I want to produce something similar using Python. Have been playing with both wxPython & PyXPCOM/XULRunner before deciding on the UI toolkit to use.
  
• ActiveGrid - This is an open source framework for building Web 2.0 using LAMP. Think it's more a candiate for a lighting talk.

Will have to make up my mind soon, as proposals have to be in by Friday 19 August.

Creating an ODBC data source using Python

For the cross-platform installer I am creating, I need to create ODBC data sources for Windows installations. Thanks to the Python ctypes module the solution was as simple as:

import ctypes

ODBC_ADD_DSN = 1        # Add data source
ODBC_CONFIG_DSN = 2     # Configure (edit) data source
ODBC_REMOVE_DSN = 3     # Remove data source
ODBC_ADD_SYS_DSN = 4    # add a system DSN
ODBC_CONFIG_SYS_DSN = 5 # Configure a system DSN
ODBC_REMOVE_SYS_DSN = 6 # remove a system DSN

def create_sys_dsn(driver, **kw):
    """Create a  system DSN
    Parameters:
        driver - ODBC driver name
        kw - Driver attributes
    Returns:
        0 - DSN not created
        1 - DSN created
    """
    nul = chr(0)
    attributes = []
    for attr in kw.keys():
        attributes.append("%s=%s" % (attr, kw[attr]))
    
    return ctypes.windll.ODBCCP32.SQLConfigDataSource(0, ODBC_ADD_SYS_DSN, driver, nul.join(attributes))

if __name__ == "__main__":
    print create_sys_dsn("SQL Server",SERVER="(local)", DESCRIPTION="SQL Server DSN", DSN="SQL SERVER DSN", Database="mydatabase", Trusted_Connection="Yes")
    print create_sys_dsn("mySQL",SERVER="local", DESCRIPTION="mySQL Server Test1", DSN="mySQL DSN", DATABASE="mySQLDb", UID="username", PASSWORD="password", PORT="3306", OPTION="3")

Require a cross-platform installer, Python to the rescue

The company I work for develops an ERP system that has it's heritage in implementations for medium size companies running on UNIX systems. As the global ERP vendors have moved in our space, we have had to move into the SME market which means running on Linux or Windows server systems. And since some of our new clients cannot afford the cost of a technical resource going on-site to install and configure the technical infrastructure, we need a cross-platform installer. As our application is not written in Java, we are not interested in a Java based installer. But since our supporting scripts are written in Python, and I have had some success creating a cross-platform installer in Python to install and build the the required 3rd party packages for Python Paste, I decided Python could provide a solution. Currently a work in progress, I have a Python script that installs our complete ERP app, it's supporting runtimes and configures it to run in under 10 minutes. Still need to get database setup working, and as our app can run under Oracle, IBM Informix or MS SQL Server 2000 there is abit of work to be done.

Of course to make it look like a "real" installer under Windows I use Inno Setup, and call an executable version of the python script created by py2exe. Rather try trying to write in Delphi to get custom forms for the installor I used ctypes and the Windows port of EasyDialogs to collect site specific setup data. So thanks to the hard work of the creators of the various tools I have used and Python, it looks like we will have our cross-platform installer.

build-pk.py now part of Python Paste

Ian has given me check-in access to the PythonPaste SVN, so I have checked in my cross-platform build-pkg.py script. Also looking at making a distribution of the 3rd party packages using Python Eggs.

Installing WSGIKit, opps I mean Python Paste under Win32

Ian has been unhappy with the name WSGIKit, so it has been renamed Python Paste I think the name better matches what the package does, the WSGI glue that sticks all the python components together. Anyway have made some minor changes to my cross-platform build-pkg.py script so that it works more like Ian's shell script. Have sent a copy to him, so who knows, maybe it will be part of Python Paste.

                                                                               
#!/usr/bin/env python
"""
Install required third party packages for Python Paste.

Known limitations

 - Running under Windows, it must be run with Python 2.3+
   as it requires the tarfile module. On Unix/Linix platform
   will fallback to tar program.
"""
import os
import sys
import errno
import urllib
import shutil

TMP="/tmp"
for tmp in ['TEMP','TMPDIR','TMP']:
    if os.environ.has_key(tmp):
        TMP = os.environ[tmp]
        break
TMP = TMP + os.sep + "tmp-build"

BASE=os.path.dirname(sys.argv[0])
if BASE == '' or BASE == ".":
    BASE = os.getcwd()

THIRD = BASE + "/paste/3rd-party"

def basename(filename, ext):
    try:
        i = filename.index(ext)
        base = filename[0:i]
    except ValueError:
        base = filename
        pass

    return base

def mkdirs(dirname, mode=0777):
    # Python Cookbook Edition 1 Recipe 4.17
    try:
        os.makedirs(dirname)
    except OSError, err:
            if err.errno != errno.EEXIST or not os.path.isdir(TMP):
                raise

def delete_pyc(arg, dirname, files):
    """
    Called by os.path.walk to delete .pyc and .pyo files
    """
    for name in files:
        if name.endswith(".pyc") or name.endswith(".pyo"):
            os.remove(os.path.join(dirname, name))

def get_file(zip_type, filename, url):
    """
    If required, download requested third party package and extract.
    On exit, the script is in the extracted package directory.
    """
    mkdirs(TMP)
    os.chdir(TMP)
    if not os.path.exists(filename):
        print "Download %s ..." % filename
        urllib.urlretrieve(url, filename)
    DIR=basename(filename, ".tar.gz")
    DIR=basename(DIR, ".tar.bz2")
    DIR=basename(DIR, ".tgz")
    if not os.path.exists(DIR):
        try:
            import tarfile
            tar = tarfile.open(filename, "r:" + zip_type)
            for file in tar.getnames():
                tar.extract(file)
        except ImportError:
            # No tarfile module, so use actual tar program
            if zip_type == "gz":
                zip_type = "z"
            else:
                zip_type = "j"
            os.system('tar fx%s "%s"' % (zip_type, filename))

    try:
        os.chdir(DIR)
    except OSError:
        pass

def installer(name):
    mkdirs(THIRD + "/" + name +"-files")
    cmd = '%s setup.py install -f --install-lib="%s/%s-files" --install-scripts="%s/%s-files/scripts" --no-compile' % (sys.executable, THIRD, name, THIRD, name)
    print cmd
    os.system(cmd)

get_file("gz","WSGI Utils-0.5.tar.gz",
        "http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz")
installer("wsgiutils")

get_file("gz", "Component-0.1.tar.gz", 
        "http://webwareforpython.org/downloads/Component-0.1.tar.gz")
DEST = THIRD + "/Component-files/Component"
mkdirs(DEST)
shutil.rmtree(DEST,ignore_errors=1)
os.chdir(THIRD)
shutil.copytree(TMP + os.sep +"Component-0.1", DEST)

zptkit_tmpdir = TMP + "/" + "ZPTKit"
if os.path.exists(zptkit_tmpdir):
    os.chdir(zptkit_tmpdir)
    cmd ="svn up"
else:
    cmd ="svn co http://svn.w4py.org/ZPTKit/trunk %s/ZPTKit" % TMP
os.system(cmd)
DEST = THIRD + "/ZPTKit-files/ZPTKit"
mkdirs(DEST)
shutil.rmtree(DEST,ignore_errors=1)
os.chdir(THIRD)
shutil.copytree(TMP + "/ZPTKit", DEST)

get_file("gz", "ZopePageTemplates-1.4.0.tar.gz", 
        "http://belnet.dl.sourceforge.net/sourceforge/zpt/ZopePageTemplates-1.4.0.tgz")
os.chdir("ZopePageTemplates")
installer("ZopePageTemplates")

get_file("gz", "scgi-1.2.tar.gz", 
        "http://www.mems-exchange.org/software/files/scgi/scgi-1.2.tar.gz")
installer("scgi")

# Do not clean up compiled python files for now
#os.path.walk(THIRD,delete_pyc,None)

sqlobject_tmpdir = TMP + "/" + "SQLObject"
if os.path.exists(sqlobject_tmpdir):
    os.chdir(sqlobject_tmpdir)
    cmd ="svn up"
else:
    cmd ="svn co http://svn.colorstudy.com/trunk/SQLObject %s/SQLObject" % TMP
os.system(cmd)
os.chdir(sqlobject_tmpdir)
installer("sqlobject")
mkdirs(THIRD + "/sqlobject-files/scripts")
shutil.copy(TMP + "/SQLObject/scripts/sqlobject-admin", 
           THIRD + "/sqlobject-files/scripts")


if not os.path.exists(TMP + "/PySourceColor.py"):
    urllib.urlretrieve("http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py", 
           TMP + "/PySourceColor.py")
mkdirs(THIRD + "/PySourceColor-files")
shutil.copy(TMP + "/PySourceColor.py", THIRD + "/PySourceColor-files")

Installing WSGIKit on a Win32 platform

So I could have a play with WSGIKit and look at how I could make ISAPI_WSGI work with it, I decided to install it on my home Windows XP box. As part of the svn checkout, there is a shell script called build-pkg which will download the required third party packages and install them under WSGIKit for you. Tried to run the script with a standard install of Cygwin but it failed due to various errors. I could have debugged my way thru the errors but thought why not create a truly cross-platform version of buld-pkg in python. So an hour later, the following script did the job for me. It currently lacks the checks to see if the package has already been downloaded, but it did the job for me.

import os
import sys
import errno
import tarfile
import urllib
import shutil

TMP="/tmp/tmp-build"
BASE=os.path.dirname(sys.argv[0])

if BASE == "" or BASE == ".":
    BASE = os.getcwd()

THIRD = BASE + "/wsgikit/3rd-party"

def basename(filename, ext):
    try:
        i = filename.index(ext)
        base = filename[0:i]
    except ValueError:
        base = filename
        pass

    return base

def mkdirs(dirname, mode=0777):
    # Python Cookbook Recipe edition 1 4.17
    try:
        os.makedirs(dirname)
    except OSError, err:
            if err.errno != errno.EEXIST or not os.path.isdir(TMP):
                raise

def get_file(zip_type, filename, url):
    print "Getting %s ..." % filename
    mkdirs(TMP)
    os.chdir(TMP)
    # TODO: check for existing package, download if not there
    urllib.urlretrieve(url, filename)
    DIR=basename(filename, ".tar.gz")
    DIR=basename(DIR, ".tar.bz2")
    DIR=basename(DIR, ".tgz")
    tar = tarfile.open(filename, "r:" + zip_type)
    for file in tar.getnames():
        tar.extract(file)
    try:
        os.chdir(DIR)
    except OSError:
        pass

def installer(name):
    mkdirs(THIRD + "/" + name +"-files")
    cmd = 'python setup.py install -f --install-lib="%s/%s-files" --install-scripts="%s/%s-files/scripts" --no-compile' % (THIRD, name, THIRD, name)
    os.system(cmd)

get_file("gz","WSGI Utils-0.5.tar.gz",
        "http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz")
installer("wsgiutils")

get_file("gz", "Component-0.1.tar.gz", 
        "http://webwareforpython.org/downloads/Component-0.1.tar.gz")
mkdirs(THIRD + "/Component-files")
DEST = THIRD + "/Component-files/Component"
shutil.rmtree(DEST,ignore_errors=1)
try:
    shutil.move(TMP + "/Component-0.1", DEST)
except OSError:
    pass

get_file("gz", "ZPTKit-0.1.tar.gz", 
        "http://imagescape.com/software/ZPTKit/ZPTKit-0.1.tar.gz")
mkdirs(THIRD + "/ZPTKit-files")
DEST = THIRD + "/ZPTKit-files/ZPTKit"
shutil.rmtree(DEST,ignore_errors=1)
try:
    shutil.move(TMP + "/ZPTKit-0.1", DEST)
except OSError:
    # To handle permission errors under Windows
    pass

get_file("gz", "ZopePageTemplates-1.4.0.tar.gz", 
        "http://belnet.dl.sourceforge.net/sourceforge/zpt/ZopePageTemplates-1.4.0.tgz")
os.chdir("ZopePageTemplates")
installer("ZopePageTemplates")

get_file("gz", "scgi-1.2.tar.gz", 
        "http://www.mems-exchange.org/software/files/scgi/scgi-1.2.tar.gz")
installer("scgi")

cmd ="svn co http://svn.colorstudy.com/trunk/SQLObject %s/SQLObject" % TMP
os.system(cmd)
os.chdir(TMP + "/SQLObject")
installer("sqlobject")
mkdirs(THIRD + "/sqlobject-files/scripts")
shutil.copy(TMP + "/SQLObject/scripts/sqlobject-admin", 
           THIRD + "/sqlobject-files/scripts")


urllib.urlretrieve("http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py", 
                TMP + "/PySourceColor.py")
mkdirs(THIRD + "/PySourceColor-files")
shutil.copy(TMP + "/PySourceColor.py", THIRD + "/PySourceColor-files")

Back to WSGI land

It's been over a month since my last entry as family and work have consumed most of my time. Some interesting developments for WSGI over the last month. Ian Bickings PyCon presentation provides a very good description of what WSGI is and what it is not. He used his WSGIKit for the usage examples. And just in case, we didn't appreciate what you can do with WSGIKit, a week later he provided an online tutorial on how to implement something like the famous Ruby on Rails ToDo List example with it. The WSGIKit tutorial can be found here.

Also came across some work that Allan Saddi has done. At last I can simply use FastCGI with my WSGI enabled web scripts. So using either isapi_wsgi or his fcgi.py I can run my scripts as long running processes on both IIS and Apache with no real extra effort.

ZopeX3 learning from others.

As I said in a previous post, the documentation for ZopeX3 is much better than what came with Zope 1. I found the programmers tutorial a very good starting point. Also some googling found this on-going series of blog entries creating a Todo. Nothing like learning about something by example.

Zope Deja vu

It's 1999, I have downloaded Zope 1, and I managed to create a simple website that gets it's data using SQLMethods from a database. I had to learn a new markup language called dtml to get my dynamic webpages. It has no session management but not an issue at the time. To enhance it's functionality I need to create a product, that's were it starts to get hard. The documentation is minimal. But I stick with it, and six years later Zope 2 is still the Web Application Framework at work.

It's today, I have downloaded ZopeX3, and I managed to create a simple website that gets it's data using SQLMethods from a database. I had to learn a new markup language called zcml to get my dynamic webpages. It has no session management but not an issue at the time. To enhance it's functionality I need to create a product, that's were it starts to get hard.

The documentation is much better, but I seem to spend more time editting xml than actually writing code in the python. At least the coding seems more like normal python programming, and the use of components, interfaces, adaptors and testing looks promising and maybe Zope will once again lead the way as the Python Web Application Framework. The bigger question for me is, should we be considering it as a replacement for Zope 2 at work or wait and have a look in another 3 months. Or maybe we should look at Five, the Zope 3 in Zope 2 project. Just like I did with Zope 1, I will stick with it and keep playing, and hopefully it will deliver like Zope 1 and 2 did.

Python and Beer, it must be a Python Meetup Group

Last thursday was the bi-monthly Sydney Python Meetup. Excellent beer, interesting talks and people. Thanks for organising the meeting Alan.

What's all the excitement about XMLHttpRequest

The buzz on many blogs is Google's use of XMLHttpRequest to allow the browser to do requests of the backend without a page refresh. To the best of my knowledge XMLHttpRequest or similar functionality has been part of javascript support in IE and Mozilla for a couple of years. We have been using it for an application at work for the last 20 months to make XMLRPC calls to dynamically update the browser and a SVG view. The biggest issue in the past is getting it to work across browsers, and rather than writing javascript with lots of "if (browser == 'X')" code I would recommend taking at look at the joslait javascript library. As well as providing a set of routines which allow XMLHttpRequest functionality on many browser versions and a XMLRPC client, it has a number of other useful routines.

WSGI Wiki Application

For my WSGI talk at the Sydney Python Meetup I decided that showing a simple "hello world" or echo WSGI application would not hammer home why WSGI is so good. So I decided to create a WSGI wiki app. Also to show what is needed to convert an existing Python CGI script I decided to base it on Ian Bicking's CGI wiki in the Web Framework Shootout. So after about 80 minutes work this is what I came up with and was able to run under isapi_wsgi. It still needs some final touches but it works.

import cgi, os
import cgitb; cgitb.enable()
import Wiki


# There's several different actions that may occur -- we
# may just display the page, we may edit it, save the edits
# that were submitted, or show a preview.  We define each of
# these as a function:

def printPage(page):
    pp = []
    if not page.exists():
        pp.append("This page does not yet exist.  Create it:")
        pp.append(printEdit(page))
    else:
        pp.append(page.html)
        pp.append('<hr noshade>')
        pp.append('<a href="%s?edit=yes">Edit this page</a>'               % page.name)
    return "".join(pp)

def printEdit(page, text=None):
    # The text argument is used in previews -- you want the
    # textarea to be loaded with the same text they submitted.
    pe = []
    if text is None:
        # This isn't a preview, so we get the page's text.
        text = page.text
    pe.append('<h1>Edit %s</h1>' % page.title)
    pe.append('<form action="%s" method="POST">' % page.name)
    pe.append('<textarea name="text" rows=10 cols=50 style="width: 90%%">'           '%s</textarea><br>' % cgi.escape(text))
    pe.append('<input type="submit" name="save" value="Save">')
    pe.append('<input type="submit" name="preview" value="Preview">')
    pe.append('</form>')
    return "".join(pe)

def printSave(page, form):
    ps = []
    page.text = form['text'].value
    ps.append(printPage(page))
    return "".join(ps)

def printPreview(page, form):
    ps = []
    ps.append('<h1>Preview %s</h1>' % page.title)
    ps.append(page.preview(form['text'].value))
    ps.append('<hr noshade>')
    ps.append(printEdit(page, form['text'].value))
    return "".join(ps)

def application(environ, start_response):
    # PATH_INFO contains the portion of the URL that comes after
    # script name.  For all actions, this is the page we are using.
    pagename = environ['PATH_INFO'][1:]
    if len(pagename) == 0:
        pagename = "frontpage"
    page = Wiki.WikiPage(pagename)
    form = cgi.FieldStorage(fp=environ['wsgi.input'],environ=environ)
    start_response('200 OK', [('Content-type','text/html')])
    response = []
    response.append("<html><head><title>%s</title>" % page.title)
    response.append(page.css)
    response.append("</head><body>")
    # Here we use fields to determine what action we should take:
    # edit, save, preview, or display
    if form.has_key('edit'):
        response.append(printEdit(page))
    elif form.has_key('save'):
        response.append(printSave(page, form))
    elif form.has_key('preview'):
        response.append(printPreview(page, form))
    else:
        response.append("<h1>%s</h1>" % page.title)
        response.append(printPage(page))
    response.append("</body></html>")
    return response

dotNet client testing of python web services

As I said in a previous post I have been testing .NET client calls of our financial app web services. I found a good "free" tool called WebServiceStudio that allows quick and easy interactive testing of web services. Our web services use the Python ZSI web service package and this seems to work fine with dotNET clients, as long as you keep the parameter and return types simple. Otherwise you will spend alot of time handcrafting WSDL and for that you need a good WSDL editor.

As if I haven't learnt enough programming languages..

As much as I enjoy programming in Python and feel it is the closet fit for how my brain works, I still have the need to checkout other languages.

After listening to Dr Chris Wright at OSDC last year, I made a mental note to checkout Scheme and also learn about continuations. After finding some tutorials on the web, I started playing with PyScheme It was great for trying out simple examples and made me interested enough for something to try out more complex things. I found Dr Scheme which is an interactive, integrated, graphical programming environment for the Scheme, MzScheme, and MrEd programming languages. It is multi-platform running on Windows (95 and up), Mac OS X, and Unix/X. It has great interactive syntax checking which is very helpful when debugging complex scheme code. There is even talk of a plug-in that will allow you to use the tool to program in Python called PLT Spy

At work I needed to check that some web services for our financial app worked when called from .NET As we currently only support RPC style, writing WS clients in C# is not much fun. I really needed an agile programming language. I could have used PythonNet but really wanted to try out a CLI language. As IronPython seems to be going nowhere, I decided to try out Boo Since some of the syntax is based on Python, I feel at home using it. And based on one days use, it makes working in .NET fun.

ISAPI-WSGI - Runs a WSGIUtils application

As part of researching for my WSGI talk I have been working with Colin Stewart's WSGIUtils. It is a package of wsgi helper libraries that should make the development of wsgi enabled apps easier. The two main components are:

wsgiServer is a multi-threaded WSGI web server based on SimpleHTTPServer.

wsgiAdaptor is a simple WSGI application that provides a Basic authentication, signed cookies and persistent sessions.

After getting his examples working under wsgiServer, I decided to see if the wsgiAdaptor examples worked under isapi_wsgi. Apart from needing to make a change to where the session dbm is created (due to permission issues) and an issue with how my copy of IE 6.0 handles cookies :-(, it worked very well. Below is how to do it:

""" Basic Example

  Copyright (c) 2004 Colin Stewart (http://www.owlfish.com/)
  All rights reserved.
    
  THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  
  If you make any bug fixes or feature enhancements please let me know!
  
  A simple example - a counting web site.
  Note that increments in modern browsers go up twice because two
  requests are made - one for '/' and one for '/favicon.ico'
"""
import logging, time
import isapi_wsgi

from wsgiutils import SessionClient, wsgiAdaptor, wsgiServer
class TestApp:
    def requestHandler (self, request):
        session = request.getSession()
        count = session.get ('counter', 0)
        count += 1
        session ['counter'] = count
        request.sendContent ("<html><body><h1>Visits: %s</h1><body></html>" % str (count), "text/html")
  
 
def __ExtensionFactory__():
    return isapi_wsgi.ISAPISimpleHandler(test = testadaptor.wsgiHook)

testclient = SessionClient.LocalSessionClient('T:/session.dbm', 'testappid')
testadaptor = wsgiAdaptor.wsgiAdaptor (TestApp(), 'siteCookieKey', testclient)

if __name__=='__main__':
    # If run from the command-line, install ourselves.
    from isapi.install import *
    params = ISAPIParameters()
    # Setup the virtual directories - this is a list of directories our
    # extension uses - in this case only 1.
    # Each extension has a "script map" - this is the mapping of ISAPI
    # extensions.
    sm = [
        ScriptMapParams(Extension="*", Flags=0)
    ]
    vd = VirtualDirParameters(Name="isapi-wsgi-wsgiutils",
                              Description = "ISAPI-WSGI WSGIUtils Test",
                              ScriptMaps = sm,
                              ScriptMapUpdate = "replace"
                              )
    params.VirtualDirs = [vd]
    HandleCommandLine(params)

Looking at other WSGI implementations

Life has been very busy on the family front. My son and I currently are sharing the house with 4 woman. Not the best environment for software development :-), so haven't progressed very far with the threaded version of isapi_wsgi.

Looks like I will be giving a short presentation on WSGI at the next Sydney Python Meetup, so I have been looking at other WSGI implementations and adaptors. To date my presentation demos have run with a number of them, so the promise of WSGI looks like reality.

ISAPI-WSGI - Runs a CherryPy2 application

In my ongoing testing of ISAPI-WSGI, with a Subversion checkout of CherryPy2 I have some of the tutorials running as an ISAPI extension. Only issue discovered relates to how CherryPy handles an empty PATH_INFO but I am sure this will be resolved soon. For the time being the url to access the example must have a trailing backslash. This has now been resolved. Thanks to Remco and Remi for helping me with it.

So now that I have proved CherryPy2 can work with ISAPI-WSGI, Subway should be able to run as an ISAPI extension.

I will add the example to ISAPI-WSGI subversion repository tomorrow. Below is how to get tutorial 3 working:

# Demo of tutorial 3 running cherrypy2 under isapi-wsgi
#
# Executing this script (or any server config script) will install the extension
# into your web server and will create a "loader" DLL _cherrypy.dll in the
# current directory. As the server executes, the PyISAPI framework will load
# this module and create the Extension object.
# A Virtual Directory named "isapi-wsgi-cherrypy" is setup. This dir has the ISAPI
# WSGI extension as the only application, mapped to file-extension '*'.
# Therefore, isapi_wsgi extension handles *all* requests in this directory.
#
# To launch this application from a web browser use a url similar to:
#
#  http://localhost/isapi-wsgi-cherrypy/test/
#
# NOTE: Due to an issue in how CherryPy2 currently handles PATH_INFO
# the url must have the trailing /

import isapi_wsgi
import cherrypy
from cherrypy import wsgiapp
from cherrypy import cpg

class WelcomePage:

    def index(self):
        # Ask for the user's name.
        return '''
            <form action="greetUser" method="get">
            What is your name?
            <input style="BACKGROUND-COLOR: #ffffa0" name="name">
            <input type="submit" value="Submit Query">
            </form>
        '''

    index.exposed = True


    def greetUser(self, name = None):
        # CherryPy passes all GET and POST variables as method parameters.
        # It doesn't make a difference where the variables come from, how
        # large their contents are, and so on.
        #
        # You can define default parameter values as usual. In this
        # example, the "name" parameter defaults to None so we can check
        # if a name was actually specified.

        if name:
            # Greet the user!
            return "Hey %s, what's up?" % name
        else:
            # No name was specified
            return 'Please enter your name <a href="./">here</a>.'

    greetUser.exposed = True


class CherryPyWsgiApp:
    cpg.root = WelcomePage()
    wsgiapp.init(configDict = {'socketPort': 80, 'sessionStorageType': 'ram'})
    def __call__(self, environ, start_response):
        return wsgiapp.wsgiApp(environ, start_response)

# The entry points for the ISAPI extension.
def __ExtensionFactory__():
    return isapi_wsgi.ISAPISimpleHandler(test = CherryPyWsgiApp())

if __name__=='__main__':
    # If run from the command-line, install ourselves.
    from isapi.install import *
    params = ISAPIParameters()
    # Setup the virtual directories - this is a list of directories our
    # extension uses - in this case only 1.
    # Each extension has a "script map" - this is the mapping of ISAPI
    # extensions.
    sm = [
        ScriptMapParams(Extension="*", Flags=0)
    ]
    vd = VirtualDirParameters(Name="isapi-wsgi-cherrypy",
                              Description = "ISAPI-WSGI Cherrypy Test",
                              ScriptMaps = sm,
                              ScriptMapUpdate = "replace"
                              )
    params.VirtualDirs = [vd]
    HandleCommandLine(params)

ISAPI-WSGI - Runs a Quixote application using QWIP

Titus Brown asked me to test if his simple Quixote-WSGI adapter (QWIP) would work with ISAPI-WSGI. The first launch worked, but every other launch failed until IIS was restarted. Traced it to the write method in ISAPISimpleHandler where I was using "print >> self.ecb, data" and this was raising a 10053 error 'An established connection was aborted by the software in your host machine.' Changing it to "self.ecb.WriteData(data)" fixed the problem. Why? will investigate later. Change checked-in as revision 6.

So running quixote.demo with QWIP using ISAPI=WSGI now works. I think this proves that WSGI is a very important standard for Python and the web.

Below is the code to do it:

import isapi_wsgi

import qwip



# The entry points for the ISAPI extension.

def __ExtensionFactory__():

    return isapi_wsgi.ISAPISimpleHandler(demo = qwip.QWIP('quixote.demo'))



if __name__=='__main__':

    # If run from the command-line, install ourselves.

    from isapi.install import *

    params = ISAPIParameters()

    # Setup the virtual directories - this is a list of directories our

    # extension uses - in this case only 1.

    # Each extension has a "script map" - this is the mapping of ISAPI

    # extensions.

    sm = [

        ScriptMapParams(Extension="*", Flags=0)

    ]

    vd = VirtualDirParameters(Name="isapi-wsgi-qwip-test",

                              Description = "ISAPI-WSGI QWIP Test",

                              ScriptMaps = sm,

                              ScriptMapUpdate = "replace"

                              )

    params.VirtualDirs = [vd]

    HandleCommandLine(params)

Announce:ISAPI-WSGI 0.4 Beta

Created a distribution and sent the following to the Python Web-Sig today:

Hello everyone,

I am happy to announce the release of ISAPI-WSGI 0.4 beta.

ISAPI-WSGI will (hopefully) allow any WSGI application to run inside a windows webserver that supports ISAPI. I believe that it meets the requirements of the WSGI PEP. It has been only tested against the examples from the PEP, Ian Bickings' echo example from wsgi-webkit, and Titus Browns' WSGI enabled Simple Commenting System under IIS 5.1. It has one major limitation being that it is only single threaded. I am currently working on a fully threaded version, but wanted to release it now so others could have a look at it. I am interested in any feedback, suggestions or bug reports.

See http://isapi-wsgi.python-hosting.com/wiki/DocsPage for info and get the python source & some examples at http://isapi-wsgi.python-hosting.com/wiki/ISAPISimpleHandler-0.4-beta

Regards

Mark Rees

ISAPI-WSGI runs a real WSGI application

In a previous post I wrote about wanting to test isapi_wsgi using a real WSGI enabled app rather than my simple test apps. The good news is I was able to get Titus Brown's WSGI middleware: a simple commenting system running.

Now working on getting CherryPy2 running under it. But now happy to try and release beta of isapi_wsgi.py this weekend. Also the source code has been published under Subversion at http://isapi-wsgi.python-hosting.com/browser/

ISAPI-WSGI hosting url has changed.

As Ian Bicking pointed out to me, underscores in a url is not allowed (even if some OS's supported it). This was causing problems for some people accessing the site, so the guys have changed the url for me. Thanks Richard & Remi. The url is now http://isapi-wsgi.python-hosting.com.

ISAPI-WSGI What else should I do before release?

Almost ready to release the beta, but still concerned I may have missed something or will make a fool of myself. Based on Ian Bicking "like a school play" blog entry, the Python community should be kind to me. :-) As a final test of my implementation I decided to look for apps that support WSGI. Found Titus Brown's WSGI middleware: a simple commenting system so will try running it under isapi_wsgi. Also will see if I can get the latest checkout of CherryPy2 running with it.

ISAPI-WSGI now has a Trac/Subversion site

Thanks to the guys at python-hosting.com the project development will be hosted at http://isapi_wsgi.python-hosting.com Will import the source into Subversion in the next couple of days.

ISAPI-WSGI Getting ready for first release

I am happy with the current functionality of the ISAPISimpleHandler. Used Ian Bicking's WSGI Webkit unit tests for the final testing. It highlighted a couple of issues in SCRIPT_NAME and PATH_INFO handling but these are now fixed. So arranging for a either a CVS or Subversion location on the net to hold the "small" source tree and download files. Will hopefully create a distutils package over the weekend, send it to a couple of friends to test, then make the announcement.

CATEGORY: PYTHON

My other interest is wine

Just to prove that I do have other interests, I decided to create this entry. Four years ago I decided that I needed to consider life after programming (not the fun programming, but the stressful, corporate, day job programming). Also it was mid-life crisis time, so I looked for what else I was passonate about and came up with wine. 13 years ago my cousin, his wife and my parents started a vineyard called Hatton Estate. At the time I was searching for the future, they decided that we needed to expand into winemaking, so I became a shareholder. So every March/April I head back across the Tasman Sea to my place of birth, New Zealand and work the vintage. It's great to spend time out in the open air working with nature, or plunging the open top frementors. It really clears the mind.

The good news is our premium wine, Hatton Estate Tahi (tahi is the Maori word for One) has been selected as one of the top 22 New Zealand wines by the San Francisco Chronicle wine panel.

CATEGORY: WINE

ISAPI - How do you use ReadClient

While working on my ISAPI-WSGI implementation I needed to create a wrapper so i could emulate stdin. After a quick read of the Python isapi docs, I decided I needed to use the EXTENSION_CONTROL_BLOCK.ReadClient method. 30 minutes of blocked input and debugging, I started searching on Goggle for more info on ReadClient. It turns out that you only call ReadClient if ecb.TotalBytes is greater than ecb.AvailableBytes and that you must pass the value of difference to ReadClient else it blocks. So the code needed to be something like:

    self._ecb = ecb

    if self._ecb.AvailableBytes > 0:

        data = self._ecb.AvailableData

        # Check if more data from client than what is in

        # ecb.AvailableData

        excess = self._ecb.TotalBytes - self._ecb.AvailableBytes

        if excess > 0:

            extra = self._ecb.ReadClient(excess)

            data = data + extra

        self._in.write(data)

    # rewind to start

    self._in.seek(0)

Morale of story: win32 extension docs are light, due to it being a Python wrapper around the Windows API, and you need to read the Microsoft docs to get all the info.

Now just need to find some Microsoft docs that I can understand about Windows completion ports. Then I can start work on the threaded extension.

The simple ISAPI-WSGI handler seems to be working. Need to create some unit tests to exercise it.

CATEGORY:PYTHON

Subway

Peter Hunt is working on an interesting project called Subway

From the homepage it is:

"The Subway project aims to create a Web development stack for Python similar to Ruby on Rails, utilizing as many pre-written software components as possible."

Two of the software components are Cheetah and SQLObject, favorites of mine. It is currently pre-alpha, but after checking out from the Subversion trunk and having a play, it looks like something worth keeping track of.

CATEGORY:PYTHON

ISAPI-WSGI

As I said in the previous post, I decided to become more involved in the community. I have been following the development of various web server implementations of Python WSGI. There have been a number of requests for an implementation for IIS using ISAPI and since I was playing with ISAPI filters using Python's win32 extensions, I decided to see if I could do it. So in conjunction with Peter Hunt (someone with WSGI implementation experience and a much better Python programmer than me) I have been moving forward. To date I have a non-threaded version that works but does not fully support the WSGI spec yet. Hopefully in the next week or so, we can release a beta.

CATEGORY:PYTHON

Python and Open Source Community

Last December I attended the Open Source Developers Conference in Melbourne. As well as speaking there, I got to hear and meet many members of the Open Source community. My main reason for attending was the Python Track, but the keynotes, lighting talks and discussion after each day in the pub really "expanded" my mind. It also got me thinking about what I give back to the Open Source community. Apart from the ZInformixDA and abit of beta testing and some patches for a couple of projects, I decided not enough. So the New Year resolution is to become more involved in the community.

CATEGORY:PYTHON

Everyone's doing it, so why not me

I read other people's blogs, I learn things I didn't know from them, so I decided I should give it a go myself.

In my day job I manage a software development team for a small private company based in Sydney, Australia. And since programming and technology are my interests, this blog will contain ramblings about these. And of course any other things I consider are worth recording.

CATEGORY:GENERAL