OSDC Proposal Outline

It would appear that you need a login to view my proposal. So here is a copy.

WSGI - Gateway or Glue?

Short Description: Present a review of where WSGI currently is and describe/show how it is being used to create a framework-neutral set of components.

Stream: Python

30 minute Paper Presentation in English


Abstract
The Web Service Gateway Interface is an API defined in PEP 333 that allows web servers to talk to Python web applications. At the time of OSDC 2005, the PEP will be 2 years old and the presentation will review the state of WSGI implementations for the various web servers including Apache, IIS, Twisted etc. It will also show how the API is used to implement a web server interface and create an application. In the last 6 months, the main WSGI community focus has been on creating web applications by composing them from components that communicate via WSGI. The remainder of the presentation will discuss this and show what can be done.

OSDC Proposal Accepted

"Your proposal has now been reviewed, and I am pleased to be able to inform you that it has been accepted for inclusion in the conference program" so it is now time for me to gather together all the ideas that have flowing around in my head and create a coherent paper from them. The proposal description can be found at http://osdc2005.cgpublisher.com/proposals/24

After looking at the other accepted proposals, I know this is going to be an even better OSDC than last year. Also as mentioned in a previous post there is a much better balance in the "P" scripting languages than last year ;-)

So if you want to spend an interesting 3 days in Melbourne in early December, why not goto the Open Source Developers Conference

Argh, having to learn java again

After managing to keep myself employed for almost the last decade without having to program in Java, a project at work means that the end is nigh. Pulled the "Teach yourself Java in 21 days" out of storage and noted that it was published in 1996. So Java and I have kept our distance for a long time. Also means I need to get myself up to speed with what has happened to Java over the period. In the end, I just need to learn enough to embed Jython 2.1 within an existing Java front-end so I can access it's API, well that's the theory anyway. Have spent the weekend consuming lot's of articles from the web and playing in eclipse. Have lot's of respect for what "real" Java programmers are doing out there, but hard to get back into writing lot's of code to achieve something. Python has spoilt me.

OSDC Proposal Submitted

Due to the "massive" response from my readers (thanks Richard) I went with WSGI and submitted a proposal. As of last night, it looks like they have 55 proposals of a wide and interesting range of topics. It would also appear there are more Python than Perl proposals, but I had my rose coloured glasses on, so I may be mistaken.

Goodbye Windows, Welcome Linux to my home

At work I use Windows XP as my workstation because that's what our clients use to run our frontend. Most of our development work is done on servers running Linux and truthfully the most used program on my XP box is ssh. But at home, I have always run a Microsoft OS. Of late, I have been questioning why. The home machine is used for paying the bills via the Internet, reading email, browsing the web, updating this blog and the odd bit of wordprocessing. In the past, it was also used to play games, but the XBox has taken over that role. Based on the last couple of months, the home machine seems to spend most of it's time downloading service packs or the lastest anti-virius update. So I decided to give Linux a chance to prove that it could do my home tasks. Using a "Live" CD version of Ubuntu 5.04, I have survived two weeks without having to see a Microsoft logo. Another interesting thing I have discovered is thanks to GMail etc, that I haven't needed any local hard disk storage.

Database design for newbies

Found this whitepaper on database design by Ian Bicking & Brian Moloney. A good explanation of Dr Codd's rules but in a language a newbie could understand.

How fast a year goes by, OSDC decisions again...

It doesn't seem that long ago that I was considering presenting a paper at OSDC 2004. But I did, and here we are a year later in the same situation. I really enjoyed last year's conference, both giving my presentation, meeting many interesting people, and deep discussions over many beers each night. I want to submit a proposal for a paper but still haven't decided what I want to talk about. Currently I have narrowed it down to 3 topics:

• WSGI - Present a review of where it currently is and describe/show how it is being used to create a framework-neutral set of components.
  
• A Smart Client using Python - Microsoft is pushing 2 types of user interface for an n-tier application, a web client and what they refer to as a "smart" client. In M$'s case, it is written using WinForms and other .NET technologies. I want to produce something similar using Python. Have been playing with both wxPython & PyXPCOM/XULRunner before deciding on the UI toolkit to use.
  
• ActiveGrid - This is an open source framework for building Web 2.0 using LAMP. Think it's more a candiate for a lighting talk.

Will have to make up my mind soon, as proposals have to be in by Friday 19 August.

Creating an ODBC data source using Python

For the cross-platform installer I am creating, I need to create ODBC data sources for Windows installations. Thanks to the Python ctypes module the solution was as simple as:

import ctypes

ODBC_ADD_DSN = 1        # Add data source
ODBC_CONFIG_DSN = 2     # Configure (edit) data source
ODBC_REMOVE_DSN = 3     # Remove data source
ODBC_ADD_SYS_DSN = 4    # add a system DSN
ODBC_CONFIG_SYS_DSN = 5 # Configure a system DSN
ODBC_REMOVE_SYS_DSN = 6 # remove a system DSN

def create_sys_dsn(driver, **kw):
    """Create a  system DSN
    Parameters:
        driver - ODBC driver name
        kw - Driver attributes
    Returns:
        0 - DSN not created
        1 - DSN created
    """
    nul = chr(0)
    attributes = []
    for attr in kw.keys():
        attributes.append("%s=%s" % (attr, kw[attr]))
    
    return ctypes.windll.ODBCCP32.SQLConfigDataSource(0, ODBC_ADD_SYS_DSN, driver, nul.join(attributes))

if __name__ == "__main__":
    print create_sys_dsn("SQL Server",SERVER="(local)", DESCRIPTION="SQL Server DSN", DSN="SQL SERVER DSN", Database="mydatabase", Trusted_Connection="Yes")
    print create_sys_dsn("mySQL",SERVER="local", DESCRIPTION="mySQL Server Test1", DSN="mySQL DSN", DATABASE="mySQLDb", UID="username", PASSWORD="password", PORT="3306", OPTION="3")

Require a cross-platform installer, Python to the rescue

The company I work for develops an ERP system that has it's heritage in implementations for medium size companies running on UNIX systems. As the global ERP vendors have moved in our space, we have had to move into the SME market which means running on Linux or Windows server systems. And since some of our new clients cannot afford the cost of a technical resource going on-site to install and configure the technical infrastructure, we need a cross-platform installer. As our application is not written in Java, we are not interested in a Java based installer. But since our supporting scripts are written in Python, and I have had some success creating a cross-platform installer in Python to install and build the the required 3rd party packages for Python Paste, I decided Python could provide a solution. Currently a work in progress, I have a Python script that installs our complete ERP app, it's supporting runtimes and configures it to run in under 10 minutes. Still need to get database setup working, and as our app can run under Oracle, IBM Informix or MS SQL Server 2000 there is abit of work to be done.

Of course to make it look like a "real" installer under Windows I use Inno Setup, and call an executable version of the python script created by py2exe. Rather try trying to write in Delphi to get custom forms for the installor I used ctypes and the Windows port of EasyDialogs to collect site specific setup data. So thanks to the hard work of the creators of the various tools I have used and Python, it looks like we will have our cross-platform installer.

build-pk.py now part of Python Paste

Ian has given me check-in access to the PythonPaste SVN, so I have checked in my cross-platform build-pkg.py script. Also looking at making a distribution of the 3rd party packages using Python Eggs.

Installing WSGIKit, opps I mean Python Paste under Win32

Ian has been unhappy with the name WSGIKit, so it has been renamed Python Paste I think the name better matches what the package does, the WSGI glue that sticks all the python components together. Anyway have made some minor changes to my cross-platform build-pkg.py script so that it works more like Ian's shell script. Have sent a copy to him, so who knows, maybe it will be part of Python Paste.

                                                                               
#!/usr/bin/env python
"""
Install required third party packages for Python Paste.

Known limitations

 - Running under Windows, it must be run with Python 2.3+
   as it requires the tarfile module. On Unix/Linix platform
   will fallback to tar program.
"""
import os
import sys
import errno
import urllib
import shutil

TMP="/tmp"
for tmp in ['TEMP','TMPDIR','TMP']:
    if os.environ.has_key(tmp):
        TMP = os.environ[tmp]
        break
TMP = TMP + os.sep + "tmp-build"

BASE=os.path.dirname(sys.argv[0])
if BASE == '' or BASE == ".":
    BASE = os.getcwd()

THIRD = BASE + "/paste/3rd-party"

def basename(filename, ext):
    try:
        i = filename.index(ext)
        base = filename[0:i]
    except ValueError:
        base = filename
        pass

    return base

def mkdirs(dirname, mode=0777):
    # Python Cookbook Edition 1 Recipe 4.17
    try:
        os.makedirs(dirname)
    except OSError, err:
            if err.errno != errno.EEXIST or not os.path.isdir(TMP):
                raise

def delete_pyc(arg, dirname, files):
    """
    Called by os.path.walk to delete .pyc and .pyo files
    """
    for name in files:
        if name.endswith(".pyc") or name.endswith(".pyo"):
            os.remove(os.path.join(dirname, name))

def get_file(zip_type, filename, url):
    """
    If required, download requested third party package and extract.
    On exit, the script is in the extracted package directory.
    """
    mkdirs(TMP)
    os.chdir(TMP)
    if not os.path.exists(filename):
        print "Download %s ..." % filename
        urllib.urlretrieve(url, filename)
    DIR=basename(filename, ".tar.gz")
    DIR=basename(DIR, ".tar.bz2")
    DIR=basename(DIR, ".tgz")
    if not os.path.exists(DIR):
        try:
            import tarfile
            tar = tarfile.open(filename, "r:" + zip_type)
            for file in tar.getnames():
                tar.extract(file)
        except ImportError:
            # No tarfile module, so use actual tar program
            if zip_type == "gz":
                zip_type = "z"
            else:
                zip_type = "j"
            os.system('tar fx%s "%s"' % (zip_type, filename))

    try:
        os.chdir(DIR)
    except OSError:
        pass

def installer(name):
    mkdirs(THIRD + "/" + name +"-files")
    cmd = '%s setup.py install -f --install-lib="%s/%s-files" --install-scripts="%s/%s-files/scripts" --no-compile' % (sys.executable, THIRD, name, THIRD, name)
    print cmd
    os.system(cmd)

get_file("gz","WSGI Utils-0.5.tar.gz",
        "http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz")
installer("wsgiutils")

get_file("gz", "Component-0.1.tar.gz", 
        "http://webwareforpython.org/downloads/Component-0.1.tar.gz")
DEST = THIRD + "/Component-files/Component"
mkdirs(DEST)
shutil.rmtree(DEST,ignore_errors=1)
os.chdir(THIRD)
shutil.copytree(TMP + os.sep +"Component-0.1", DEST)

zptkit_tmpdir = TMP + "/" + "ZPTKit"
if os.path.exists(zptkit_tmpdir):
    os.chdir(zptkit_tmpdir)
    cmd ="svn up"
else:
    cmd ="svn co http://svn.w4py.org/ZPTKit/trunk %s/ZPTKit" % TMP
os.system(cmd)
DEST = THIRD + "/ZPTKit-files/ZPTKit"
mkdirs(DEST)
shutil.rmtree(DEST,ignore_errors=1)
os.chdir(THIRD)
shutil.copytree(TMP + "/ZPTKit", DEST)

get_file("gz", "ZopePageTemplates-1.4.0.tar.gz", 
        "http://belnet.dl.sourceforge.net/sourceforge/zpt/ZopePageTemplates-1.4.0.tgz")
os.chdir("ZopePageTemplates")
installer("ZopePageTemplates")

get_file("gz", "scgi-1.2.tar.gz", 
        "http://www.mems-exchange.org/software/files/scgi/scgi-1.2.tar.gz")
installer("scgi")

# Do not clean up compiled python files for now
#os.path.walk(THIRD,delete_pyc,None)

sqlobject_tmpdir = TMP + "/" + "SQLObject"
if os.path.exists(sqlobject_tmpdir):
    os.chdir(sqlobject_tmpdir)
    cmd ="svn up"
else:
    cmd ="svn co http://svn.colorstudy.com/trunk/SQLObject %s/SQLObject" % TMP
os.system(cmd)
os.chdir(sqlobject_tmpdir)
installer("sqlobject")
mkdirs(THIRD + "/sqlobject-files/scripts")
shutil.copy(TMP + "/SQLObject/scripts/sqlobject-admin", 
           THIRD + "/sqlobject-files/scripts")


if not os.path.exists(TMP + "/PySourceColor.py"):
    urllib.urlretrieve("http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py", 
           TMP + "/PySourceColor.py")
mkdirs(THIRD + "/PySourceColor-files")
shutil.copy(TMP + "/PySourceColor.py", THIRD + "/PySourceColor-files")

Installing WSGIKit on a Win32 platform

So I could have a play with WSGIKit and look at how I could make ISAPI_WSGI work with it, I decided to install it on my home Windows XP box. As part of the svn checkout, there is a shell script called build-pkg which will download the required third party packages and install them under WSGIKit for you. Tried to run the script with a standard install of Cygwin but it failed due to various errors. I could have debugged my way thru the errors but thought why not create a truly cross-platform version of buld-pkg in python. So an hour later, the following script did the job for me. It currently lacks the checks to see if the package has already been downloaded, but it did the job for me.

import os
import sys
import errno
import tarfile
import urllib
import shutil

TMP="/tmp/tmp-build"
BASE=os.path.dirname(sys.argv[0])

if BASE == "" or BASE == ".":
    BASE = os.getcwd()

THIRD = BASE + "/wsgikit/3rd-party"

def basename(filename, ext):
    try:
        i = filename.index(ext)
        base = filename[0:i]
    except ValueError:
        base = filename
        pass

    return base

def mkdirs(dirname, mode=0777):
    # Python Cookbook Recipe edition 1 4.17
    try:
        os.makedirs(dirname)
    except OSError, err:
            if err.errno != errno.EEXIST or not os.path.isdir(TMP):
                raise

def get_file(zip_type, filename, url):
    print "Getting %s ..." % filename
    mkdirs(TMP)
    os.chdir(TMP)
    # TODO: check for existing package, download if not there
    urllib.urlretrieve(url, filename)
    DIR=basename(filename, ".tar.gz")
    DIR=basename(DIR, ".tar.bz2")
    DIR=basename(DIR, ".tgz")
    tar = tarfile.open(filename, "r:" + zip_type)
    for file in tar.getnames():
        tar.extract(file)
    try:
        os.chdir(DIR)
    except OSError:
        pass

def installer(name):
    mkdirs(THIRD + "/" + name +"-files")
    cmd = 'python setup.py install -f --install-lib="%s/%s-files" --install-scripts="%s/%s-files/scripts" --no-compile' % (THIRD, name, THIRD, name)
    os.system(cmd)

get_file("gz","WSGI Utils-0.5.tar.gz",
        "http://www.owlfish.com/software/wsgiutils/downloads/WSGI%20Utils-0.5.tar.gz")
installer("wsgiutils")

get_file("gz", "Component-0.1.tar.gz", 
        "http://webwareforpython.org/downloads/Component-0.1.tar.gz")
mkdirs(THIRD + "/Component-files")
DEST = THIRD + "/Component-files/Component"
shutil.rmtree(DEST,ignore_errors=1)
try:
    shutil.move(TMP + "/Component-0.1", DEST)
except OSError:
    pass

get_file("gz", "ZPTKit-0.1.tar.gz", 
        "http://imagescape.com/software/ZPTKit/ZPTKit-0.1.tar.gz")
mkdirs(THIRD + "/ZPTKit-files")
DEST = THIRD + "/ZPTKit-files/ZPTKit"
shutil.rmtree(DEST,ignore_errors=1)
try:
    shutil.move(TMP + "/ZPTKit-0.1", DEST)
except OSError:
    # To handle permission errors under Windows
    pass

get_file("gz", "ZopePageTemplates-1.4.0.tar.gz", 
        "http://belnet.dl.sourceforge.net/sourceforge/zpt/ZopePageTemplates-1.4.0.tgz")
os.chdir("ZopePageTemplates")
installer("ZopePageTemplates")

get_file("gz", "scgi-1.2.tar.gz", 
        "http://www.mems-exchange.org/software/files/scgi/scgi-1.2.tar.gz")
installer("scgi")

cmd ="svn co http://svn.colorstudy.com/trunk/SQLObject %s/SQLObject" % TMP
os.system(cmd)
os.chdir(TMP + "/SQLObject")
installer("sqlobject")
mkdirs(THIRD + "/sqlobject-files/scripts")
shutil.copy(TMP + "/SQLObject/scripts/sqlobject-admin", 
           THIRD + "/sqlobject-files/scripts")


urllib.urlretrieve("http://bellsouthpwp.net/m/e/mefjr75/python/PySourceColor.py", 
                TMP + "/PySourceColor.py")
mkdirs(THIRD + "/PySourceColor-files")
shutil.copy(TMP + "/PySourceColor.py", THIRD + "/PySourceColor-files")

Back to WSGI land

It's been over a month since my last entry as family and work have consumed most of my time. Some interesting developments for WSGI over the last month. Ian Bickings PyCon presentation provides a very good description of what WSGI is and what it is not. He used his WSGIKit for the usage examples. And just in case, we didn't appreciate what you can do with WSGIKit, a week later he provided an online tutorial on how to implement something like the famous Ruby on Rails ToDo List example with it. The WSGIKit tutorial can be found here.

Also came across some work that Allan Saddi has done. At last I can simply use FastCGI with my WSGI enabled web scripts. So using either isapi_wsgi or his fcgi.py I can run my scripts as long running processes on both IIS and Apache with no real extra effort.

ZopeX3 learning from others.

As I said in a previous post, the documentation for ZopeX3 is much better than what came with Zope 1. I found the programmers tutorial a very good starting point. Also some googling found this on-going series of blog entries creating a Todo. Nothing like learning about something by example.