CLI - a journey of discovery

My open source interest has been focused on CLI for the last month, not the Command Line Interface but the Common Language Infrastructure. Thanks to mono there are tools to develop cross platform, cross language applications. As I am discovering the reality is a little different. So expect a few entries in this blog documenting the challenges, disappointments and victories as my journey of discovery continues. Maybe it's a good topic for OSDC 2006 paper.

Today I am an Australian

After living in Australia for the last 7 years, marrying an Australian, and creating a new little Australian, last night with 80 others, I attended my Australian citizenship ceremony. So another phase in my life begins.

OSDC 2006 Call for Papers

The Open Source Developers Conference is happening again this December in Melbourne, Australia. The last two conferences were excellent. Of course what makes a great conference is the people that attend and the content. The Call for Papers is now on.

I hope to attend this year but not sure if I will submit a paper proposal.

SQLAlchemy, more than an ORM

I remember seeing references to Michael Bayers' SQLAlchemy and thinking yet another Python ORM to look at sometime in the future. I have used SQLObject for various hobby projects, but since I deal with legacy databases (some database schema's just do not map well to an ORM) at the day job, I have tended to use SQL and the Python DBI for my database access work. Part of my job is to evaluate solutions to make my team more productive, and in reviewing where we allocate our time during development, I noted that a good portion is ensuring our SQL works across multiple RDBMS's. So I have been looking for Python solutions that abstract the creation of SQL against different RDBMS's. On investigating SQLAlchemy I was surprised to see that it you can use it to do exactly what I wanted. Also one of goals of Ian Bickings' SQL-API is to offer this functionality, but what follows is a discussion of how SQLAlchemy could help.

First connect to a database

from sqlalchemy import *
engine = create_engine("sqlite://filename=mydb",echo=True)

The option echo is set so we can view the generated SQL.

Now describe the tables using using table metadata objects

entries = Table('entries', engine,
        Column('entry_id', Integer, primary_key = True),
        Column('title', String(255), nullable = True),
        Column('content', String(), nullable = True),
        Column('status', String(10), nullable = False),
        Column('creation_date', DateTime()),
        Column('publish_date', DateTime())
    )

comments = Table('comments', engine,
        Column('comment_id', Integer, primary_key = True),
        Column('entry_id', Integer, ForeignKey("entries")),
        Column('comment', String(255), nullable = False),
        Column('submitter', String(60), nullable = False),
        Column('submit_date', DateTime())
    )

# Create the tables
entries.create()
comments.create()

Now load some data.

entries.insert().execute(entry_id=1,
            title="My first blog entry",
            content="A blogging I will go.....",
            status="published",
            creation_date="2006-04-17 08:15:30",
            publish_date="2006-04-17 08:25:00")

entries.insert().execute(entry_id=2,
            title="My second blog entry",
            content="Another day, another blog entry....",
            status="draft",
            creation_date="2006-04-18 09:35:30")

comments.insert().execute(comment_id=1,
            entry_id=1,
            comment="The entry needs more substance.",
            submitter="joe.blogger@home.net",
            submit_date="2006-04-17 09:05:00")

comments.insert().execute(comment_id=2,
            entry_id=1,
            comment="I disagree with the first comment.",
            submitter="jane.doe@rip.net",
            submit_date="2006-04-17 09:05:00")

Get some data with a simple select with where clause and order by

cursor = entries.select(entries.c.status=='published',
                order_by=[entries.c.publish_date]).execute()

rows = cursor.fetchall()
for row in rows:
    # Get column data by index
    title = row[1]
    # by column name
    content = row['content']
    # by column accessor
    status = row.status

Now a simple join

cursor = comments.select(comments.c.entry_id==entries.c.entry_id).execute()

which generates the following SQL:

SELECT comments.comment_id, comments.entry_id, comments.comment, 
comments.submitter, comments.submit_date
FROM comments, entries
WHERE comments.entry_id = entries.entry_id

Of course, this quick overview only scratches the surface of what you can do with SQLAlchemy. It also has support for outer joins, subqueries, unions, updates and deletes. See here for more in depth documentation.

Pylons - another great tool for my programming toolkit

In a previous post I mentioned that I had been looking at Pylons, a WSGI enabled web framework and had a project that would be a good fit. Finally have found some time to start work on it and my experiences with Pylons so far have been good. I picked Pylons for a number of reasons:

• WSGI enabled
• Uses Paste
• Not aligned with any particular ORM
• Support of different template packages via Buffet
  
• Reasonable documentation to get you started
  

I originally had concerns about learning yet another template syntax - Myghty, Pylons default template package, but have found it easy enough to learn and use. So one of the reasons I choose Pylons - support of different template packages really has no substance. But some of the other things I have discovered while using Pylons certainly should be added to the reasons list. Both of them are re-implementations of Rails functionality.

• Routes - makes it easy to create nice and concise URL's like http://timesheets.com/user/fred-smith/edit. No more ugly http://timesheets.com/user-edit.html?name=fred-smith
  
• WebHelpers - functions that simplify web development with template languages by providing common view patterns in re-usable modules. In addition to the Rails webhelpers, there helpers for HTML generation and pagination for collections and ORMs.

And if you are not interested in using Pylons, both these packages can be used standalone.

As I progress with my project, I am sure there will be more posts about my Pylons experiences.

Getting ISAPI-WSGI from Subversion Repository

Had an anonymous email asking how to check-out isapi-wsgi from subversion. The following command will do it:

svn co http://svn.isapi-wsgi.python-hosting.com/trunk

Hex Dump Tools written in Python

In reviewing my blog access stats, it would appear that most used search engine keyword combination that sends visitors to my site are "hex" and "dump". So to not disappoint these 20% of visitors, I have tracked down a number of hex dumping tools written in my favorite programming language.

READBIN by Tony Dycks is a Text Console-based program which reads a single Input File specified on the command line one character at a time and prints out a formatted hex "dump" representation of the files contents 16 characters per display line. A prompt for continuation is issued after displaying 20 lines (320 characters of information). An entry of "X" or "x" followed by the key terminates the program execution. Any other entry followed by continues the display of the formatted hex and character information. A "." character is used for any non-displayable hex character

Hex Dumper by Sébastien Keim is a function which will display to stdout, the classic 3 column hex dump of a string passed to it.

Hexdump by Ned Batchelder prints a 3 column hex dump to stdout of a list of files or stdin.

What's been happening in the world of WSGI

Good to see some activity in the development of WSGI component based microframeworks.

I have been following Ben Bangert and James Gardners' work on Pylons which is based on Myghty, with a custom Resolver, full Paste and WSGI integration. Haven't had a play with it yet, but have an idea for a project where Pylons may be the perfect fit.

Also Julian Krause has just released RhubarbTart, a light object publishing web framework built on WSGI and Paste. It's object publishing model is similar to CherryPy. Since it built on Paste, RhubardTart makes the most of Paste's selection of WSGI middleware components to provide the other functionality expected of a web framework. So if have been using CherryPy because of it's ease in exposing object methods, but want to customise the component stack you use for a web project, have a look at RhubarbTart.

Then again, if you want to create your own WSGI/Paste framework, Ian Bickings has created a great tutorial to get you started. If you have a problem with understanding what WSGI/Paste is all about, read the tutorial, you will be enlightened.

And even Guido this week has nice things to say about WSGI - 'Maybe the current crop of Python web frameworks (as well as Rails BTW) have it all wrong. Maybe the WSGI folks are the only ones who are "getting" it.'

One of the joys of programming in Python is "batteries included" and the fact I get to pick which batteries to use to help me solve the programming task at hand. With WSGI I get the freedom to pick what components I want use in the HTTP request/response area for a web project, and get to focus on solving the actual programming problem or challenge.

The Year in Review

Tomorrow my blog is one year old. So thought it was time to review it's value. In the last year I have made 44 posts, not quite once a week based on the math. But truthfully, I appear to blog in bursts directly related to when I am doing open source stuff. And since any open source work can only be done in my free time, which is hard to find as I try to balance life between work and family, there are large gaps of time between these bursts. I am in awe of the open source developers that seem able to hold down a day job, produce heaps of amazing code, and also update their blogs most days. Have these people discovered a secret source of free time, or can survive with 2 hours sleep a day, or maybe they are younger than me :-)

In reviewing what I have blogged about, it is great to see that the focus in the WSGI world has moved from server gateway implementations to frameworks and components that use it. In the future, I expect to write more posts on using WSGI and Paste as I implement systems at work using them.

Having a blog has hopefully improved my writing skills. When I started a year ago, my first posts were based on my thoughts, saved as drafts, then tweaked until they became coherent, then published. Now I find myself able to get these thoughts in something readable in the first attempt most of the time. Also having a blog means, other people leave comments and links to their blogs. This has meant greater exposure to others thoughts and experiences.

So this blog is of great value to me and hopefully the odd thought I have posted has been of value to someone else.

Why is the PC running so slow?

At work we created a new VMWare image of the latest release of our Financial Software. Since it was required for a demo at our KL office, we needed to compress it prior to sending over the Internet to them. Since it was over 4GB, we used WinRAR to compress each virtual disk. It took forever and transferring via scp was just as bad. The cpu was the bottleneck, not because of the compression but the virus scanner scanning the VMWare image :-(. Morale of the story disable scanning for your VM image directories.

I guess a sign of times, were there are more background tasks running than "real" work being done in the foreground. And please, no comments about moving our work desktops to Linux or buying Mac's. Our server components are developed under Linux and run under most versions of Unix and Linux, but all our customers use M$ operating systems for the clientside. So we need to as well.

Merry Christmas

Heading off on computer free Christmas holidays tomorrow. Back to my homeland of New Zealand and looking forward to crayfish and beef on a spit for Christmas dinner. So just wanted to post a greeting now. Thanks to all the people who have read my blog, the members of the open source community who have helped me and have provided feedback. To those who celebrate Christmas, have a great one with family and friends.

Recharged after OSDC

After attending OSDC last week, I came back energised and wanting to get some more open source stuff done.

So apart from getting Roundup to work with WSGI, I have been busy integrating a newly released version of informixdb into the Zope Informix DA and SQLObject. Informix is our development database at work and for any *ix Python DBI access we have been using mxODCBC as the previous release of informixdb had suspect threading support. The new version can be compiled with or without thread support and uses distutils to do the compile. The use of distutils alone makes it worthwhile upgrading as anyone who has tried to get the old informixdb to link has a horror story to tell. It also supports the newer data types of IDS 9 and 10.

The Zope Informix DA work is done and is currently being tested prior to release.

The experience of creating an Informix data connection for SQLObject has been interesting. I was surprised to discover that SQLObject creates insert and update SQL statements as complete strings including the data values rather than using placeholders. This created a problem for me with TEXT BLOBS as the informixdb did not translate from a string but required a buffer. The various solutions I tried can be viewed here. As none of the solutions were really perfect, I decided to request an enhancement to informixdb so it would do the string to BLOB translation automatically and the Open Source community delivered yet again. In less than 24 hours, Carsten Haese had provided a patch that did exactly what I wanted.

It had been my intention to get all these changes tested and checked in prior to going on Christmas holidays but as usual my free time as been limited with other important things like family. So looks like it will have to wait until January.

OSDC 2005 - My WSGI Talk Errata

I have updated the copy of the paper to include more info on using paste.deploy pipelines. This info was presented at the conference but wasn't part of my submitted paper due to time constraints.

OSDC 2005 - My WSGI talk

Due to lack of an Internet connection during the conference (my issue not OSDC's), I have not had a chance to post anything about the conference until now. On Monday 5 December I gave my talk on WSGI. Not really sure how it went across as no one asked any questions at the end or really talked to me about it during the rest of the conference. Of course maybe it was so good, everyone now fully understands WSGI, but that is very unlikely. Anyway the paper can be found here and the only photographic evidence of my talk can be seen here.

In the closing keynote "Tools for Freedom", Jeff Waugh made an interesting observation about how a user of our software doesn't really give a damm about the technical aspects, they just want it to make their life easier. That got me thinking about how to improve the uptake of WSGI. As much as TurboGears can use WSGI, most developers will just run the standard CherryPY server. Maybe when they start thinking about production deployment, then WSGI may be used as the "making life easier" solution. So for the next couple of months I hope to focus on making some existing python apps WSGI enabled so the users of these apps will find deployment easier. So during breaks in the conference, I wrote an adaptor to run Roundup as a WSGI app. Fixed the last of the problems on the plane to Sydney this morning and once I have fully tested it and tried it at work for one of our existing Roundup tracker sites, I will look at checking it in to the Roundup CVS.

Not sure, about the next Python app, any suggestions?