My OSDC 2007 talks

Today I gave my two talks at OSDC.

This morning I gave a talk on web testing using twill and selenium remote control. A pdf of the slides can be downloaded:

Testing Web Applications with Scripting Languages

And this afternoon I spoke about Moonlight, the Mono implementation of Silverlight.

Moonlight - Shiny, Pretty Things with XML?

I had intended to blog about the various OSDC sessions I have attended, but thanks to the lazy web, Alan Green has already done it for me

A sneak peek at mod_wsgi 1.2

Errata: Opps, the title should be "A sneak peek at mod_wsgi 2.0"

Today Graham Dumpleton announced the first release candidate of mod_wsgi 2.0. mod_wsgi is a simple to use Apache module which can host any Python application which supports the Python WSGI interface. The new features added are:

• 'Process' option for WSGIReloadMechanism to make it easier to restart an application running in a mod_wsgi daemon process when code changes are made. When this option is enabled, you only need to touch the WSGI script file and the application to be reloaded.
  
• Ability to hook into the Apache 2.2 auth provider mechanism
• Support for Python virtual environments
• Provision of hooks to be able to work with the internal Apache APIs.

Read the changes document for more information.

If you are in Sydney, Australia on Thursday 1 October you could find out more by attending the SyPy meetup where Graham will be talking about mod_wsgi. Details can be found here.

November 28, a busy day for me

Just been notified by the OSDC 2007 Program Committee that my two presentations are on the same day.

11:00-11:30 Testing Web Applications with Scripting Languages

14:30-15:00 Moonlight - Shiny, Pretty Things with XML?

The complete conference program can be viewed here. So if you want to spend an interesting 3-4 days in Brisbane mixing with open source developers, early bird registration is open until October 14th. Then full price registration will be available until October 31th.

Version 1.0 of mod_wsgi is now available

Graham Dumpleton announced on the mod_wsgi mailing list yesterday:

"Okay folks, official 1.0 release of mod_wsgi is now available from
http://www.modwsgi.org.

The only difference between this version and mod_wsgi-1.0c4 is the
version string and some minor changes in README file.

Now that this is out of the way, I'll be able to get onto implementing
all the nice new features I want to add to make it even better. You
can get an idea for what some of these might be by looking at:

http://code.google.com/p/modwsgi/issues/list

If you have your own ideas, please post them here to the list and we
can all discuss them.

Thanks to everyone who has helped test the release candidates."

mod_wsgi 1.0 Release Candiate Available

Today Graham Dumpleton tagged the SVN and has made available the release candidate of mod_wsgi. The mod_wsgi adapter is an Apache module that provides a WSGI compliant interface for hosting Python 2.3+ Python based web applications under Apache. Graham has written it completely in C code so it has a lower overhead than the mod_python or CGI adapters. It can be used with Apache 1.3, 2.0 or 2.2 and normally runs in 'embedded mode' where the application code is executed within the context of normal Apache child processes. If you run mod_wsgi with Apache 2.x on a Unix/Linux system, it can also run in a 'daemon mode' where one or more distinct processes can be dedicated to one or more WSGI applications. Graham has put together great documentation including instructions on using mod_wsgi with some of the common Python web frameworks and applications including Trac, Django, Pylons, MoinMoin and TurboGears.

Adal Chiriliuc has built Win32 binaries of the release candidate and these can be found here.

I have lucky enough to have been associated with the testing of mod_wsgi since it's inception and want to thank Graham for all his efforts in developing and releasing a WSGI adapter of high quality and functionality. Also thanks to other testers who kept Graham honest and busy. In my humble and most likely biased opinion, I think it will become the WSGI adapter of choice when serving Python applications under Apache.

IronRuby Pre Alpha Source Code Released

Today John Lam announced the first release of IronRuby source code. Of interest to me in the announcement was the following quote:

"We're also happy to announce that we will be accepting source code contributions into the IronRuby libraries."

This policy is what many users of IronPython have been wanting since it's first release. Since both language implementations are released under the same license, hopefully Microsoft will see fit to change their policy for IronPython as well. And once the DLR reaches 1.0, hopefully we will be able to contribute to the IronPython compiler codebase as well.

So congratulatons to the IronRuby team on the progress to date, great to see dynamic languages getting more exposure in the .NET world.

Sydney traffic chaos disrupts SyPy meeting

Yesterday I hosted the July SyPy meetup at my work. Since the office is North Sydney based, many pythoneers would need to travel to the meeting using Sydney's rail network. It doesn't have the best record for reliability, and yesterday was a classic example. A panel was ripped off a CityRail train as it crossed the Harbour Bridge resulting to commuter chaos for many hours. The good news is that python programmers are a hardy bunch and ten overcame the chaos so we still managed to have a great meeting. Tim Churches gave a talk on the NetEpi Project which is open-source, network-enabled tools for epidemiology and public health practice. It is consists of two components:

1. NetEpi Collection, a Web-based outbreak data management tool, written in Python and using open-source components and infrastructure, including the PostgreSQL database.
2. NetEpi Analysis, a tool for interactive exploratory data analysis of large (10-100 million rows) population health/clinical data sets. It is mainly written in Python, uses Numeric Python (NumPy) and the R statistical environment.

It is good to see that at least one NSW government department is making an investment in open source. Nigel Tao talked about the Gnome Deskbar Applet, which is written in Python and can be extended using python scripts. He also introduced us to YubNub, the commandline for the web. Something else to distract me, thanks Nigel. Dyan Jay, web designer, and entrepreneur did a lightning talk about AGX which is a code-generator for CMF/Plone Products based on the Archetypes framework. The design is done in UML, saved in XMI-Format, and AGX parses the saved model, producing the python code.

Submitted a talk proposal for OSDC 2007

Yesterday I submitted my talk proposal for OSDC. If accepted, the talk will be about web testing using scripting languages.

The submission deadline for proposals is 30 June, so why not submit one as well. So in no particular order, Alan, Keith or Ben take this as me rattling your cages.

Google Developer Day tomorrow but not in Malaysia

Tomorrow is Google Developer Day and if I was in my normal country of residence I would be attending. But since I have been working in Malaysia for the last week, I will just have to wait for the Google videos. Very interested in the GData stuff as I have been working on some things with the Google Spreadsheet Data API. Expect some blog posts about it soon.

OSDC 2007 Call For Papers

This year's Open Source Developers Conference will be in Brisbane at the Royal on the Park Hotel (opposite the City Botanical Gardens). On their website I see that the "Call For Papers" is now open.

The key dates are:

Submission deadline 30th June 2007
Proposal acceptance 31st July 2007
Submission deadline 31st August 2007
OSDC 2007 Tutorials 26th November 2007
OSDC 2007 Conference 27th - 29th November 2007

Decision time again, do I want to submit a paper this year and if I do, what will it be about.

A new computer for work

I am writing this post with my new MacBook Pro (2.3GHz Core 2/2Gb DDR/120Gb SATA). Alan, Andy and Graham, I know I said I didn't want one but thanks to BootCamp and rEFIt, it certainly runs my favorite OS well. And I admit Mac OS X is proving to be a pleasant experience. You can quiz me why I changed my mind at the next SyPy meeting.

When I programmed in 6809 assembler

I have been asked how could I have programmed in 6809 assembler since a ZX-81 computer used the Zilog/NEC Z80 CPU. So to clarify things, I only ever programmed in BASIC on the ZX-81, but in the mid eighties I had a job designing and programming realtime controllers for fruit grading and handling machinery. These controllers used the 6809 processor. It was a very interesting and challenging job. The code was created on a system running Flex 9 OS, burned to EPROM and then tested in the real hardware. Debugging was done using a logic analyser.

Mono and the Dynamic Language Runtime

In my post about the IronPython 2.0 Alpha release, I wondered how the mono team would implement or support DLR. The answer appeared in this post by Miguel de Icaza

"The release for the DLR is done under the terms of the Microsoft Permissive License (MsPL) which is by all means an open source license. This means that we can use and distribute the DLR as part of Mono without having to build it from scratch. A brilliant move by Microsoft."

Hopefully there will not be many changes required to the mono compiler and runtime so it will compile and run the DLR. Looking forward to running IronPython 2.0 on Mono soon.

Dynamic Language Runtime Source Code

In my previous post I said I was unsure what license Microsoft's DLR was released under. But in this blog post from Jim Hugunin, I discovered it is released under the same license as IronPython 2.0 - the BSD-style Microsoft Permissive License. This means there is access to the source code for DLR. Currently the DLR source is shipped with source code distribution of IronPython 2.0 Alpha 1.

IronPython 2.0 Alpha 1 Released

Microsoft has just released the first alpha of IronPython 2.0.

This version of IronPython is built on top of the new Dynamic Language Runtime (DLR) which is a shared runtime for dynamic languages on CLI. One of the goals of DLR is to allow different dynamic languages to share library code, so in theory and the future my IronPython code would be able to access code in a IronRuby gem. The DLR is distributed as part of the IronPython release and is implemented with two assemblies - Microsoft.Scripting.dll and Microsoft.Scripting.Vestigial.dll

The final release of IronPython 2.0 will support the language features added to Python 2.5.

Microsoft has changed the license for IronPython to the Microsoft Permissive License. Comparing this with the IronPython 1.1 license, the new license doesn't appear to add any restrictions.

If you have read my other blog posts about IronPython, you would have noticed that I am a fan of of Mono, the open source implementation of the CLI. Sadly IronPython 2.0 will not run with Mono 1.2.3.1, as calls to the DLR assemblies cause a seg fault. I am not sure what path the Mono developers will take in relation to supporting DLR with Mono. They could modify the mono runtime to work with the DLR assemblies but it is likely the DLR license may not allow this. If they implement the DLR API inside Mono with a Mono namespace like Mono.Scripting, then any dynamic language using DLR would need to be conditionally compiled dependant on CLI implementation. So I await with interest for an "offical" comment from Miguel de Icaza.

Last nights Sydney Python Group Meetup

Last night was the monthly SyPy meetup. Thanks to generosity of Google the venue was the Fraser meeting room at the Sydney Googleplex. It was a great turnout of Sydney Pythoneers with 30+ attending.

First Alec Thomas gave a talk about PyCon 2007 and what he felt were the 10 best talks he attended. Alec's presentation can be found here. Then Andrew Bennetts gave a talk about the Bazaar GPL Distributed Version Control Software. Andrew did a good introduction of what Bazaar can do, then used the rest of the talk time to answer questions. Based on the number of questions, there is certainly alot of interest in this software tool.



After the meetup had finished, a number of us with a thirst for more discussion and beer headed to the cafe/bar next to the Google building where access to happy hour prices had been extended for us. Had some interesting discussions including one with cafe owner about the ultimate ui design for a bar point of sale.

Thanks to Alan Green for his organising skills and Google for providing an excellent venue.

PyCon Envy

I am jealous! I want to be at PyCon 2007 and reading the few blog posts (I think everyone there is having too good a time to blog) makes me want to be there even more. Hopefully the video and audio will be uploaded soon. Guess I will just have to save, convince the "better half" it's for the good of the family and make it to PyCon 2008. Alec Thomas will be giving a debrief on his time at PyCon 2007 for the April SyPy meeting so more envy likely.

IronPython 1.1 Beta 1 Released

Just in time for PyCon 2007, Dino Viehland of the IronPython Dev Team announced the release of the first IronPython 1.1 Beta.

<snip>
This release primarily focuses on adding the remaining new functionality for the 1.1 release (array module, importing pre-compiled modules) along with many new bug fixes. Also included but previously available with 1.1 Alpha 1 are the new XML Doc comment integration with the help system, the SHA, MD5, and select modules. The remaining of the 1.1 cycle will focus on bug fixes.
</snip>

OSDC 2007 will be in Brisbane

Arjen Lentz announced today that Brisbane will be the host city for this years OSDC. Time to start thinking about possible paper topics again.

Update

Thanks for reminding me Keith. Once the actual conference venue is announced, we will also need to find a venue for the nightly Beer BOF's.

FePy now shipped with Mono 1.2.3

From the Mono 1.2.3 release notes:

"In this release, we are now shipping the IronPython Community Edition (IPCE) from Seo Sanghyeon. IPCE includes a number of preconfigured libraries with IronPython to improve your IronPython developing experience."

This is release 5 of IPCE but Sanghyeon continues to improve it and the latest can be checked out from IPCE repository.

Blog of Note?

Today this blog is Bloggers "Blog of Note". Well that explains the abnormal traffic statistics.

mod_wsgi Performance

Graham Dumpleton has published his benchmarks for mod_wsgi.

mod_wsgi - simple and fast apache wsgi serving

There are a many ways to serve a WSGI application via Apache with FastCGI appearing to be the favorite. If you have to support many diverse operating systems running different versions of Apache, serving a WSGI application via CGI is a guaranteed but not fast deployment method. You may be wondering why I haven't suggested the WSGI mod_python adaptor as a solution. I am sure it is a workable alternative but it requires mod_python and I have had lots of issues getting it to compile for some platforms and to work with a specific Python interpreter. Now there is a new solution.

Graham Dumpleton has been working on an Apache module called mod_wsgi. Quoting from his website:

"The mod_wsgi adapter is an Apache module that provides a WSGI compliant interface for hosting Python based web applications within Apache. The adapter is written completely in C code against the Apache C runtime and for hosting WSGI applications within Apache provides significantly better performance than using existing Python based WSGI adapters for mod_python. The module can be compiled for and used with either Apache 1.3, 2.0 or 2.2."

I have been lucky enough to get early access to the code and try it with my WSGI applications. I able to compile, configure and have mod_wsgi running a simple WSGI application under Apache 2.0.55 and Python 2.4.3 in under 5 minutes, Certainly a much better experience than I have had with mod_python. Since I have been testing it on my "gutless" laptop rather listing the results of my benchmarking, I will leave Graham to publish his more comprehensive benchmarking results. But I will make some observations. It appears to run simple WSGI applications at least twice as fast as mod_python and 15-20 times faster than using CGI. I was also able to run a number of my WSGI applications that use either Pylons or combinations of WSGI middleware with no problems. At yesterdays SyPy meeting, Graham showed it running Django. The official release will hopefully be in March.

SyPy Social Meetup

Last night was the second meeting of the Sydney Python Users Group for 2007 and we had a good turnout.

There was a wide range of topics discussed including:

• Unit Testing
• Software Deployment
• Distributed Version Control
• Apache and WSGI - more about this in another post
  

So if you are in the Sydney and are interested these types of topics, why not join us the first Thursday of each month. Details of the next meeting can be found at:

http://groups.google.com/group/sydneypython/web/sypy-home

Hope to see you in March.

LCA 2007 My Day 3

Yesterday was the last day of linux conf au. As usual it was the mad rush from work to get to my first session, at least this time I wasn't late. The session was Robert McQueen's "Telepathy - Unifying IM, Voice and Video Communications". This was very interesting and after seeing the video demo using a Nokia N800, apart from wanting one, I will be keeping an eye of this project.

The next was Andre Pang's "Erlang and Concurrency" talk. Andre is an excellent and entertaining speaker. This talk investigated everyone's dislike or fear of threading, and then explained the advantages of Erlang's message passing distributed model. Every year I try to learn a new language, maybe it will be Erlang this year or maybe I should just look at Stackless Python.

After lunch, it was Rob Collins, Erik de Castro Lopo, and Peter Miller's "Do it our way" presentation. After attending their tutorial on Thursday, I was looking forward to this talk to hear about the other testing patterns they feel we should be using. Learnt a little bit more and looking forward to the release of the paper. They also intend to create a website to allow sharing of these testing patterns.

The last session I attended was Valerie Henson's "Choosing and Tuning Linux File Systems". Picked up a couple of interesting things, but the main take home of the talk was "use ext3 unless you know you need something else".

The day for me ended with the closing ceremony. Two of the conference attendee's went home with a olpc after being selected based on their submissions for a olpc software project proposal. Others who submitted a proposal were told not too cry as it was likely that a olpc was coming there way as well. I intend to have a play with the olpc emulation soon.

Thanks to the 7 team for a great conference and to all those who took the time to discuss OSS stuff with me.

IronPython and Traceback line numbers in Mono (workaround)

One of the reasons why I need to have access to a Windows dev environment when doing my IronPython/Mono development is the lack of line numbers in IronPython tracebacks under Mono. But via Seo Sanghyeon comes a workround. He has discovered that that Mono can't display line numbers from AssemblyBuilder built in memory, but only from Assembly loaded from disk. So you need run mono in debug mode and get IronPython to compile and save the assemblies to disk. So if we have a buggy script like:

a = 1
b = 0
c = a/b

Prior to the workround the traceback would have been:

Traceback (most recent call last):
  File foo, line unknown, in Initialize
ZeroDivisionError: Division by zero

But if I start IronPython using:

mono --debug ipy.exe -X:SaveAssemblies foo.py

you get a more helpful traceback

Traceback (most recent call last):
  File /u1/ip/IPCE/IPCE-svn/Lib/site.py, line 33, in Initialize
  File /u1/ip/IPCE/IPCE-svn/Lib/fepy/__init__.py, line 7, in install
  File /u1/ip/IPCE/IPCE-svn/Lib/fepy/__init__.py, line 11, in install_option
  File /u1/ip/IPCE/IPCE-svn/Lib/fepy/ast.py, line 17, in Initialize
SystemError: Object reference not set to an instance of an object
Traceback (most recent call last):
  File /tmp/foo.py, line 3, in Initialize
ZeroDivisionError: Division by zero

Thanks again Sangyheon.

LCA 2007 My Day 2

Today was day 4 of linux conf au, and my second day of attendance.

My first session today was Peter Miller, Erik de Castro Lopo, and Robert Collins tutorial "The Hao of testing". Since I have been to work before heading to the conference, I arrived at the tutorial a couple of minutes late to discover there had been homework. So a mad rush to get connected, gmp downloaded and built. Then I needed to write a unit test for C program, yuk, I was beginning to question my decision to attend. But once I started listening to what they had to say, I got over my initial horror and enjoyed the talk for it's lessions. And the last part of the tutorial was actually unit testing part of a python application. Looking forward to their session tomorrow where they will present a paper on what was discussed today and alot more.

After lunch I went to Malcolm Tredinnick's "Building Websites with Django" tutorial. Since I already have used Django, I attended this tutorial to see how non users picked up the framework and maybe learning some news things. Malcolm warned us that he was presenting something that normally took 3 hours but would try to condense it into the 1.5 hour timeslot. I feel he spent too much time explaining where Django came from and what people had done with it, rather than showing how to use the framework. There was no actual "Wow" factor until half way thru the session, but by then a third of the attendee's had left for other sessions. The last half of the tutorial was much better and I did learn a couple of new things. The slides and the example app for the tutorial can be found here.

The last session for my day was Ralph Giles "Screencasting Howto" tutorial. The aim of the tutorial was for all of us to have created a screencast and have it peer reviewed. Only a few of us managed to achieve this due to issues with getting the various tools working but there a good interaction between the attendee's and I learnt alot. One of the tools we used was Istanbul, a desktop session recorder for the Free Desktop which is written in Python and uses GStreamer. It's a little buggy and many of us didn't have much success until we dropped the recording frame rate to 1 fps.

Many of the sessions have been videoed by the LCA team and can be accessed from the Programme web page.

It was also the conference open day but I didn't get to see much as it was well attended (read as overcrowded), but this can only be good for OSS community.

LCA 2007 My Day 1

Today was day 3 of linux conf au, but it was the first day I attended and only got there just before 2pm. As I wanted to hear Michael Carden's talk on Digital Preservation, I headed to the lecture room where it was scheduled. As I was one session early, I listened to Rafael Espindola's talk - The ARM backend of LLVM.

Then it was Michael's talk - Digital Preservation - The National Archives of Australia, Open Standards and Open Source. He is a very entertaining speaker and gave a great overview of what they have developed at NAA. It was interesting to discover the actual physical setup of the NAA digital archive is 3 separate systems:

• Quarantine - where the digital objects submitted are virius scanned, then stored for 28 days and scanned again.
• Preservation - where the digital objects media type is identified, then object is preserved as the original is base 64 encoded with metadata and as an open document format with metadata. This is all done by Xena.
  
• Digitial Archive - actually 2 systems, one running Windows Server OS and the other running Linux. Each system is connected to a RAID arrays from different vendors.
  

The most interesting thing about the physical setup is there is no network connection to the outside world and each system is protected by an "Air Gap Firewall". In other words, all data transfer between the system is via a data carrying device (USB Portable Disk Drive). I know it sounds strange but the reasons Michael gave were valid.

The final session I attended was Stewart Smith's - eat my data: how everybody gets file IO wrong
Again a very entertaining speaker, and there was alot of attendee participation. The gist of the talk was never assume the data you have just written has made it to the platter. The various issues were explained and solutions given. Maybe I should start using the XFS filesystem.

So looking forward to tomorrow.

Need to find time for LCA 2007

Linux conf 2007 is on this week and I have paid to attend. The only problem will be finding time to go, otherwise it will be the most expensive t-shirt I have ever purchased. The urge to attend LCA was induced while attending OSDC 2006 and meeting a number of interesting people who are speaking at LCA. So hopefully will be able to attend:

Digital Preservation - The National Archives of Australia, Open Standards and Open Source

Eat my data: how everybody gets file IO wrong


The Hao of testing


Concurrency and Erlang


Do it our way