WSGI is my homeboy!

A sunday afternoon is always good for a surprising journey on the Internet. Today, I was looking into WSGI, after reading some Python-related websites. Combined with the fact that I was really upset about the new perli.net Codebase being slow and unstable, I decided to have a look at WSGI to fix my problems with multi-process mod_python and Python instances. I found the great tutorial "A Do-It-Yourself framework" by Ian Bicking.

I was quickly able to hack together a small WSGI-capable test application. Gladly, I have designed the perli.net Codebase so that the underlying (mod_python) is abstracted by a "Request" class I've written. This was a really good design decision, as I was able to easily adopt that class to work with WSGI instead of mod_python without having to look through the whole codebase. One the "Request" object (and some associated handling classes and functions) were updated to WSGI, I was able to start the test server (Paste's httpserver) and had the codebase running on WSGI! Great!

There were still some problems with SQLAlchemy, so after looking around on the web and asking on IRC, I did two things: Migrate from MyISAM to InnoDB (transaction-based SQL engine) and created a threading.Lock() object in the database access object and wrapped all database access calls between lock.acquire() and lock.release() calls. Now, SQLAlchemy is running much better and stable than before. Not only because of the changes I did, but also because the WSGI makes it possible to run one process with multiple threads (instead of multiple processes, where SQLAlchemy can't share objects between page requests).

To test my changes and to see if the server will stay stable with many concurrent requests, I installed Siege and pointed it at my local webserver. I was able to fix some miscellaneous bugs that cropped up while testing with Siege, and also had statistics on how fast a page would be generated. On my 2GHz (2GB RAM) MacBook, the first photo overview page loaded for about 6 to 15 seconds (HTML only), depending on load. This was way too long, so I thought about implementing some kind of cache.

I'm now caching certain pager data that can be cached and doesn't depend on users being logged in or not. This reduces the page load time to about 0.3 seconds (HTML only, again), so perli.net should be a bit faster on certain pages. A good space-time tradeoff, I think :)