Webapp rearchitecture
Some years ago, as I was dipping my toe into the broad landscape of web programming, I (as with many others) realized that there were certain advantages to having my own website to tinker with unbounded by the confines that a paying job places on one's work. With my own website, I could design what I wanted and try out whatever different technologies caught my fancy. Indeed, having one's own website, in addition to the obvious benefits of giving an individual a certain citizenship on the Internet, is somewhat akin to owning a project house -- you can pretty much build on it however you see fit, and whether it stands or falls is (largely) your own business.
2008
I was working at Meetup.com. Meetup at that point was just starting to peek out from under its Tomcat/JSTL roots, but was still built up as a monolithic Java app. My primary language was (and still is) Python, so, being as I hadn't built a website of any complexity before, I took my cues from what I knew and decided that I, too, would build a monolithic web app. Searching around through the Python-based frameworks available at the time, I settled on what seemed to be the most powerful and flexible of the ones I encountered: Cherrypy.
At the time, Cherrypy was probably a good decision. Cherrypy is a layered HTTP application framework, with abstractions ranging from full-fledged dispatchers down to a bare WSGI server. It's a flexible, object-oriented system that allows a very Java-esque style of programming, i.e., it encourages the subclassing and use of various system fundamentals (e.g. the Application itself). While other frameworks encourage subclassing certain bits and pieces to complement well-defined functionality (e.g. subclassing form processors in Django, Cherrypy really allows one to build a web app of any sort, from Blog to API to basic CDN, no holds barred.
Cherrypy was a great learning platform, and, along with some help from Cheetah and SQLObject (both abandoned now, it would appear), I wrote a fairly reasonable first-stab at a big, complex web application with lots of request-munging and inline API interfaces. Trouble was, I hadn't heard the piper's song of regression/unit testing yet, and so every deploy was one mis-click away from dumping a stacktrace to the page (in debug mode only, of course!). Still, I had my development and production environment isolation and a one-step deploy, and so this system lasted me pretty well for several years.
Today
Years passed with minimal changes to the basic application itself. I was using Tumblr via API calls as a CMS for my blog (and I still do, dammit!), Flickr via their API for my photos, and some other APIs just to show off (github and del.icio.us, then diigo when the latter ate the toothpaste), and so I was able to update content and do style tweaks here and there without too much fear of breakage. A few years and jobs on, I decided I wanted to dig back in and put up some new content. However, I really didn't want to re-learn the mess I'd written while getting the site up the first time, and the lack of regression tests terrified me.
So, seizing on the unparalleled opportunity which being master of one's own dominion offers, I decided to do a full rewrite. This time, however, I wanted something more robust and flexible, and so decided to go the loosely-coupled system of modules approach, meaning independent components linked via APIs or XMLRPC or whatever protocol I feel like using that day.
The structure
There is an ecosystem of (mostly) mature tools available now which readily support a distributed paradigm, like Puppet, Chef, Fabric, Salt, et al. In order to keep things flexible, and to allow myself creative room, I've settled on a basic routing layer which delegates requests to other components, retrieves the data, and returns the results. It's simple, robust, flexible.
Two tools that have really helped with this are Fabric for deployment and lxc Docker for application isolation (more on this in a future post). (An additional unsung hero is rsync, ever present and usually sidelined as "old reliable" these days.) Docker is a very young (<1 year) project, but it's partially derived from an existing production system (Dotcloud). These two projects allow rapid iteration of multiple components in lockstep, and, as a bonus, make it fairly simple to stage isolated testing environments, as well. (More on these details in a future post.)
Summary
The site itself is still (always) in development, but I think that this component-wise approach will allow an iterative system that will allow buggy modules to fail with jeopardizing the site as a whole. While not as declarative as it could be, Flask works well as a basic routing layer for a reasonably pluggable infrastructure. Docker is an amazing tool, allowing an entire system's-worth of dependencies to be easily constructed and deployed, over and over again.