Questions On Repository Pollution, Website Software Releases, SQL Patching, and Derivitives Thereof.

January 20, 2007 7:40 PM

I'm doing some consulting as a release manager on a webapp. I recently got the programmers to start using subversion (by turning off their access to their old development areas...) and their own sandboxes, and releases go more smoothly than ever. But I'm getting configuration files that have to be individualized for environments leaking into the trunk, and I want to get my release process down to "type command. hit enter." What to do?

There are three or four files (including a .htaccess file) that need to be in subversion because we need to modify and track changes to *most* of the file, but not the user-specific settings in a few particular areas. I've put environment-dependent conditionals into most of those files, but there's that pesky .htaccess file, and those pesky sql changes....

This causes a big problem because the 'live' site can't easily be taken down for a few minutes to update it (high traffic e-commerce, no front controller to turn the site "off") and the updates invariably screw something up either through an SQL patch not being applied or a config file leak pointing people to a developer's password-protected sandbox instead of the live site.

I have tried hook scripts, but there aren't enough timings for me. There's things that need to be done before updates, for instance -- there is no pre-update hook.

I've read the SVN book and googled extensively, but can't seem to turn anything up-- and I *know* people are using source code repositories to manage web apps. How should I be managing this? How would you handle website software releases where source control is practiced, and updates to the live site need to be made with a finger snap? What's a better way to keep config files from polluting the main repository, without excluding them entirely?

Bonus question: What's a good way to package SQL *changes* (such as updating a particular column according to a formula, adding a column, removing a column...) in with a release in SVN so that they're installed automatically when the revision, or a later revision, is checked out?

posted by SpecialK to Computers & Internet (6 answers total)

The thing that comes to mind to me is to decouple it so that instead of just doing a "svn up" on the live files (i.e. just serving them directly from a checkout working dir) you write some small controller script that gets pinged whenever there's a checkin and marshals the changes from the repo to the live area. It could be as simple as getting a diff, filtering out hunks that shouldn't go live, and possibly scanning for special markers (i.e. you have a comment somewhere that it can parse to mean "apply such and such batched SQL statement before making this file live.)
posted by Rhomboid at 8:12 PM on January 20, 2007

My advice as a former release manager and build engineer is this: You need to conceptually separate the build process from the release process. That doesn't mean you can't automate them into one actual step in your daily workflow, but it does mean you need to consider that something like a configuration file that is deployment-dependent is going to need a build process. I'm used to a pretty large development team with a ridiculously large codebase, but the concepts can be applied to smaller projects.

Typically for something small like this, you're going to want to convert these configuration scripts into templates then write a little build.sh script to do a checkout, apply substitution to the templated files and any other fixup you need to do at build time (such as build numbering, stripping out the docs directory [you do have docs, right?], TODOs and such), and package the release for distribution. To this script you pass a parameter telling it what the deployment target is (dev, QA, staging, or production).

As for automatic SQL changes, this sounds like a fantastically bad idea that is just begging for disaster to strike at production release time. The best I've come up with is to have a sql/ directory into which scripts are checked in; they are packaged separately, routed to the DBA to be vetted and run.
posted by majick at 8:48 PM on January 20, 2007

There are several software packages that do more or less what you want. I use to work on a project called Wigwam that nicely manages things like local versions of .htaccess files and what have you. Unfortunately its development has slowed down recently; it still only supports CVS, for example, so it's less helpful than it could be given that you're using subversion. Still, running through its documentation and maybe code might give you some ideas about good ways to approach the problem conceptually.

Capistrano has many of the same goals, is actively developed and supported, and supports subversion. I don't know how well it deals with playpen-local files, but it's at least worth looking in to.

A conceptual archetype that may be useful for tracking and managing SQL changes is the "migration" functionality in Ruby on Rails. All SQL changes can be managed in code, allowing for relatively painless downgrades should problems occur, etc.
posted by corey at 9:08 PM on January 20, 2007

We use wigwam. (I work with some of the guys who wrote it, but not corey. Small world.)
posted by adamk at 1:44 AM on January 21, 2007

Thanks, gents. Worth noting that I'm working in PHP, so Rails-specific apps give me some ideas but don't help me this time around.
posted by SpecialK at 6:42 AM on January 21, 2007

I second the recommendation for Capistrano.

Also, I should share a trick I've learned for dealing with sandbox-edited configuration files: if you've got, say, a config.ini that all your developers keep editing, instead of keeping the config file in SVN, use a config-example.ini file.

Then add an svn:ignore rule for config.ini; developers will copy config-example.ini, make local changes, but those can't be checked in because of the ignore rule.
posted by jacobian at 5:12 PM on January 21, 2007