Thinking Inside a Bigger Box

As I release more frequently, I start to focus on automating the actual process of deploying a release. One of the most powerful steps of automating deployment is to automatically upgrade the database schema.

This technique first saw mainstream use with the Ruby-on-Rails framework. Today, there are several mature tools that will help you organize and execute database changes (Scala Migrations, Ruby-on-Rails Migrations, dbdeploy, Liquibase). And if none fit you perfectly, it’s easy to create your own.

In my current project, we have rolled our own solutions for this:

• All changes to the database are stored as SQL-files that are packaged into the deployment unit (in our case, a WAR-file). These files will usually contain statements like “ALTER TABLE ADD COLUMN” and “CREATE TABLE”. To get the files executed in the right order, we name the files with an increasing sequence number, like 012-add_payment_type_to_customer.sql.
• Whenever the application is started, it looks for a table named “MIGRATIONS” in the database and creates it if it doesn’t exist.
• At startup, the application looks through the list of migration files it has been packaged with and sees which file names don’t have an entry in the MIGRATIONS table.
• The application executes all the scripts that haven’t been executed already. If any script fails to execute, it makes a note of the error in the MIGRATIONS table and refuses to start the application

We run the migration procedure every time we start up the application, whether it is in test or production. Even the JUnit tests that access the database will run any pending migrations before starting. The result is that any database change that we intent to roll out into production will at the very least be executed once on each developers private copy of the database, as well as once on the continuous integration server. By the time they get executed in a controlled testing environment, we’re pretty confident that they work as intended.

Some migration tools use a more friendly (and portable) syntax than SQL DDL statements. Many allow for rollback of migrations. Most don’t automatically execute the pending migrations on application start, but require a separate command to execute them.

Your first step towards automating database migrations is to make sure that every change in the database is represented by some sort of script and that all these scripts are versioned with the rest of your code. From there, you can improve your process when you notice a step in the process that seems to involve too much work or risk.

Automating the deployment process will reduce the need for documentation and the opportunity for errors during one of the most critical times in the project. It is especially important to reduce the possibility of miscommunication and mistyping if the people responsible for deployment are in a separate organizational unit, which often seems to be the case. Make their job as easy as possible!

If you want to release your code frequently, you have to automate the release process. If your software interacts with shared components or other applications, the release script may have to update shared configuration files. In this blog post, I recall when I solved this with a crontab file.

Read the rest of this entry »

If you want to release frequently, a problem you may encounter is that some features, even though functionally complete, don’t stand well on their own, but require other features to be valuable to the user. If you want to release the system in this state, you need a way to hide features. A Feature-on/off-switch is a simple idea for dealing with this.

A feature-on/off-switch is some mechanism to hide features from a system. A feature-on/off-switch must be able to remove menu items concerning the feature and also to prevent adventuresome users from accessing the feature. It may be as crude as commenting out code (not recommended!), to enabling the feature based on a complex set of conditions (also not recommended).

I’ve encountered features switches triggered by the following mechanisms:

• A configuration file or configuration database table tells the system whether to turn the feature on or off.
• The feature is turned on for users that have a specific role (typically something like BETA_TESTER)
• The feature is turned on when the system is deployed as /foo-preview, but not when the system is deployed as /foo
• The feature is turned on after a specific date. This may seem weird, but was a potential solution when we were waiting for a release of another system and operations-freeze during summer was in effect.

There are probably many more conditions you may use to trigger a feature-on/off-switch. Maybe some of my readers have good examples?

I’m working on a collection of patterns for early releases with Niklas Bjørnerstedt. Here are some of my thoughts based on this work.

In a few different projects, I’ve noticed that the idea of “where are we going” seems to go though a familiar pattern:

1. “The old system is the requirement document, just make the new one do the same things”. After a while, someone will realize that it’s rather pointless to replace a system with a new one that does the same thing, which leads to…
2. “Analyze the business processes and make the new system automate all decisions that a human used to make.” After a while, people start realizing that business rules are interpreted slightly different by different users and finding a consensus approach is hard. Besides, some of the decisions require human judgment. On top of this, progress towards implementing the business processes is much slower than expected. As a matter of fact, people are panicking as the project gets increasingly delayed, which leads to…
3. “Just do whatever the old system did, with whatever improvements are dead easy. Just get this damned thing out the door.” Even reducing the scope to just the “bare bones of the current system with minimal improvements” doesn’t seem to give sufficient progress. Or sufficient value to justify the project. So, finally, we arrive at…
4. “Can we just add a new piece of software that makes an existing business process easier. And repeat until the budget is spent.”

The sad conclusion is that the original goal of replacing the old system begins to appear further into the future. At the same time, the new system will realize some value to some stakeholder pretty soon thereafter. Maybe the first step towards a successful replacement project is giving up replacing the old system?

The good news is that with an iterative approach to the requirement process, my current project was able to go through all these steps in a couple of months. Which beats my previous record of a year of full burn rate in stage 1.