Thinking Inside a Bigger Box

I’ve published a Norwegian language article titled: “Min supre produkteier” (“My excellent product owner”) at the company blog for Steria Norway. Go check it out if you understand Norwegian!

The more I code, the more I’ve learned to appreciate keeping the code clean even during complex refactorings. By “clean”, I mean that the code always compiles and the test always run.

I often find myself in a situation where I have a method call that’s starting to accumulate parameters. Something like this:

showPersonCreateForm(writer, firstName, firstNameErrorMessage, lastName, lastNameErrorMessage,....);

After three or four parameters, the need to refactor is starting to become evident. I would rather have something like this:

CreatePersonForm form = new CreatePersonForm();
form.setFirstName(firstName);
form.setFirstNameErrorMessage(firstName);
form.setLastName(firstName);
form.setLastNameErrorMessage(firstName);
form.show(writer);

This is one of the more complex simple refactorings you can make, and it requires several steps. In this five minute video, I show how to perform such a refactoring without any steps that break my code:

The screencast was created using the free BB FlashBack Express on Windows. All the magic you see happening while I program is either ctrl-space (complete) or ctrl-1 (quick fix).

Can you modify your code without going thought long stages of nothing working? I think you can!

I’ve published a Norwegian language article titled: “Slik lykkes du med smidig utvikling” (“How to succeed with agile development”) at the company blog for Steria Norway. Go check it out if you understand Norwegian!

As an architect for a team with a large number of people, I have a couple of problems:

• I often make decisions that turns out to be quite crappy.
• Even when I think I’ve written or drawn something that’s smart, it often turns out that it’s incomprehensible to everyone else

Luckily, I’ve noticed that most developers have characteristics that almost always counter these weaknesses:

• Most developers are pretty smart, especially when they’re trying to solve a specific problem.
• Most developers don’t read architecture documentation

So instead of trying to force the developers to read my crappy, poorly documented decisions, I decided to try to see what happened if I instead made use of the opportunities that the situation presented to me. Enter: The Great Wall of Architecture.

There are two rules for the Great Wall of Architecture:

1. Only the pictures on the Great Wall of Architecture are officially part of the system documentation.
2. The architect (me) cannot draw pictures to be included on the Great Wall of Architecture

As a playful addition, some of the developers have signed their work, just like we used to do in kindergarten. You can’t actually see it on the photo, but the hand writing on this diagram reads “made by Christin, 32 years old”.

My hope is that the Great Wall of Architecture will make the whole team feel they can use their strengths on the project. As more drawings come up every day, I’m optimistic with the result.

What if you had to report daily your progress on the tasks you’re programming in your project. Wait, you say: “I already do that in my daily standup meetings“. But if your standup meeting is anything like most standup meetings out there, you’ve got a serious blind spot.

What if I said that writing code doesn’t constitute progress. Code is effort, not value. In order to demonstrate value, you have to be able to show your progress to someone who doesn’t care about code. You have to demonstrate actual functionality, all the time.

In this blog post, I want to explore what happens when you demand that you plan and develop your project solely based on slices of functionality with business value, and what happens if you demand that these slices should become ever thinner: You know where you’re going, your customer knows how far you’ve come, and you’re always ready to ship.

Elephant Carpaccio

The more frequently you want to demonstrate progress, the more aggressive you have to be about how you break up your tasks. The most extreme version is perhaps the one demonstrated by Alistair Cockburn in his Elephant Carpaccio exercise (video). In it, the workshop participants are asked to develop a software system that does the following: “Register the unit price, quantity and US state of a purchase. Produce the total price, tax amount (state dependent) and discount (%-rate depends on total price). If there’s time, also produce reports of all sales per month and per state.”

The goal of the exercise is to divide this task into as many small feature slices as possible. In a 90 minute workshop, the teams each complete five iterations with a demo of new functionality at the end of each. In the video, you can hear teams frequently shout “Slice!” as they perform an internal demo of a functional slice several times in each iteration.

To complete the exercise, the participants develop a battle plan for how to deliver the smallest possible feature slice. For example: Input a sale, calculate and display the total price. “Slice!” Calculate the sales tax for a single state. “Slice!” Calculate the sales tax for three states. “Slice!” Introduce a table with tax rates to calculate the tax rate for any state. “Slice!” Give 3% discount on sales above $1000. “Slice!” And so on.

When you look to slice up the stories on your product backlog into tasks you should complete the next iteration, a few of the same approaches can be useful: If you need a screen to register some data, how about starting with registering just one field? “Slice!” And then register all fields, but without layout or validation. “Slice!” And then add validation. “Slice!” And then create a good layout. “Slice!”

If you split your next story into feature slices, your customer is probably going to be much more interested on your stand-up. If she learns of something she can test right after the stand-up meeting is done, she will probably feel much better about your progress as well.

If you create a carpaccio-style battle plan for how you want to solve the next story on your backlog, you will probably notice the old adage: “Plans are useless, but planning is essential.” Even though you have to rewrite your development battle plan, it will help you to know what to do and to always make progress towards a meaningful next goal post.

Finally, if the tasks you complete all mean progress towards the business goal of the project, you’re always ready to ship. Even if a vicious flu takes out the team for half an iteration, Christmas sneaks up on you as it always does, or, god forbid, the project runs out of money: You will have something you can ship.

One word of warning, though: Alistair Cockburn reports that during the Elephant Carpaccio exercise, people create crappy code. If you want to be able to deliver tomorrow as well as today, you must avoid the temptation of forgetting good engineering practices in order to ship something earlier. Start each slice with a bit of refactoring to get the code ready for a clean addition of functionality. Always write tests for your slices. And keep the code clean: You code may not do everything yet, but what it does do, it should do properly.

Are you ready to create a sprint plan that’s meaningful both to the developers and the product owner? How thin can you slice that elephant?

When working on replacement projects, I often find I need to make minor changes to an existing database that is still in use by one of several other applications. Initially, it may seem like situation will force you to conform to the current database schema. But there are other options, even though they may not be for those who are faint of heart.

The general pattern when you want to evolve a database that is in use by legacy system, is to make sure that the legacy system sees the same data structure when it reads from or writes to the database. You can achieve this by using database views and “instead-of” triggers that will execute custom SQL code when insert, update or delete statements for the view are executed.

Here is an example: I was once given the challenge of centralizing a database that the customer currently installed one per site. The rub: I could not change the current application. And the new solution should treat some information as shared across the sites and some information as exclusive per site.

Here is how I solved the problem for an example table (let’s say CUSTOMER, with shared column BONUS_LEVEL and site-exclusive column LAST_VISIT):

1. I created the tables CUSTOMER_SHARED, with column BONUS_LEVEL, and CUSTOMER_SITE with columns SITE and LAST_VISIT
2. I populated the tables with the union of the current site databases.
3. I dropped the existing CUSTOMER table
4. I created a new CUSTOMER view as “SELECT … FROM customer_site INNER JOIN customer_shared WHERE customer_site.site = “. Now, the existing applications can see, but not update data.
5. I created “instead-of triggers” for insert, update and delete on customer. The existing applications can now update data as if nothing had changed.

Instead-of triggers is a very handy mechanism for views. They are executed when insert, update or delete statements are executed on the table. Some examples of such a triggers are:

CREATE OR REPLACE TRIGGER customer_UPD_TR
    INSTEAD OF UPDATE ON customer
FOR EACH ROW BEGIN
    UPDATE customer_site SET
         last_visit = :new.last_visit
    WHERE
         id = :new.id AND site = ;

    UPDATE customer_shared SET
         bonus_level = :new.bonus_level
    WHERE
         id = :new.id;
END;

CREATE OR REPLACE TRIGGER customer_DEL_TR
    INSTEAD OF DELETE ON customer
FOR EACH ROW BEGIN
    DELETE FROM customer_site
    WHERE
         id = :new.id AND site = ;

    DELETE FROM customer_shared
    WHERE not exists (
            SELECT * FROM customer_site
            WHERE customer_site.id = customer_shared.id);
END;

This sort of transformation is not without risk, and should be carefully tested and scripted to make sure it behaves the same when you test and when you deploy it.

A final word of warning: Both the function and the performance of instead-of triggers can depend a lot on your database vendor. Make sure you test a solution that use these features with a realistic amount of data.

I’d like to hear from readers who either found the SQL code intimidating or enlightening. I’d really like to know whether my readers are as frightened by SQL-code as non-programmers are by Java. ;-)

So remember: When you’re faced with a seemingly impossible task: Think inside a bigger box!

Everyone has heard horror stories about pointy-haired-bosses counting lines of source code to track the progress of a project. We roll our eyes and laugh at their stupidity. But before you laugh too much, you might want to find out whether you’re really any better.

Most of what software projects measure are not things we care about. Not things we really care about. Do you really care about the number of coding standard violations in your code? About compiler warnings? About test coverage? About cyclomatic complexity?

No, you only care about the presumed effects of these metrics. What we really care about are things like how much it costs to change the system, and how likely we are to introduce bugs. Hopefully, most of the numbers you collect with your fancy tool will help predict what your really care about. But tools can be fooled.

You can have 100% test coverage without asserting a single thing about your code. And you can have maintainable, bug free code with super-high cyclomatic complexity and not a single line of code comments.

A leading indicator is a term investors use for measurements that change before the economy changes. A company’s stock prices is an example of leading indicators.

On the other hand, the term lagging indicator is used for a measurement that change after the economic reality changes. A company’s reported earning is an example of a lagging indicator.

Leading indicators, like stock prices and code coverage are useful because you can get hold of them early. However, they are not the real thing we’re after. The real thing we’re after is often a lagging indicator, like reported earnings or code maintainability.

Use leading indicators on your project to identify trouble areas early. But don’t be religious about them. Use lagging indicators to prove that you’ve met your commitment to quality. If your commitment to quality is reduced to code coverage, cyclomatic complexity or compiler warnings, you’re no better than the pointy-haired-boss counting lines of code.

As I release more frequently, I start to focus on automating the actual process of deploying a release. One of the most powerful steps of automating deployment is to automatically upgrade the database schema.

This technique first saw mainstream use with the Ruby-on-Rails framework. Today, there are several mature tools that will help you organize and execute database changes (Scala Migrations, Ruby-on-Rails Migrations, dbdeploy, Liquibase, dbmaintain). And if none fit you perfectly, it’s easy to create your own.

In my current project, we have rolled our own solutions for this:

• All changes to the database are stored as SQL-files that are packaged into the deployment unit (in our case, a WAR-file). These files will usually contain statements like “ALTER TABLE ADD COLUMN” and “CREATE TABLE”. To get the files executed in the right order, we name the files with an increasing sequence number, like 012-add_payment_type_to_customer.sql.
• Whenever the application is started, it looks for a table named “MIGRATIONS” in the database and creates it if it doesn’t exist.
• At startup, the application looks through the list of migration files it has been packaged with and sees which file names don’t have an entry in the MIGRATIONS table.
• The application executes all the scripts that haven’t been executed already. If any script fails to execute, it makes a note of the error in the MIGRATIONS table and refuses to start the application

We run the migration procedure every time we start up the application, whether it is in test or production. Even the JUnit tests that access the database will run any pending migrations before starting. The result is that any database change that we intent to roll out into production will at the very least be executed once on each developers private copy of the database, as well as once on the continuous integration server. By the time they get executed in a controlled testing environment, we’re pretty confident that they work as intended.

Some migration tools use a more friendly (and portable) syntax than SQL DDL statements. Many allow for rollback of migrations. Most don’t automatically execute the pending migrations on application start, but require a separate command to execute them.

Your first step towards automating database migrations is to make sure that every change in the database is represented by some sort of script and that all these scripts are versioned with the rest of your code. From there, you can improve your process when you notice a step in the process that seems to involve too much work or risk.

Automating the deployment process will reduce the need for documentation and the opportunity for errors during one of the most critical times in the project. It is especially important to reduce the possibility of miscommunication and mistyping if the people responsible for deployment are in a separate organizational unit, which often seems to be the case. Make their job as easy as possible!

A software architect who doesn’t care about what his system is supposed to do isn’t worth his salt. For the term “software architect” to hold any meaning at all, it must be to describe someone who understands what the customer needs and designs a system that is fit for this purpose.

Sometimes, however, people talk about “technical architects”. I have myself been guilty of falling into this category once or twice myself. The fallacy of the “technical software architect” is that there is a large number of solutions that will work independent of what problem the customer have.

This is ludicrous. The very meaning of architecture is to apply technology to a context. Leave out the context, and you have no basis to choose a technology.

This is not to say that you don’t need experts within technologies and design strategies used in the solution. If the customer wants to connect a number of old and new systems into a shared portal, an Enterprise Service Bus (or a Data Warehouse!) may be a good solution. And then you will need people who are experts on the technology in question.

However, I would not call such an expert an architect. If you’re really good in some technology, I have no problem calling you a freaking good developer. Heck, you can even put that on your business card. Just don’t think you’re an architect.

If you want to release frequently, a problem you may encounter is that some features, even though functionally complete, don’t stand well on their own, but require other features to be valuable to the user. If you want to release the system in this state, you need a way to hide features. A Feature-on/off-switch is a simple idea for dealing with this.

A feature-on/off-switch is some mechanism to hide features from a system. A feature-on/off-switch must be able to remove menu items concerning the feature and also to prevent adventuresome users from accessing the feature. It may be as crude as commenting out code (not recommended!), to enabling the feature based on a complex set of conditions (also not recommended).

I’ve encountered features switches triggered by the following mechanisms:

• A configuration file or configuration database table tells the system whether to turn the feature on or off.
• The feature is turned on for users that have a specific role (typically something like BETA_TESTER)
• The feature is turned on when the system is deployed as /foo-preview, but not when the system is deployed as /foo
• The feature is turned on after a specific date. This may seem weird, but was a potential solution when we were waiting for a release of another system and operations-freeze during summer was in effect.

There are probably many more conditions you may use to trigger a feature-on/off-switch. Maybe some of my readers have good examples?