Category Archives: Agile Release Patterns

Deliver early without sleepless nights

Trailblazing the first delivery of a software system requires courage and conviction, especially on projects that replace existing business critical software. When I’ve been acting as system architect I’ve employed a number of tricks in order to structure functionality and technical solutions in such a way that we can complete these early deliveries without sleepless nights. The most important is to find a subset of functionality that can be used with the rest being completed and investing in bridges from the old to the new.

Next Thursday (April 27th), I share my experience in the keynote for the ARK architecture conference in Oslo. A few tickets are still available if you want to make it.

In this blog post I explore the topic of replacing business critical software step by step.

As an architect I haven’t always have the joy of seeing my projects all the way to completion, but in those projects where I can see the software in the hands of my users early on are those I enjoy the most.

This article is not about streamlining a working delivery team to get from releases every few months to continuous deliveries. Many others talk about that. This is about how to get to production early the first time on a greenfield project. I will illustrate with two of my favorite projects.

Getting to the production the first time on a greenfield project takes courage and conviction, especially when you’re replacing part of business operations with a new system. Project stakeholders have many reasons to wait and often see no compelling reason to delivery a partial solution early. But when you get to production the first time, it’s like the sun has finally risen. The priorities and discussions in the project totally change. Now it’s real!

The first project I want to talk about was with the Norwegian electricity transmission system operator Statnett. We spent 4 years replacing the system that handles all reserve capacity of electricity deliveries on the Norwegian grid, but less than a year after the contract was signed, the users started using our software to control the power grid. Without any sleepless nights. Actually, they considered it such a non-event that they forgot to inform the development team about it!

The second project I would like to talk about is an app for internal mobile workers in private sector. As it is a project which is part of the competitive strategy of my customer, I cannot mention the purpose of the software, but I can describe techniques and technologies.

mobile workforce architecture

The mobile workforce app is especially interesting. Just like with Statnett, we were replacing an existing system in business use and integrated with other business functions. But in this project, we were live with the first users after 3 months, most of the user base after half a year and the remain users after around 9 months.

Here is what we did:

  • Realize that the journey is going to feel long and temporary investments made to ease the journey are often worth it. In both cases, I made sure the new system integrated well with the new system. I cannot stress how much it lowers everyone’s stress level when you can say: Why don’t we try the new system and if you experience problems, you can continue the very same process in your old system.
  • Isolate functionality that is small enough to deliver with a small fraction of the total project effort yet interesting enough to care about. In the case of Statnett we were able to replace the most used feature in the old system (while preserving the path of retreat). Even though the feature was a small part of the system, it was a large part of the usage. In the mobile workforce app, we found a business process that was missing in the old system and implemented this first. The process was interesting enough to the customer that they dedicated a few users to only perform this task.
  • Don’t wait for the rest of the world to complete their work. All projects I’ve been on has been part of a larger landscape of change where several of your dependencies are still under development by other projects. In the case of the mobile workforce app, we were dependent on core data that was to be owned by a system still under development. Realizing that our integration model would in any case be based on synchronizing copies of data, we decided that our copy of the core data would have several sources: 1. A manual CSV file exported by a sysadmin, 2. An automated dump from the legacy source which we established after half a year, 3. A feed from the new system (which is still not completed).
  • Spend effort making your system faster to deploy and easier to monitor. When you make a greenfield project, there is actually nothing that stops you from creating the production environment immediately (as long as you don’t put production data there before it’s hardened!). I’ve never regretted spending a few extra hours making it easier to deploy new version or making my logging smoother. Currently, I receive error logs on Slack with a clickable stack trace and a link to the user who experienced the problem.

When you do a major rehaul of a building, you may end up spending a lot of your effort setting up the scaffold to make sure the work is safe and effective. When you build a new software system, you should do the same!

If you do, you may be able to get users on your new system sooner rather than later without losing your sleep.

And when you get actual users to use your system the primary question of the project changes from “when is it all going to be done” to “what can we do next to deliver value.” And that’s much more fun.

Posted in Agile Release Patterns, English, SOA, Software Development | Leave a comment

Er IT-prosjektenes tid forbi?

Man kan lese fra moderne tanker på IT-utvikling at prosjekter er en avleggs arbeidsform. For de som har erfaring med utviklingsaktiviteter innen offentlig og privat sektor kan dette virke som en rar påstand. De aller fleste ser behovet for å unngå store prosjekter, men er prosjektet som arbeidsform virkelig avleggs?

Det spør naturligvis på hva man mener: I alle de årene jeg har jobbet med IT-utvikling er det en ting som er sikkert. Arbeidet endrer seg til det bedre når man går fra å bygge noe på laben og over til å lansere programvaren for brukere som benytter seg av den til daglig. Før dette tidspunktet er “kvalitet” en subjektiv og spekulativ egenskap, “fremdrift” er et begrep som må kvalifiseres med mange forbehold og formålet til hele initiativet er fortatt en hypotese der alle har sin egen tolkning.

I dag er den ingen gode grunner til å levere programvare gjennom et prosjekt som sitter flere kvartal eller år og bygger noe som så skal være klart til en stor lanseringsdag. I den betydningen er prosjekter en avleggs modell.

Men noen ting har ikke endret seg: Mange organisasjoner vil ha behov for et team som er større i en periode – et prosjektteam. Mange organisasjoner kunne ha verdi av leverandører som kan ta ansvar for at det arbeidet som gjøres er av god kvalitet og har en rimelig kostnad.

Men organisasjonen gjør smart i å koble dette prosjektteamet tett opp mot den mer permanente organisasjonen. Linjeorganisasjonen kjenner behovet og rammevilkårene bedre enn den midlertidige organisasjonen og vil ta over forvaltningen av produktet på sikt.

Og nye produkter vil ikke lanseres dager eller en gang uker etter at de har blitt påbegynt. Ved å levere så ofte som hver uke eller endog flere ganger om dagen kan organisasjoner heve kvaliteten og redusere arbeidet og bekymringen i forbindelsen med nye leveranser. Men det er ikke gjort på en dag. Dersom man skal etablere et nytt prosjekt kan man med fordel drifte dette (for eksempel i en skybasert løsning) allerede etter få dagers utviklingsjobb. Men det vil ta noen måneder før man har nok funksjonalitet til at man ønsker å invitere noen utenfor prosjektet til å bruke løsningen.

Er IT-prosjektenes tid forbi? Jeg håper vi ikke lenger vil se initiativene som bruker titalls eller hundretalls millioner kroner før man lanserer noe for brukere. Men organisasjoner vil fortsatt måtte benytte seg av økt bemanning i satsningsperioder. Og nye produkter bygges ikke på en dag før de kan lanseres.

Posted in Agile Release Patterns, Non-technical, Norsk, Software Development | Leave a comment

“Slice!” Making meaningful progress visible

What if you had to report daily your progress on the tasks you’re programming in your project. Wait, you say: “I already do that in my daily standup meetings“. But if your standup meeting is anything like most standup meetings out there, you’ve got a serious blind spot.

What if I said that writing code doesn’t constitute progress. Code is effort, not value. In order to demonstrate value, you have to be able to show your progress to someone who doesn’t care about code. You have to demonstrate actual functionality, all the time.

In this blog post, I want to explore what happens when you demand that you plan and develop your project solely based on slices of functionality with business value, and what happens if you demand that these slices should become ever thinner: You know where you’re going, your customer knows how far you’ve come, and you’re always ready to ship.

Elephant Carpaccio

The more frequently you want to demonstrate progress, the more aggressive you have to be about how you break up your tasks. The most extreme version is perhaps the one demonstrated by Alistair Cockburn in his Elephant Carpaccio exercise (video). In it, the workshop participants are asked to develop a software system that does the following: “Register the unit price, quantity and US state of a purchase. Produce the total price, tax amount (state dependent) and discount (%-rate depends on total price). If there’s time, also produce reports of all sales per month and per state.”

The goal of the exercise is to divide this task into as many small feature slices as possible. In a 90 minute workshop, the teams each complete five iterations with a demo of new functionality at the end of each. In the video, you can hear teams frequently shout “Slice!” as they perform an internal demo of a functional slice several times in each iteration.

To complete the exercise, the participants develop a battle plan for how to deliver the smallest possible feature slice. For example: Input a sale, calculate and display the total price. “Slice!” Calculate the sales tax for a single state. “Slice!” Calculate the sales tax for three states. “Slice!” Introduce a table with tax rates to calculate the tax rate for any state. “Slice!” Give 3% discount on sales above $1000. “Slice!” And so on.

When you look to slice up the stories on your product backlog into tasks you should complete the next iteration, a few of the same approaches can be useful: If you need a screen to register some data, how about starting with registering just one field? “Slice!” And then register all fields, but without layout or validation. “Slice!” And then add validation. “Slice!” And then create a good layout. “Slice!”

If you split your next story into feature slices, your customer is probably going to be much more interested on your stand-up. If she learns of something she can test right after the stand-up meeting is done, she will probably feel much better about your progress as well.

If you create a carpaccio-style battle plan for how you want to solve the next story on your backlog, you will probably notice the old adage: “Plans are useless, but planning is essential.” Even though you have to rewrite your development battle plan, it will help you to know what to do and to always make progress towards a meaningful next goal post.

Finally, if the tasks you complete all mean progress towards the business goal of the project, you’re always ready to ship. Even if a vicious flu takes out the team for half an iteration, Christmas sneaks up on you as it always does, or, god forbid, the project runs out of money: You will have something you can ship.

One word of warning, though: Alistair Cockburn reports that during the Elephant Carpaccio exercise, people create crappy code. If you want to be able to deliver tomorrow as well as today, you must avoid the temptation of forgetting good engineering practices in order to ship something earlier. Start each slice with a bit of refactoring to get the code ready for a clean addition of functionality. Always write tests for your slices. And keep the code clean: You code may not do everything yet, but what it does do, it should do properly.

Are you ready to create a sprint plan that’s meaningful both to the developers and the product owner? How thin can you slice that elephant?

Posted in Agile Release Patterns, English, Extreme Programming, Software Development | 2 Comments

Database refactoring: Replace table with view

When working on replacement projects, I often find I need to make minor changes to an existing database that is still in use by one of several other applications. Initially, it may seem like situation will force you to conform to the current database schema. But there are other options, even though they may not be for those who are faint of heart.

The general pattern when you want to evolve a database that is in use by legacy system, is to make sure that the legacy system sees the same data structure when it reads from or writes to the database. You can achieve this by using database views and “instead-of” triggers that will execute custom SQL code when insert, update or delete statements for the view are executed.

Here is an example: I was once given the challenge of centralizing a database that the customer currently installed one per site. The rub: I could not change the current application. And the new solution should treat some information as shared across the sites and some information as exclusive per site.

Here is how I solved the problem for an example table (let’s say CUSTOMER, with shared column BONUS_LEVEL and site-exclusive column LAST_VISIT):

  1. I created the tables CUSTOMER_SHARED, with column BONUS_LEVEL, and CUSTOMER_SITE with columns SITE and LAST_VISIT
  2. I populated the tables with the union of the current site databases.
  3. I dropped the existing CUSTOMER table
  4. I created a new CUSTOMER view as “SELECT … FROM customer_site INNER JOIN customer_shared WHERE customer_site.site = “. Now, the existing applications can see, but not update data.
  5. I created “instead-of triggers” for insert, update and delete on customer. The existing applications can now update data as if nothing had changed.

Instead-of triggers is a very handy mechanism for views. They are executed when insert, update or delete statements are executed on the table. Some examples of such a triggers are:

This sort of transformation is not without risk, and should be carefully tested and scripted to make sure it behaves the same when you test and when you deploy it.

A final word of warning: Both the function and the performance of instead-of triggers can depend a lot on your database vendor. Make sure you test a solution that use these features with a realistic amount of data.

I’d like to hear from readers who either found the SQL code intimidating or enlightening. I’d really like to know whether my readers are as frightened by SQL-code as non-programmers are by Java. ;-)

So remember: When you’re faced with a seemingly impossible task: Think inside a bigger box!

Posted in Agile Release Patterns, English, Software Development | 10 Comments

Cleaning up the release process

How many steps do you need to perform to release a new version of your software? Do you even know?

Releasing frequently requires the release process to be as streamlined as possible. A good way to get started is to write a step-by-step instruction that explicitly state everything that needs be performed for an installation or upgrade.

Then get to work cleaning it up:

  • Make sure you can build and package everything that’s needed for the installation with a single command
  • Does the installation or upgrade require several files to be installed? Bundle them in a ZIP-file, MSI-installer or RPM. If you’re installing to an application server, find out how to script deployment to your particular server.
  • Does the installation require files to be copied, moved, linked or otherwise managed? Provide a script that automatically executes the necessary actions
  • Are there dependencies that you assume are already installed in the environment? State these explicitly, or even better, internalize them, to reduce variation
  • Does changing the database schema require a separate step? Why not bundle database changes in the application and execute them as part of the startup-procedure? (see my blogpost on database migrations)

Most projects can get down to upgrading the system safely by executing a single command. But as long as upgrading is a manual and complicated procedure, it is risky to release frequently. Clean up your release process to make it foolproof.

Posted in Agile Release Patterns, English | 2 Comments

Agile Release Pattern: Support multiple versions

No computer system is an island. At least not these days. This creates a challenge when you want to change APIs: Do you want to upgrade all affected systems at once, or do you want to support multiple versions?

Both approaches have downsides. Coordinating several upgrades increase the risk that one of the upgrades fail, in which case, you may want to roll everything back. In some situations, rollback may not even be possible. So for everything except the simplest cases, a distributed system must expect to support multiple versions of APIs.

For high-risk upgrades within a system, it may still be useful to support multiple version. A common example is database changes. Many database changes are impossible to rollback once they are done. If these changes are made at the same time as a new version is put into production, the result may be disastrous: Imagine that the new version contains some serious defect that makes it unusable. If the accompanying database change cannot be rolled back, the consequence of this defect will be severe indeed.

A special case of multiple versions of an interface is that of a backwards compatible interface. In SQL, adding a new table or adding non-required columns to an existing table is backwards compatible. With web services, adding a new element may be backwards compatible, depending on the XSD that defines the web service. In Java, adding a method to an existing class is backwards compatible.

Backwards compatible changes require very little coordination, especially if clients can stay with the old changes if they don’t care about the new features.

When your changes are not backwards compatible, having to support clients on multiple versions will add to the maintenance cost of the system. You will want to support as few versions as possible and give clients incentives to upgrade to a new version. Incentives can be anything from asking nicely, to providing a cut-off date (if you can live without your clients) or hike up the service fee (if you have one).

The long and short of it is that you want to avoid interfaces where you have to manage the upgrade and versioning. This is why SOA projects that split up the systems in too fine grained services suddenly discover the need for a bureaucratic service governance effort. Once you’ve published an interface, you’re stuck with it forever.

Posted in Agile Release Patterns, English | Leave a comment

Agile Release Pattern: Database migrations

As I release more frequently, I start to focus on automating the actual process of deploying a release. One of the most powerful steps of automating deployment is to automatically upgrade the database schema.

This technique first saw mainstream use with the Ruby-on-Rails framework. Today, there are several mature tools that will help you organize and execute database changes (Scala Migrations, Ruby-on-Rails Migrations, dbdeploy, Liquibase, dbmaintain). And if none fit you perfectly, it’s easy to create your own.

In my current project, we have rolled our own solutions for this:

  • All changes to the database are stored as SQL-files that are packaged into the deployment unit (in our case, a WAR-file). These files will usually contain statements like “ALTER TABLE ADD COLUMN” and “CREATE TABLE”. To get the files executed in the right order, we name the files with an increasing sequence number, like 012-add_payment_type_to_customer.sql.
  • Whenever the application is started, it looks for a table named “MIGRATIONS” in the database and creates it if it doesn’t exist.
  • At startup, the application looks through the list of migration files it has been packaged with and sees which file names don’t have an entry in the MIGRATIONS table.
  • The application executes all the scripts that haven’t been executed already. If any script fails to execute, it makes a note of the error in the MIGRATIONS table and refuses to start the application

We run the migration procedure every time we start up the application, whether it is in test or production. Even the JUnit tests that access the database will run any pending migrations before starting. The result is that any database change that we intent to roll out into production will at the very least be executed once on each developers private copy of the database, as well as once on the continuous integration server. By the time they get executed in a controlled testing environment, we’re pretty confident that they work as intended.

Some migration tools use a more friendly (and portable) syntax than SQL DDL statements. Many allow for rollback of migrations. Most don’t automatically execute the pending migrations on application start, but require a separate command to execute them.

Your first step towards automating database migrations is to make sure that every change in the database is represented by some sort of script and that all these scripts are versioned with the rest of your code. From there, you can improve your process when you notice a step in the process that seems to involve too much work or risk.

Automating the deployment process will reduce the need for documentation and the opportunity for errors during one of the most critical times in the project. It is especially important to reduce the possibility of miscommunication and mistyping if the people responsible for deployment are in a separate organizational unit, which often seems to be the case. Make their job as easy as possible!

Posted in Agile Release Patterns, Software Development | 8 Comments

Agile release pattern: Feature-on/off-switch

If you want to release frequently, a problem you may encounter is that some features, even though functionally complete, don’t stand well on their own, but require other features to be valuable to the user. If you want to release the system in this state, you need a way to hide features. A Feature-on/off-switch is a simple idea for dealing with this.

A feature-on/off-switch is some mechanism to hide features from a system. A feature-on/off-switch must be able to remove menu items concerning the feature and also to prevent adventuresome users from accessing the feature. It may be as crude as commenting out code (not recommended!), to enabling the feature based on a complex set of conditions (also not recommended).

I’ve encountered features switches triggered by the following mechanisms:

  • A configuration file or configuration database table tells the system whether to turn the feature on or off.
  • The feature is turned on for users that have a specific role (typically something like BETA_TESTER)
  • The feature is turned on when the system is deployed as /foo-preview, but not when the system is deployed as /foo
  • The feature is turned on after a specific date. This may seem weird, but was a potential solution when we were waiting for a release of another system and operations-freeze during summer was in effect.

There are probably many more conditions you may use to trigger a feature-on/off-switch. Maybe some of my readers have good examples?

Posted in Agile Release Patterns, Software Development | 8 Comments

Staggering toward the project goal

I’m working on a collection of patterns for early releases with Niklas Bjørnerstedt. Here are some of my thoughts based on this work.

In a few different projects, I’ve noticed that the idea of “where are we going” seems to go though a familiar pattern:

  1. “The old system is the requirement document, just make the new one do the same things”. After a while, someone will realize that it’s rather pointless to replace a system with a new one that does the same thing, which leads to…
  2. “Analyze the business processes and make the new system automate all decisions that a human used to make.” After a while, people start realizing that business rules are interpreted slightly different by different users and finding a consensus approach is hard. Besides, some of the decisions require human judgment. On top of this, progress towards implementing the business processes is much slower than expected. As a matter of fact, people are panicking as the project gets increasingly delayed, which leads to…
  3. “Just do whatever the old system did, with whatever improvements are dead easy. Just get this damned thing out the door.” Even reducing the scope to just the “bare bones of the current system with minimal improvements” doesn’t seem to give sufficient progress. Or sufficient value to justify the project. So, finally, we arrive at…
  4. “Can we just add a new piece of software that makes an existing business process easier. And repeat until the budget is spent.”

The sad conclusion is that the original goal of replacing the old system begins to appear further into the future. At the same time, the new system will realize some value to some stakeholder pretty soon thereafter. Maybe the first step towards a successful replacement project is giving up replacing the old system?

The good news is that with an iterative approach to the requirement process, my current project was able to go through all these steps in a couple of months. Which beats my previous record of a year of full burn rate in stage 1.

Posted in Agile Release Patterns, English, Extreme Programming, Software Development | 3 Comments