Category Archives: SOA

Deliver early without sleepless nights

Trailblazing the first delivery of a software system requires courage and conviction, especially on projects that replace existing business critical software. When I’ve been acting as system architect I’ve employed a number of tricks in order to structure functionality and technical solutions in such a way that we can complete these early deliveries without sleepless nights. The most important is to find a subset of functionality that can be used with the rest being completed and investing in bridges from the old to the new.

Next Thursday (April 27th), I share my experience in the keynote for the ARK architecture conference in Oslo. A few tickets are still available if you want to make it.

In this blog post I explore the topic of replacing business critical software step by step.

As an architect I haven’t always have the joy of seeing my projects all the way to completion, but in those projects where I can see the software in the hands of my users early on are those I enjoy the most.

This article is not about streamlining a working delivery team to get from releases every few months to continuous deliveries. Many others talk about that. This is about how to get to production early the first time on a greenfield project. I will illustrate with two of my favorite projects.

Getting to the production the first time on a greenfield project takes courage and conviction, especially when you’re replacing part of business operations with a new system. Project stakeholders have many reasons to wait and often see no compelling reason to delivery a partial solution early. But when you get to production the first time, it’s like the sun has finally risen. The priorities and discussions in the project totally change. Now it’s real!

The first project I want to talk about was with the Norwegian electricity transmission system operator Statnett. We spent 4 years replacing the system that handles all reserve capacity of electricity deliveries on the Norwegian grid, but less than a year after the contract was signed, the users started using our software to control the power grid. Without any sleepless nights. Actually, they considered it such a non-event that they forgot to inform the development team about it!

The second project I would like to talk about is an app for internal mobile workers in private sector. As it is a project which is part of the competitive strategy of my customer, I cannot mention the purpose of the software, but I can describe techniques and technologies.

mobile workforce architecture

The mobile workforce app is especially interesting. Just like with Statnett, we were replacing an existing system in business use and integrated with other business functions. But in this project, we were live with the first users after 3 months, most of the user base after half a year and the remain users after around 9 months.

Here is what we did:

  • Realize that the journey is going to feel long and temporary investments made to ease the journey are often worth it. In both cases, I made sure the new system integrated well with the new system. I cannot stress how much it lowers everyone’s stress level when you can say: Why don’t we try the new system and if you experience problems, you can continue the very same process in your old system.
  • Isolate functionality that is small enough to deliver with a small fraction of the total project effort yet interesting enough to care about. In the case of Statnett we were able to replace the most used feature in the old system (while preserving the path of retreat). Even though the feature was a small part of the system, it was a large part of the usage. In the mobile workforce app, we found a business process that was missing in the old system and implemented this first. The process was interesting enough to the customer that they dedicated a few users to only perform this task.
  • Don’t wait for the rest of the world to complete their work. All projects I’ve been on has been part of a larger landscape of change where several of your dependencies are still under development by other projects. In the case of the mobile workforce app, we were dependent on core data that was to be owned by a system still under development. Realizing that our integration model would in any case be based on synchronizing copies of data, we decided that our copy of the core data would have several sources: 1. A manual CSV file exported by a sysadmin, 2. An automated dump from the legacy source which we established after half a year, 3. A feed from the new system (which is still not completed).
  • Spend effort making your system faster to deploy and easier to monitor. When you make a greenfield project, there is actually nothing that stops you from creating the production environment immediately (as long as you don’t put production data there before it’s hardened!). I’ve never regretted spending a few extra hours making it easier to deploy new version or making my logging smoother. Currently, I receive error logs on Slack with a clickable stack trace and a link to the user who experienced the problem.

When you do a major rehaul of a building, you may end up spending a lot of your effort setting up the scaffold to make sure the work is safe and effective. When you build a new software system, you should do the same!

If you do, you may be able to get users on your new system sooner rather than later without losing your sleep.

And when you get actual users to use your system the primary question of the project changes from “when is it all going to be done” to “what can we do next to deliver value.” And that’s much more fun.

Posted in Agile Release Patterns, English, SOA, Software Development | Leave a comment

Felles IKT-arkitektur for offentlig sektor

This Norwegian language post describes my response to the report from a task force exploring a common IT-architecture for the public sector in Norway.

Den norske regjeringen har besluttet at en felles IKT-arkitektur for offentlig sektor ville være fint. Jeg fikk greie på arbeidet på tirsdag, og har lest rapport til den store gullmedalje. Jeg er fortsatt ikke helt sikker på hva som menes med “felles IKT-arkitekt”, men jeg kan se omrisset av mange store evighetsprosjekter i dokumentet.

Jeg har forfattet et svar på rapporten.

Spesielt er jeg bekymret for at dette skal bli en unnskyldning for store SOA-prosjekter uten veldefinerte formål. Rapporten beskriver en god del ønskede effektmål, men disse beskrives i såpass runde former at man aldri kommer til å etterprøve om prosjekter faktisk oppfyller dem. Formålene er ting som økt grad av interoperabilitet.

Frykt nummer to ligger i hvordan SOA-krigen går om dagen. Rapporten nevner veldig lite konkret på teknologier, så jeg slapp unna å gå inn i dette minefeltet. I stedet må jeg ligge våken og frykte effekten av de usagte ordene i rapporten.

Jeg har startet å høre om dogmatiske SOA-prosjekter som ikke lykkes, til tross for store investeringer. REST starter å bli mer akseptabelt å snakke om i høflig selskap. Men fremdeles virker det som om at “SOA prosjekt” er et kodeord for “vi skal kjøpe dyr WS-* programvare fra Oracle, IBM eller Microsoft”.

Jeg skulle ønske jeg hadde mer erfaring innen offentlig sektor før jeg skulle uttale meg om dette, men dersom det er som noe annet jeg har sett, handler interoperabilitet grunnleggende sett om å sende eller motta informasjon på et standardisert format. Det betyr tre ting: En aksjon (les, oppdater, lever), noe som identifiserer et objekt (søknad, personopplysning) og et innhold (“Jeg ønsker med dette herved å søke om bla bla bla”).

Og her har WS-* skadet oss mye. Når man bygde standarden SOAP (Simple Object Access Protocol, som verken er enkel eller handler om “object access”), skrellet man vekk to fine deler av HTTP standarden: Verb (GET, POST, PUT – det vil si “aksjoner”) og URL’er (det vil si den nøkkelen som representerer et objekt). Det betyr at vi ikke lenger kan si “legg til” (POST) en “søknad” (http://etat.no/tjeneste/soknad/) som inneholder noe data (ikke en del av standarden). I stedet må vi si “send en melding som inneholder informasjon om noe vi tenker å gjøre” (ikke en del av standarden).

I en rimelig verden burde interoperabilitet betydd at å utveksle informasjon om objekter burde være en del av en universell standard som er akseptert rundt hele verden. En standard basert på å sende meldinger om å utføre én eller annen oppgave (som for eksempel å utveksle et objekt) virker sørgelig utilstrekkelig. Og offentlig sektor i lille Norge kan vel ikke være den som brøyter vei her.

Så lenge SOA-krigen pågår er jeg ikke optimistisk for sjansene til at en rimelig standard dukker opp.

Posted in Norsk, SOA, Software Development, Technology | 5 Comments

Users judge your service by its interface

Sometimes it feels like I’m stating the obvious. But the fact that users only care about the interface of the service is something we often say, but seldom understand.

If this is true, how can we think that we can develop the interface as an after though to the central system.

Not all your users might be human. And the computerized users are treated even worse.

If the non-human users also care about the interface, how do we think that they will be satisfied with the same interface that is used internally between different parts of the application? These interfaces are the scraps of the service table.

If this is true, why do so many system diagrams display the “service” layer for external users below the user interface layer. It should be next to the human user interface layer, shouldn’t it?

External programmatic interfaces, whether they be SOAP, REST or otherwise, must be treated as any other user interface. This means both to understand the needs of the non-human users and to only develop parts of the service interface that are needed now.

Any user interface expert will tell you: There are no taking-backs with user interfaces. Once you’ve given something to the user, they will react very negative to substantial changes. The non-human user is even more unforgiving. Yet, the scraps of the system internals that we give them will be even more necessary to change.

If the users judge your service by its interface, we should only give non-human users interfaces that are developed based on their requirements, and that we don’t mind never getting to change.

Posted in SOA, Software Development | 3 Comments

The Myth of the Silo

Warning: This article requires a lot of editing love before it is very useful. It might be somewhat incoherent. Read at your own risk. ;-)

Silo (software): A silo system cannot easily integrate with any other system.

In software, the term “silo” is used to refer to a system that is constructed as one unit from the front end to the data storage. Everything is tied together to work with the rest of the silo, but not with other elements.

This is considered a bad thing.

The problem comes when we want to integrate the system with other systems or reuse parts of the system.

Many of the new ideas in software development has as one of their big goals to try and rectify the silo problem. In general this is achieved by splitting up the system into services that may or may not be distributed across different computers.

But the badness of the silo hinges on two claims: First, that the applications built as an integrated stack cannot be integrated with, and second, that a system of distributed services can be integrated with.

(more…)

Posted in SOA, Software Development | 4 Comments

Fire påstander om SOA

This article is a Norwegian-language version of my article Four bold claims about SOA.

Dette er et utkast til en artikkel jeg ønsker å få publisert. Jeg setter stor pris på tilbakemeldinger om uklare tanker og formuleringer.

To av de vanskeligste problemene vi møter innen programvareutvikling er integrasjon og det som gjerne kalles “business-IT alignment” eller forretningsorientering, altså: IT skal understøtte virksomhetens strategiske mål.

Tjenesteorientert arkitektur, eller Service Oriented Architecture (SOA), hevder å kunne løse begge disse problemene. I det siste har jeg forsøkt å lytte mer aktivt til påstander fra evangelister av SOA, og jeg begynner å få en forståelse for hva det er SOA foreslår som løsningen på problemene vi står overfor. Dette er min oppfatning av påstandene om hvordan SOA skal løse problemene med integrasjon og forretningsorientering:

  • Web service-standarder vil løse de tekniske integrasjonsproblemene (“WS-stjerne-påstanden”)
  • Å sentralisere integrasjon via ett knutepunkt vil løse de organisasjonsmessige utfordringene rundt integrasjon (“ESB-påstanden”)
  • Å modellere funksjonalitet som en arbeidsflyt mellom tjenester vil gi oss bedre forretningsorientering (“BPM-påstanden, del 1”)
  • Å kunne restrukturere arbeidsflyten mellom tjenestene vil gi oss en en smidig forretningslogikk (“BPM-påstanden, del 2”)

(more…)

Posted in Norsk, SOA, Software Development | 14 Comments

Link: Open Source in the Enterprise

CIO JP Rangaswami at investment bank Dresder Kleinwort Wasserstein talks about why he considers open source a corporate IT asset. In this talk, Rangaswami describes how DrKW wanted to create an internal incubator environment in order to combat skill attrition in the late 90s. In the course of doing this, they acquired OpenAdaptor and discovered almost accidentally benefits of the open source development model.

(more…)

Posted in Links, SOA, Software Development | 4 Comments

A Brief Adventure with Universal Repositories and REST Web Services

Inspired by Per Mellqvist (and myself, to be fair), I wanted to explore the possibility of using a generic DAO or Repository interface for REST. Based on this simple idea, I was able to create a very cute and testable prototype of a full Web Service stack for REST based Web Services. The most interesting aspect was creating a universal test case for Repositories.

This article shows how little code is required to implement and test a REST based Web Service in Java, despite the horror of the Java HTTP client API. The source code can be downloaded from my subversion repository. I also want to illustrate how to create black box tests that can be reused efficiently with different implementations of a Repository.

(more…)

Posted in Java, SOA | 14 Comments

CRUD, REST, DDD, Rails – these are a few of my favorite things

Some time back, I watched a video David Heinemeier Hansson give a talk on ActiveResource on RailsConf. The thing that struck me is how much Rails’ ideas are connected to those of Domain-Driven Design. Watching DHH is like seeing a version of Eric Evans on speed.

(more…)

Posted in Ruby-on-Rails, SOA, Software Development | 7 Comments

On Integration: Consolidated View

In my last post, I wrote about four integration scenarios using databases: Reference data, Consolidated view, Subscription and Publishing. Of these, the Consolidated View scenario requires the most interaction between the server and the client roles. This post will examine how to make the pieces fit together.

Consolidated view joins the data of multiple clients into a consolidated view. This makes you able to create administrative applications that span a set of subapplications without having to change the central view when a new application joins the mix.

(more…)

Posted in SOA, Software Development | Leave a comment

On Integration: Organizing the data

In my last post on using the database for integration, I argued that the best metaphor for creating systems that are interconnected is that of One-Large Database. Carl-Henrik asked some very relevant questions about this, which I will interpret (for now) mainly as “how do you avoid drowning in complexity”. This post will address the issue of maintainability, especially when things grow to be large.

The On Large Database metaphor is mostly useful as a starting point. The first thing I notice when I look at our systems that have been organized this way is the fact that the data-structure is far from flat. Each application will deal with four categories of tables. First, an application has a large private domain, containing tables that the rest of the world has no business messing with. Second, it exports some tables that other applications use. Third, it imports tables exported by other applications. And finally, there are some tables that are shared by a number of applications with no clear owner. The last case is something we normally want to avoid.

In my experience, the integration scenarios can further be divided into four cases: Reference data, Consolidated data, Published data, and Subscription data.

(more…)

Posted in SOA, Software Development | Leave a comment