Category Archives: SOA

Felles IKT-arkitektur for offentlig sektor

This Norwegian language post describes my response to the report from a task force exploring a common IT-architecture for the public sector in Norway.

Den norske regjeringen har besluttet at en felles IKT-arkitektur for offentlig sektor ville være fint. Jeg fikk greie på arbeidet på tirsdag, og har lest rapport til den store gullmedalje. Jeg er fortsatt ikke helt sikker på hva som menes med “felles IKT-arkitekt”, men jeg kan se omrisset av mange store evighetsprosjekter i dokumentet.

Jeg har forfattet et svar på rapporten.

Spesielt er jeg bekymret for at dette skal bli en unnskyldning for store SOA-prosjekter uten veldefinerte formål. Rapporten beskriver en god del ønskede effektmål, men disse beskrives i såpass runde former at man aldri kommer til å etterprøve om prosjekter faktisk oppfyller dem. Formålene er ting som økt grad av interoperabilitet.

Frykt nummer to ligger i hvordan SOA-krigen går om dagen. Rapporten nevner veldig lite konkret på teknologier, så jeg slapp unna å gå inn i dette minefeltet. I stedet må jeg ligge våken og frykte effekten av de usagte ordene i rapporten.

Jeg har startet å høre om dogmatiske SOA-prosjekter som ikke lykkes, til tross for store investeringer. REST starter å bli mer akseptabelt å snakke om i høflig selskap. Men fremdeles virker det som om at “SOA prosjekt” er et kodeord for “vi skal kjøpe dyr WS-* programvare fra Oracle, IBM eller Microsoft”.

Jeg skulle ønske jeg hadde mer erfaring innen offentlig sektor før jeg skulle uttale meg om dette, men dersom det er som noe annet jeg har sett, handler interoperabilitet grunnleggende sett om å sende eller motta informasjon på et standardisert format. Det betyr tre ting: En aksjon (les, oppdater, lever), noe som identifiserer et objekt (søknad, personopplysning) og et innhold (“Jeg ønsker med dette herved å søke om bla bla bla”).

Og her har WS-* skadet oss mye. Når man bygde standarden SOAP (Simple Object Access Protocol, som verken er enkel eller handler om “object access”), skrellet man vekk to fine deler av HTTP standarden: Verb (GET, POST, PUT – det vil si “aksjoner”) og URL’er (det vil si den nøkkelen som representerer et objekt). Det betyr at vi ikke lenger kan si “legg til” (POST) en “søknad” (http://etat.no/tjeneste/soknad/) som inneholder noe data (ikke en del av standarden). I stedet må vi si “send en melding som inneholder informasjon om noe vi tenker å gjøre” (ikke en del av standarden).

I en rimelig verden burde interoperabilitet betydd at å utveksle informasjon om objekter burde være en del av en universell standard som er akseptert rundt hele verden. En standard basert på å sende meldinger om å utføre én eller annen oppgave (som for eksempel å utveksle et objekt) virker sørgelig utilstrekkelig. Og offentlig sektor i lille Norge kan vel ikke være den som brøyter vei her.

Så lenge SOA-krigen pågår er jeg ikke optimistisk for sjansene til at en rimelig standard dukker opp.

Posted in Norsk, SOA, Software Development, Technology | 5 Comments

Users judge your service by its interface

Sometimes it feels like I’m stating the obvious. But the fact that users only care about the interface of the service is something we often say, but seldom understand.

If this is true, how can we think that we can develop the interface as an after though to the central system.

Not all your users might be human. And the computerized users are treated even worse.

If the non-human users also care about the interface, how do we think that they will be satisfied with the same interface that is used internally between different parts of the application? These interfaces are the scraps of the service table.

If this is true, why do so many system diagrams display the “service” layer for external users below the user interface layer. It should be next to the human user interface layer, shouldn’t it?

External programmatic interfaces, whether they be SOAP, REST or otherwise, must be treated as any other user interface. This means both to understand the needs of the non-human users and to only develop parts of the service interface that are needed now.

Any user interface expert will tell you: There are no taking-backs with user interfaces. Once you’ve given something to the user, they will react very negative to substantial changes. The non-human user is even more unforgiving. Yet, the scraps of the system internals that we give them will be even more necessary to change.

If the users judge your service by its interface, we should only give non-human users interfaces that are developed based on their requirements, and that we don’t mind never getting to change.

Posted in SOA, Software Development | 3 Comments

The Myth of the Silo

Warning: This article requires a lot of editing love before it is very useful. It might be somewhat incoherent. Read at your own risk. ;-)

Silo (software): A silo system cannot easily integrate with any other system.

In software, the term “silo” is used to refer to a system that is constructed as one unit from the front end to the data storage. Everything is tied together to work with the rest of the silo, but not with other elements.

This is considered a bad thing.

The problem comes when we want to integrate the system with other systems or reuse parts of the system.

Many of the new ideas in software development has as one of their big goals to try and rectify the silo problem. In general this is achieved by splitting up the system into services that may or may not be distributed across different computers.

But the badness of the silo hinges on two claims: First, that the applications built as an integrated stack cannot be integrated with, and second, that a system of distributed services can be integrated with.

(more…)

Posted in SOA, Software Development | 4 Comments

Fire påstander om SOA

This article is a Norwegian-language version of my article Four bold claims about SOA.

Dette er et utkast til en artikkel jeg ønsker å få publisert. Jeg setter stor pris på tilbakemeldinger om uklare tanker og formuleringer.

To av de vanskeligste problemene vi møter innen programvareutvikling er integrasjon og det som gjerne kalles “business-IT alignment” eller forretningsorientering, altså: IT skal understøtte virksomhetens strategiske mål.

Tjenesteorientert arkitektur, eller Service Oriented Architecture (SOA), hevder å kunne løse begge disse problemene. I det siste har jeg forsøkt å lytte mer aktivt til påstander fra evangelister av SOA, og jeg begynner å få en forståelse for hva det er SOA foreslår som løsningen på problemene vi står overfor. Dette er min oppfatning av påstandene om hvordan SOA skal løse problemene med integrasjon og forretningsorientering:

  • Web service-standarder vil løse de tekniske integrasjonsproblemene (“WS-stjerne-påstanden”)
  • Å sentralisere integrasjon via ett knutepunkt vil løse de organisasjonsmessige utfordringene rundt integrasjon (“ESB-påstanden”)
  • Å modellere funksjonalitet som en arbeidsflyt mellom tjenester vil gi oss bedre forretningsorientering (“BPM-påstanden, del 1”)
  • Å kunne restrukturere arbeidsflyten mellom tjenestene vil gi oss en en smidig forretningslogikk (“BPM-påstanden, del 2”)

(more…)

Posted in Norsk, SOA, Software Development | 14 Comments

Link: Open Source in the Enterprise

CIO JP Rangaswami at investment bank Dresder Kleinwort Wasserstein talks about why he considers open source a corporate IT asset. In this talk, Rangaswami describes how DrKW wanted to create an internal incubator environment in order to combat skill attrition in the late 90s. In the course of doing this, they acquired OpenAdaptor and discovered almost accidentally benefits of the open source development model.

(more…)

Posted in Links, SOA, Software Development | 4 Comments

A Brief Adventure with Universal Repositories and REST Web Services

Inspired by Per Mellqvist (and myself, to be fair), I wanted to explore the possibility of using a generic DAO or Repository interface for REST. Based on this simple idea, I was able to create a very cute and testable prototype of a full Web Service stack for REST based Web Services. The most interesting aspect was creating a universal test case for Repositories.

This article shows how little code is required to implement and test a REST based Web Service in Java, despite the horror of the Java HTTP client API. The source code can be downloaded from my subversion repository. I also want to illustrate how to create black box tests that can be reused efficiently with different implementations of a Repository.

(more…)

Posted in Java, SOA | 14 Comments

CRUD, REST, DDD, Rails – these are a few of my favorite things

Some time back, I watched a video David Heinemeier Hansson give a talk on ActiveResource on RailsConf. The thing that struck me is how much Rails’ ideas are connected to those of Domain-Driven Design. Watching DHH is like seeing a version of Eric Evans on speed.

(more…)

Posted in Ruby-on-Rails, SOA, Software Development | 7 Comments

On Integration: Consolidated View

In my last post, I wrote about four integration scenarios using databases: Reference data, Consolidated view, Subscription and Publishing. Of these, the Consolidated View scenario requires the most interaction between the server and the client roles. This post will examine how to make the pieces fit together.

Consolidated view joins the data of multiple clients into a consolidated view. This makes you able to create administrative applications that span a set of subapplications without having to change the central view when a new application joins the mix.

(more…)

Posted in SOA, Software Development | Leave a comment

On Integration: Organizing the data

In my last post on using the database for integration, I argued that the best metaphor for creating systems that are interconnected is that of One-Large Database. Carl-Henrik asked some very relevant questions about this, which I will interpret (for now) mainly as “how do you avoid drowning in complexity”. This post will address the issue of maintainability, especially when things grow to be large.

The On Large Database metaphor is mostly useful as a starting point. The first thing I notice when I look at our systems that have been organized this way is the fact that the data-structure is far from flat. Each application will deal with four categories of tables. First, an application has a large private domain, containing tables that the rest of the world has no business messing with. Second, it exports some tables that other applications use. Third, it imports tables exported by other applications. And finally, there are some tables that are shared by a number of applications with no clear owner. The last case is something we normally want to avoid.

In my experience, the integration scenarios can further be divided into four cases: Reference data, Consolidated data, Published data, and Subscription data.

(more…)

Posted in SOA, Software Development | Leave a comment

On Integration: Why I enjoy working with databases

Status: This article is currently pretty dry. I’d like feedback on how to make it more eloquent.

In my previous blog post, I promised to write more about using databases as the main integration strategy. In the current post, I plan to cover maybe the most important question: “Why?”

Imagine an application where every time it wants to communicate with another system, it reads or writes to the database. For now, let’s ignore how this would work, and how it would evolve, which will be the subject of later posts. What advantages does this offer?

The alternative is usually to integrate with another system though a variety of means. In Java, the most common ones are Web Services, RMI, EJBs (which offers it own quirks in addition to those of RMI), Sockets, and various tricks using the file system.

The most important issue to me is invariably productivity. When I work with databases, I generally can use Object-Relation Mapping tools. This is a very productive way of accessing database data in an application. RMI offers similar advantages, but you will have to build lazy loading on top of the domain model if you want to have a rich model where the objects are interconnected. Web Services generally have some bindings to Java, but in my experience, these are really inadequate. Either the Java side suffers, for example by forcing you to have getters and setters, by forcing you to use arrays instead of collections, or by forcing you to use strings as the main data type. Alternatively, the XML-side suffers by having non-specific types (if you use collections). Sockets, of course are very unproductive. They give up productivity for simplicity.

The data that is managed by the remote service generally will come from a database anyway. This means that the data access code will be have to be developed somewhere anyway. A remoting layer will have to be developed in addition.

To maintain sustainable productivity, we need unit tests. Unit testing has for me proved to be hard to do well for both Web Services and RMI, and EJBs are of course out of the question. As my regular readers know, using a test database for standalone unit testing is quite simple. As an added bonus, tests that use the database will essentially have verified the integration. When I use a remoting protocol, I always run into strange problems very late in the test process.

Both unit testing and productivity benefits from the fact that dealing with databases is something we’ve done for a long time. The tools and techniques for doing so are very mature, compared to other methods of integration.

Secondly, there is the problem of reliability. If you use a single database, everything you do is within one transaction. Either all work will be committed, or it will be rolled back. This vastly simplifies your logic if you care about your correctness. For distributed systems, this will in theory be solved by the 2-phase commit protocol. However, my experience is that this adds so much complexity to a solution that the system can metaphorically collapse under its own weight. As a result, most solutions I’ve seen (and, I suspect, most solutions I haven’t) simply ignore this problem. This means that the odd resource error that occurs might very well have very unpredictable results.

A remote layer will also introduce another place where things can go wrong. Many developers end up coding recovery rutines for dealing with these kinds of errors. In my experience, this is some of the most error prone code you can write.

Third, performance-wise it is hard to beat the database. Most other methods will eventually hit the database anyway, and as a general rule, adding more steps to a solution seldom makes it faster. There are some issues with scalabilitity, however, that I will address in a later post.

Last, and maybe most importantly, I have never seen a standard interface for dealing with remote services. Solutions generally end up having half-a-dozen or more different policies for accessing different back end systems. There is one thing we will always be sure of, though: There’ll always be a database among these backend systems, no matter what else you have to talk to. Every extra communication mechanism you remove will reduce the shoestring-and-paperclip-factor of your system.

By using a single data source as the place for communicating with other systems, we will reduce complexity and improve testability, performance and reliabilty.

I hope that in this post, I have demonstrated why, in an ideal world, you would want to use a single database as your primary integration mechanism. However, the world is rarely ideal. Database schemas change, more load is added than what a single database can tackle, you have to understand a forest of database schemas, some applications should not be allowed to access all the data. In my next blog post, I will talk about how to solve these problems with database without giving up the single database vision. Stay tuned for evolution, scalability, security, reuse, and understandability.

Posted in SOA, Software Development | 4 Comments