Four bold claims about SOA

Two of the hardest problems of software development are integration and what we could call business-IT alignment: The whole organization working towards the same goal.

SOA claims to address both of these problems. After listening harder than I’ve ever done before to SOA evangelists, I think I understand what mechanisms SOA proposes to solve these problems. I think the idea of SOA is based on a set of rather bold claims:

  • Web Services standards will solve the technical integration problems (“the WS-* claim”)
  • Centralizing integration will solve the governance issues surrounding integration (“the ESB claim”)
  • Modeling use cases as workflows of services will solve the business alignment problem and promotes reuse (“the BPM claim, part I”)
  • The flexibility to restructure the workflow between services will enable business agility (“the BPM claim, part II”)

I hope that an SOA evangelist will agree that these are important claims in SOA. Even if he will disagree with my evaluation of them.

Business-IT alignment and integration are extremely important and complex problems. In order for the claims about SOA to be true, either the complexity of these problems must be accidental and avoidable, or the tools of SOA can abstract away the complexity. That is an extremely bold claim. And in my experience, it is false. However, I do not have the same wide experience as many SOA evangelists. I know only few system and organizations. However, I would claim that the systems and organizations I know, I know extremely well.

And the view is not the same on the ground as it is at 30 000 feet.

When I’ve worked with systems composed of web services, my experience has been that the web services standards have far from solved the technical integration problems. Splitting up a system into two web services adds a large number of bugs and schedule time, even if the two endpoints run on the same platform. If we integrate web services written in Java with .NET it’s much worse. Maybe if we just had the right web service development tools, development and testing would be easier? But when I integrate two systems, the technical issues tend to disappear relative to the complexities of the organizational issues: When can you make the change we need? How and when can we test our integration? What is the reliability and performance of the service I am integrating with? This has to be answered in each individual case. Integration is always peer-to-peer on the organizational level, even when it is centralized at technical level. Maybe if we just hide this fact in an ESB, it will make the problems go away? Software development is very different from an assembly line. In systems I’ve worked with, 99 percent of the components are only used in one context. A use case step is generally not reusable, it is too context sensitive. Being context sensitive also means a developer who is told to ignore the context probably will do a suboptimal job. Maybe we just need to use a better tool for expressing the contract of the services? Flexibility adds complexity. Especially if that flexibility is created by distributing parts of the system. And I have never been particularly good at finding out what flexibility is needed, anyway. At the same time, I have a hard time testing a distributed system. So all my feedback cycles are slowed down. And I don’t know how to increase “business agility” without a good feedback cycle. Maybe if I just get the right tool for testing, building, versioning and deploying my services, I can have a distributed system with fast feedback cycles?

Can essential complexity be abstracted away?

Ultimately, I don’t think adding tools and technologies will fix our problems. High level programming languages were tools that fixed our problem, because they were able to abstract the accidental complexity well. Object relation mapping tools, on the other hand, provide leverage, but ultimately, I can’t use Hibernate well if I don’t understand both the underlying database and Hibernate itself. These tools add a lot of leverage, but the leverage comes at the cost of added complexity. Will ESB, WS-* and BPM be able to hide the complexity of the problems they are trying to address, or can they ultimately just give us more leverage when we deal with these problems? And exactly what problems are SOA technologies addressing? The problem of dealing with a large ecosystem of distributed services that needs to be integrated. But if workflow steps aren’t generally easy to develop in isolation, not generally as reuseable as we would like to think and don’t make the business more agile, isn’t SOA manufacturing complexity? How about we save the complexity of integration for the situations when we really need to deal with it? Here, WS-* and ESBs may have a role. Not as silver bullets, but as complex tools with a lot of leverage. Maybe. I am not sure about the leverage. But I am pretty sure about the complexity.

Comments:

[Geir Hedemark] - May 5, 2008

I mean ESBs, but explained myself very badly.

For me, SOA is a quite amorphous term. What I tried to say was that I like the idea of being able to use an ESB for tasks that are asynchronous in nature - stuff like EDIFACT-like messages flowing through your system.

I have seen people try to use servlets for handling message-based problems. I don’t think servlets are very well suited to publish-consume style tasks. Have a look at JSR 32 if you are wondering what I am on about - this is the servlet API applied to asynchronous messaging.

I have also been subjected to opinions that ended up as “we only need ESBs” in my head. I may be to blame here. though. ESBs are not a very good environment to implement request-response style applications - the technology tends to get in the way. We have servlets to do that kind of stuff.

There is also a possible mismatch between a rich domain model - where the subclasses has behaviour - and a message-based paradigm, where the intelligence sits in transformers along the path of the messages. I keep feeling that the messages are anemic domain objects getting passed around between transaction scripts, but I am also confident I need to move carefully here, lest I start a holy war. You can move to a rich domain model within the services/transformers themselves, but the technologies I have been using seem to get in the way of doing this - I tend to spend most of my time copying object contents if I force the information into a domain model.There may be something I don’t see, though.

I really, really want some way of bolting behaviour to the side of a data carrier without having to write boilerplate code. Qi4j or annotations would work, but nobody seems to know these technologies out there yet, so I tend to try not to use them in projects.

Hope this helps.

Lars Arne Skår - May 7, 2008

I think you hit the spot about popular claims. In addition, myself and others throw in a claim that this SOA-fad also put an emphasis on semantic interoperability as well as the technical interoperability; i.e. we often times get into an argument “what is a good service”. A similar claim was given with OO-fad (“What is good object”). For some reason still does not get that wide attention. Maybe because the technical stuff is easier to disagree/agree on.

Of these claim, I have to admit it is the 2. one I have the most faith in at the moment. The WS-* gang went into the same trap as the Corba-gang did with to much focus on solving all possible techical interoperability challenges which in turn created tight coupling in practice which again violated the originial intention. The WS-thing was so much easier to deal with when it was only XML and HTTP. Maybe that is the reason why REST is getting so popular these days.

Many smart people claim that REST is insufficient for “real stuff”; in particular vendors of tools supporting SOAP and the other WS-* stuff. However, we are clearly observing that the use of REST tend to be much more dynamic and agile in “real life” than these “ultra-flexible” and “ultra-powerful” standardized WS-stuff.

The usefulness of the BPM stuff is greatly overrated to say the least - at least at the current state. In many ways it resembles the state of the case tools we had in the 80s and the 90s. Still, I like to live in the hope that one day this could be really useful; the fact that BPEL is being used and getting some support - even outside of the vendors providing tools for BPEL show some promise. BPEL doesn’t solve everything, but being fairly simple and “low-key” is in my mind a good thing. Maybe it should be kept that way.

Getting back to claim nr. 2, I am in much favour of the view that a service bus is no more than a “galvanic divider” between a service provider and a service requestor, and in some circumstances this gives value; i.e. when you want loose coupling between systems. The reason why there is issues with this claim, is all the other stuff vendors put in the ESB. They create ambiguity in where to put stuff, and also introduces yet another operational complexity which again also gives even more deployment complexity which is hard enough as it is.

My faith lies on the open source movement and that the vendors will react constructively to what is usually called light weight ESB which again may lead to more practically oriented ESBs. Similar things did happen to J2EE/Java EE with Spring and some important adjustments to the EJB spec in EJB 3.0. While not being perfect yet - the movement seems to be healthy.

- My reaction to this very insightful blog. Thanks!

Johannes Brodwall - May 7, 2008

Both Geir and Lars Arne seems to have more insight into the use of ESBs than me.

Personally, I have never seem the problem that Geir is describing with using OO in an event driven world. Granted, I have never perceived this to be a major source of problems, but I end up translating events (usually incoming documents) to “command objects” which update the state of the object model. I’ve never missed an ESB when doing this. What am I misunderstanding?

Lars Arne’s “galvanic divider” is closer to things I have encountered. But in my cases the “galvanic divider” became just another layer of crud on top of many more layers of crud. As I understand it, this view of ESBs would either mean that you have to have a generic representation of any message type or you need to have individual representations for different messages. If you go for the first option, you’re screwed, as most people who’ve tried know. If you go for the second option, don’t you just add another layer of complexity with no corresponding value, as the new representation will have to match the old one, anyway?

Maybe it would be clearer to me if I saw the ESB tool you guys use.

Marcus Ahnve - May 2, 2008

Brilliant - one of the best pieces I have read on SOA.

Johannes Brodwall - May 2, 2008

Geir Hedemark points out another thing that I didn’t really think of about BPM: If you expose and reuse your services (Use Case steps, remember?), you can no longer change these. So your “flexible” processes may be composed of “frozen” steps.

Geir Hedemark - May 2, 2008

Good post, Johannes, and thanks for the kind words.

I have a slightly more positive view on SOA than you do, especially when it comes to ESBs, and if you have chosen to stay clear of the “SOA is WS-*” crowd.

Some tasks out there are by nature sequential and/or asynchronous. You can take a big hammer and force these things into a synchronous, object-oriented way of developing, but the mismatch between what you are solving and the tools you are using to create a solution will cause pain. I think that pain may very often be larger than using an ESB. In these cases, I think an ESB is a very good option.

Johannes Brodwall - May 5, 2008

Hi, Geir

Do you mean ESBs or BPM? I thought BPM was what people used to deal with events?

Personally, I’ve never found a conflict between object orientation and event driven applications. I would love to hear a bit of elaboration on this.