Two of the hardest problems of software development are integration and what we could call business-IT alignment: The whole organization working towards the same goal.
SOA claims to address both of these problems. After listening harder than I’ve ever done before to SOA evangelists, I think I understand what mechanisms SOA proposes to solve these problems. I think the idea of SOA is based on a set of rather bold claims:
- Web Services standards will solve the technical integration problems (“the WS-* claim”)
- Centralizing integration will solve the governance issues surrounding integration (“the ESB claim”)
- Modeling use cases as workflows of services will solve the business alignment problem and promotes reuse (“the BPM claim, part I”)
- The flexibility to restructure the workflow between services will enable business agility (“the BPM claim, part II”)
I hope that an SOA evangelist will agree that these are important claims in SOA. Even if he will disagree with my evaluation of them.
Business-IT alignment and integration are extremely important and complex problems. In order for the claims about SOA to be true, either the complexity of these problems must be accidental and avoidable, or the tools of SOA can abstract away the complexity.
That is an extremely bold claim. And in my experience, it is false. However, I do not have the same wide experience as many SOA evangelists. I know only few system and organizations. However, I would claim that the systems and organizations I know, I know extremely well.
And the view is not the same on the ground as it is at 30 000 feet.
When I’ve worked with systems composed of web services, my experience has been that the web services standards have far from solved the technical integration problems. Splitting up a system into two web services adds a large number of bugs and schedule time, even if the two endpoints run on the same platform. If we integrate web services written in Java with .NET it’s much worse. Maybe if we just had the right web service development tools, development and testing would be easier?
But when I integrate two systems, the technical issues tend to disappear relative to the complexities of the organizational issues: When can you make the change we need? How and when can we test our integration? What is the reliability and performance of the service I am integrating with? This has to be answered in each individual case. Integration is always peer-to-peer on the organizational level, even when it is centralized at technical level. Maybe if we just hide this fact in an ESB, it will make the problems go away?
Software development is very different from an assembly line. In systems I’ve worked with, 99 percent of the components are only used in one context. A use case step is generally not reusable, it is too context sensitive. Being context sensitive also means a developer who is told to ignore the context probably will do a suboptimal job. Maybe we just need to use a better tool for expressing the contract of the services?
Flexibility adds complexity. Especially if that flexibility is created by distributing parts of the system. And I have never been particularly good at finding out what flexibility is needed, anyway. At the same time, I have a hard time testing a distributed system. So all my feedback cycles are slowed down. And I don’t know how to increase “business agility” without a good feedback cycle. Maybe if I just get the right tool for testing, building, versioning and deploying my services, I can have a distributed system with fast feedback cycles?
Can essential complexity be abstracted away?
Ultimately, I don’t think adding tools and technologies will fix our problems. High level programming languages were tools that fixed our problem, because they were able to abstract the accidental complexity well. Object relation mapping tools, on the other hand, provide leverage, but ultimately, I can’t use Hibernate well if I don’t understand both the underlying database and Hibernate itself. These tools add a lot of leverage, but the leverage comes at the cost of added complexity.
Will ESB, WS-* and BPM be able to hide the complexity of the problems they are trying to address, or can they ultimately just give us more leverage when we deal with these problems?
And exactly what problems are SOA technologies addressing? The problem of dealing with a large ecosystem of distributed services that needs to be integrated. But if workflow steps aren’t generally easy to develop in isolation, not generally as reuseable as we would like to think and don’t make the business more agile, isn’t SOA manufacturing complexity?
How about we save the complexity of integration for the situations when we really need to deal with it? Here, WS-* and ESBs may have a role. Not as silver bullets, but as complex tools with a lot of leverage. Maybe.
I am not sure about the leverage. But I am pretty sure about the complexity.