Den norske regjeringen har besluttet at en felles IKT-arkitektur for offentlig sektor ville være fint. Jeg fikk greie på arbeidet på tirsdag, og har lest rapport til den store gullmedalje. Jeg er fortsatt ikke helt sikker på hva som menes med “felles IKT-arkitekt”, men jeg kan se omrisset av mange store evighetsprosjekter i dokumentet.

Jeg har forfattet et svar på rapporten.

Spesielt er jeg bekymret for at dette skal bli en unnskyldning for store SOA-prosjekter uten veldefinerte formål. Rapporten beskriver en god del ønskede effektmål, men disse beskrives i såpass runde former at man aldri kommer til å etterprøve om prosjekter faktisk oppfyller dem. Formålene er ting som økt grad av interoperabilitet.

Frykt nummer to ligger i hvordan SOA-krigen går om dagen. Rapporten nevner veldig lite konkret på teknologier, så jeg slapp unna å gå inn i dette minefeltet. I stedet må jeg ligge våken og frykte effekten av de usagte ordene i rapporten.

Jeg har startet å høre om dogmatiske SOA-prosjekter som ikke lykkes, til tross for store investeringer. REST starter å bli mer akseptabelt å snakke om i høflig selskap. Men fremdeles virker det som om at “SOA prosjekt” er et kodeord for “vi skal kjøpe dyr WS-* programvare fra Oracle, IBM eller Microsoft”.

Jeg skulle ønske jeg hadde mer erfaring innen offentlig sektor før jeg skulle uttale meg om dette, men dersom det er som noe annet jeg har sett, handler interoperabilitet grunnleggende sett om å sende eller motta informasjon på et standardisert format. Det betyr tre ting: En aksjon (les, oppdater, lever), noe som identifiserer et objekt (søknad, personopplysning) og et innhold (“Jeg ønsker med dette herved å søke om bla bla bla”).

Og her har WS-* skadet oss mye. Når man bygde standarden SOAP (Simple Object Access Protocol, som verken er enkel eller handler om “object access”), skrellet man vekk to fine deler av HTTP standarden: Verb (GET, POST, PUT – det vil si “aksjoner”) og URL’er (det vil si den nøkkelen som representerer et objekt). Det betyr at vi ikke lenger kan si “legg til” (POST) en “søknad” (http://etat.no/tjeneste/soknad/) som inneholder noe data (ikke en del av standarden). I stedet må vi si “send en melding som inneholder informasjon om noe vi tenker å gjøre” (ikke en del av standarden).

I en rimelig verden burde interoperabilitet betydd at å utveksle informasjon om objekter burde være en del av en universell standard som er akseptert rundt hele verden. En standard basert på å sende meldinger om å utføre én eller annen oppgave (som for eksempel å utveksle et objekt) virker sørgelig utilstrekkelig. Og offentlig sektor i lille Norge kan vel ikke være den som brøyter vei her.

Så lenge SOA-krigen pågår er jeg ikke optimistisk for sjansene til at en rimelig standard dukker opp.

If this is true, how can we think that we can develop the interface as an after though to the central system.

Not all your users might be human. And the computerized users are treated even worse.

If the non-human users also care about the interface, how do we think that they will be satisfied with the same interface that is used internally between different parts of the application? These interfaces are the scraps of the service table.

If this is true, why do so many system diagrams display the “service” layer for external users below the user interface layer. It should be next to the human user interface layer, shouldn’t it?

External programmatic interfaces, whether they be SOAP, REST or otherwise, must be treated as any other user interface. This means both to understand the needs of the non-human users and to only develop parts of the service interface that are needed now.

Any user interface expert will tell you: There are no taking-backs with user interfaces. Once you’ve given something to the user, they will react very negative to substantial changes. The non-human user is even more unforgiving. Yet, the scraps of the system internals that we give them will be even more necessary to change.

If the users judge your service by its interface, we should only give non-human users interfaces that are developed based on their requirements, and that we don’t mind never getting to change.

Silo (software): A silo system cannot easily integrate with any other system.

In software, the term “silo” is used to refer to a system that is constructed as one unit from the front end to the data storage. Everything is tied together to work with the rest of the silo, but not with other elements.

This is considered a bad thing.

The problem comes when we want to integrate the system with other systems or reuse parts of the system.

Many of the new ideas in software development has as one of their big goals to try and rectify the silo problem. In general this is achieved by splitting up the system into services that may or may not be distributed across different computers.

But the badness of the silo hinges on two claims: First, that the applications built as an integrated stack cannot be integrated with, and second, that a system of distributed services can be integrated with.

Fact 1: Layers reduce understanding

One way that services are introduced is as an abstraction on top of an existing system. More often that we’d think, that system is again an abstraction upon an older system. Each layer was added in an attempt to make it easier for someone to integrate with the system. Each layer makes the whole harder to understand.

Just a few weeks ago, I experienced a profound sense of deja vu as I had gradually peeled of layers of a set of services and discovered that integrating directly to the ultimate data source was the easiest option. Each layer had added confusion, complexity, delays and unneeded constraints. Integrating with the “silo” was easier than integrating with the service layers on top of it.

Fact 2: Silos often aren’t

We’re currently building Java-systems that will extend and ultimately (hopefully) replace systems running on old mainframes. The common perception is that the mainframe apps are silos. They are hard to understand and impossible to split up.

Have you tried? As it turned out, of our many Java-developers, almost none have dared to started looking into how the mainframe works. When someone finally did, he found that it was fairly easy to integrate with. It requires hard work, but when did integration not?

Fact 3: Services might not be

“Fine,” you say, “but what about large grained services?” My experience is that even these are hits and misses. For example, one would think that distributing information via postal mail, electronic channels and fax might be a reusable service. It can be, but it could also be a cost driver.

Even when it comes to large scale services, reuse is costly. If your “enterprise architecture” focuses on what services should be reused by whom, you might be wasting a lot of time for a lot of people.

Ode to the silo

Maybe the classical silo isn’t as bad as we’re often told. Focusing on just what the system needs to do and dealing with the integration by talking to others sure is expensive. But maybe trying to architect a reusable set of services is just as expensive.

Dette er et utkast til en artikkel jeg ønsker å få publisert. Jeg setter stor pris på tilbakemeldinger om uklare tanker og formuleringer.

To av de vanskeligste problemene vi møter innen programvareutvikling er integrasjon og det som gjerne kalles “business-IT alignment” eller forretningsorientering, altså: IT skal understøtte virksomhetens strategiske mål.

Tjenesteorientert arkitektur, eller Service Oriented Architecture (SOA), hevder å kunne løse begge disse problemene. I det siste har jeg forsøkt å lytte mer aktivt til påstander fra evangelister av SOA, og jeg begynner å få en forståelse for hva det er SOA foreslår som løsningen på problemene vi står overfor. Dette er min oppfatning av påstandene om hvordan SOA skal løse problemene med integrasjon og forretningsorientering:

• Web service-standarder vil løse de tekniske integrasjonsproblemene (“WS-stjerne-påstanden”)
• Å sentralisere integrasjon via ett knutepunkt vil løse de organisasjonsmessige utfordringene rundt integrasjon (“ESB-påstanden”)
• Å modellere funksjonalitet som en arbeidsflyt mellom tjenester vil gi oss bedre forretningsorientering (“BPM-påstanden, del 1″)
• Å kunne restrukturere arbeidsflyten mellom tjenestene vil gi oss en en smidig forretningslogikk (“BPM-påstanden, del 2″)

Jeg håper at SOA-evangelister kjenner seg igjen i disse påstandene og er enige i at disse påstandene er sentrale når det gjelder SOA. Men jeg tror evangelistene vil være uenige i min vurdering av påstandene om SOA.

Forretningsorientering og integrasjon er svært viktige og vanskelige problemer. Dersom påstandene om SOA skal holde mål, betyr dette at problemene er unødvendige, at de er unngåelige eller at de tekniske og designmessige verktøyene SOA gir oss kan abstrahere vekk kompleksiteten.

Dette er en uvanlig vågal påstand. Og etter min dømmekraft, en usann påstand. Min erfaring er ikke så bred som den til mange SOA-evangelister. Jeg har erfaring med et fåtall organisasjoner og et fåtall systemer. Men de systemene jeg kjenner, kjenner jeg svært godt, og

Systemer ser svært forskjellig ut på nært hold og når man ser dem fra 30 tusen fot.

Når jeg har jobbet med systemer som består av web services, er min erfaring at standardene på dette feltet har vært langt fra å løse de tekniske integrasjonsproblemene. Å splitte et system opp i distribuerte komponenter øker antall feil og kostnaden ved å utvikle systemet, selv i de tilfellene der de to tjenestene kjører på samme plattform. Dersom man ønsker å integrere Java-løsninger med .NET-løsninger, er ting mye verre. Kanskje utvikling og testing av distribuerte systemer ville vært enklere om vi bare hadde hatt de rette verktøyene?

Eller kanskje ikke. Når vi integrerer to systemer, er de tekniske utfordringene små sammenlignet med de organisasjonsmessige utfordringene: “Når kan du utføre den endringen vi trenger?” Hvordan tester vi integrasjon mellom systemene? Hva er ytelsen, robustheten og påliteligheten til tjenesten jeg integrerer meg med? Slike spørsmål må besvares for hvert integrasjonspunkt. Når man ser på de organisasjonsmessige problemstillingene, er integrasjon alltid et bilateralt problem, selv når de tekniske problemstillingene er løst multilateralt. Kanskje problemene ville forsvinne dersom vi bare skjuler dem i en sentral komponent?

Programvareutvikling er forskjellig fra en samlebånd på en fabrikk. I de systemene jeg kjenner, er 99 % av den funksjonaliteten man har utviklet kun brukt én gang. Enkeltsteg i use cases eller andre kravbeskrivelser er sjelden gjenbrukbare. De er altfor avhengige av konteksten de brukes i. Ettersom funksjonaliteten essensielt er avhengig av konteksten, vil en utvikler som befales å ignorere denne konteksten etter min erfaring gjøre en dårligere jobb, fordi utvikleren vil tolke kravene til det som skal lages for bredt. Kanskje vi bare trenger en bedre måte å uttrykke kontrakten til det som skal utvikles på?

Fleksibilitet gjør en løsning mer kompleks. Aldri er dette så sant som når fleksibiliteten er oppnådd ved å distribuere delene av systemet. Personlig har jeg aldri vært flink til å vurdere hvilken fleksibilitet som er nødvendig, og hvilken fleksibilitet som jeg kun inkluderer for å tilfredsstille mitt tekniske ego. Men det jeg vet er at det er vanskeligere å teste fleksible, distribuerte systemer. Så feedback i prosessen blir blokkert av tekniske valg. Og min erfaring er at feedback er den eneste måten vi kan få en mer smidig forretning. Kanskje dersom verktøyene våre var bedre, så ville vi være i stand til å utvikle, teste, versjonshåndtere, driftsette og endre distribuerte systemer uten å sabotere feedback?

Kan grunnleggende kompleksitet abstraheres bort?

Til syvende og sist tror jeg svaret på alle disse spørsmålene blir “nei”. Å legge til teknologier og verktøy løser sjelden problemene våre. Mer avanserte programmeringsspråk lot oss løse vanskeligere problemer, fordi de lot oss abstrahere oss vekk fra teknisk, irrelevant kompleksitet. Nyere verktøy som for eksempel Object Relation Mapping som Hibernate gir oss en kraftig brekkstang til å løfte tunge løft, men vi kan ikke bruke denne type verktøy effektivt og trygt dersom vi ikke skjønner både databaser og selve verktøyet godt. Avanserte verktøy kan gi oss større effekt, men de er også vanskeligere å forstå.

Vil WS-*, ESB’er og BPM være verktøy som vil være i stand til å skjule kompleksiteten i problemene de forsøker å takle, eller vil de kun gi oss mer kraft når vi forsøker å løse disse problemene? Og hva vil være kostnaden av denne kraften når det gjelder ekstra kompleksitet?

Og nøyaktig hvilke problemer er det SOA-verktøy og -tankesett retter seg mot? De fokuserer på problemer rundt å håndtere et stort økosystem av distribuerte tjenester som må integreres. Men dersom det er kostbart å utvikle en tjeneste som kan brukes utenfor sin opprinnelige sammenheng, dersom tjenester ikke er så gjenbrukbare som vi skulle likt å tro og dersom distribuert arkitektur er et hinder for en smidig forretning, er ikke da SOA en måte å skape unødvendig kompleksitet på?

Hva om vi sparer komplekse problemstillinger som integrasjon til de tilfellene der det virkelig trengs? I slike situasjoner kan WS-* og ESB spille en rolle. Ikke som en sølvkule som skal ta livet av kompleksitetsmonsteret, men som avanserte, komplekse verktøy som kan øke effekten av arbeidet vårt. Kanskje. Jeg er ikke sikker på hvor mye effekt disse verktøyene egentlig gir. Men jeg er ganske sikker på de øker kompleksiteten på oppgaven.

Mange takk til Eivind Nordby, Hogne Kirkeby, min far Bjørn Brodwall, Anne Marie Baugstø-Hartvigsen, Geir Hedemark og Lars Arne Skår som har hjulpet meg med tilbakemeldinger på denne artikkelen.

The talk is a bit fleeting and unstructured (and someone’s phone keeps ringing during the presentation!), but Rangaswami presents a lot of very interesting opinions:

• He observes that the business environment in banking industry (and other industries?) are less focused on standards than before. He believes Open Source development has a better chance of achieving cooperation than standards in a business environment where “everyone want to be top dog”
• He explores the model of Open Source Bonds. A company issues a bond promising to pay whoever develops X (for example) a sixth of what it would cost them to built it in house. He hints to a possible network of brokers and service providers.

Personally, I found the Q&A part the most valuable:

• Question: “What should I tell the executives in my company who consider open source ‘high risk’?” Rangaswami talks about how there are only four interesting risks during software procurement: 1) The vendor disappears, 2) The vendor gets a different focus from you, 3) quality, and 4) security. Open source has a “build-in escrow” clause (just without the lawyers), which takes care of the first two as well as commercial software can. He further points out that within the industry, nobody is held accountable for software failure. This means that quality is not something you can buy. Finally, the inspection value when it comes to security is considerable.
• Question: “What should the architecture department be doing?” This was the most interesting question to me. Rangaswami gives a long and thoughtful answer. The architect is not unique and different from everyone else, except that (s)he is more committed to learning. The architect should work through “control-freakery”, but through influence, advice and support. An architect should not “try to write an architecture (I am proud of being accused of not having one)”
• Question: “What is the main cause of project failure in the software industry?” Rangaswami points out how we’re not very good at saying “no” to the customers. Project failure is caused by a culture that is unwilling to accept failure. What is “worse than failure”? Not learning from that failure because you’re not accepting it.
]]> http://johannesbrodwall.com/2007/03/11/link-open-source-in-the-enterprise/feed/ 4 http://johannesbrodwall.com/2007/03/03/a-brief-adventure-with-universal-repositories-and-rest-web-services/ http://johannesbrodwall.com/2007/03/03/a-brief-adventure-with-universal-repositories-and-rest-web-services/#comments Sun, 04 Mar 2007 03:17:03 +0000 Johannes Brodwall http://www.brodwall.com/johannes/blog/2007/03/03/a-brief-adventure-with-universal-repositories-and-rest-web-services/ Inspired by Per Mellqvist (and myself, to be fair), I wanted to explore the possibility of using a generic DAO or Repository interface for REST. Based on this simple idea, I was able to create a very cute and testable prototype of a full Web Service stack for REST based Web Services. The most interesting aspect was creating a universal test case for Repositories.

This article shows how little code is required to implement and test a REST based Web Service in Java, despite the horror of the Java HTTP client API. The source code can be downloaded from my subversion repository. I also want to illustrate how to create black box tests that can be reused efficiently with different implementations of a Repository.

The generic Repository and its friend, the generic Repository Test

A Repository or Data Access Object allows a client program to Create, Retrieve, Update and Delete object. A simple design that has served me well is something like this:

public interface Repository<T> {

    Serializable insert(T entity);

    public T retrieve(Serializable key);

    void update(Serializable key, T entity);

    void delete(Serializable key);

}

However, when you start implementing this interface, you will soon find that you need a little more functionality even in the general case. Here is a test case that shows one of my favorite patterns for black box testing of repositories:

public abstract class RepositoryTest<T> extends TestCase {

    private T insertedObject = createDistinctObject();

    private Serializable key;

    protected abstract Repository<T> getRepository();

    @Override
    protected final void setUp() throws Exception {
        key = getRepository().insert(insertedObject);
        getRepository().writeChanges();
    }

    public void testRetrieve() {
        T retrievedObject = getRepository().retrieve(key);
        assertNotSame("retrieve in new session should not return same",
                      insertedObject, retrievedObject);
        assertEquals(insertedObject, retrievedObject);

        assertSame("All retrieves in session should return same instance",
                   retrievedObject, getRepository().retrieve(key));
    }

Notice that I was the insertedObject to be equals to the updatedObject, but in order for this to make any sense, I need them to be different object instances. Most Repository implementations will have some sort of session cache, and the point of the writeChanges() method in this case is to flush this cache, so objects returned will be fresh. Having a session cache avoids a lot of confusion, so I like to test for it. That is what the final assertSame in testRetrieve verifies.

The second thing we need for a generic repository is a way to deal with unwanted aliasing. This is in particular an issue with Object-Relation Mapping technology such as Hibernate or TopLink. If I change an object I got back from the Repository, but don’t call Repository.update, the object may still be changed in the database. This is because the Object-Relation Mapper will keep track of all changes in objects that it has returned. Here is a test case that illustrates the problem (and solution):

public void testDiscardedUpdate() {
    T updatedObject = getRepository().retrieve(key);
    assertNotSame(insertedObject, updatedObject);
    changeObject(updatedObject);
    assertFalse(insertedObject.equals(updatedObject));
    getRepository().discardChanges();

    assertEquals(insertedObject, getRepository().retrieve(key));
}

With discardChanges in place to ensure that we don’t write to the Repository when we don’t want to, the final Repository interface looks like so:

public interface Repository<T> {

    Serializable insert(T entity);

    public T retrieve(Serializable key);

    void update(Serializable key, T entity);

    void delete(Serializable key);

    public void writeChanges();

    void discardChanges();

}

This interface proves to be both generic enough that we can construct a generic test case for it, and powerful enough that it can be implemented in a variety of ways.

The Memory Repository

Since the interface is so small, I decided to implement a pure in-memory variant of Repository using HashMaps as the internal storage. The code is trivial, the only interesting thing is the session cache on top of it.

As it turns out, the SessionCachedRepository can be used in other scenarios as well, and I will reuse it for the REST implementation of the repository. There are a few challenges to a generic SessionCachedRepository. Most importantly, because insert returns the key for the object, we need to have it write through to the server. In order to implement discardChanges, I save a list of inserted objects that can be removed manually:

public void discardChanges() {
    for (Serializable key : sessionInserts) {
        uncachedRepository.delete(key);
    }
    sessionUpdates.clear();
    sessionDeletes.clear();
}

(Note to self: Could I implement the key with a smart proxy to get around this problem?)

The REST repository

I implemented both the server and the client side of the REST Web Service with many of the same pieces. By using the same Repository interface on both sides of HTTP, I can easily exchange remote and local tests. The great advantage of this is that it is usually much easier to get the local part to work first, and then work on the server. Here is the client side Repository:

public class RESTClientRepository<T> implements Repository<T> {

    private RESTHttpClient httpClient;

    private EntityToXmlMapper<T> mapper;

    private URL baseUrl;

    public RESTClientRepository(URL baseUrl, RESTHttpClient httpClient,
                                EntityToXmlMapper<T> mapper) {
        this.baseUrl = baseUrl;
        this.httpClient = httpClient;
        this.mapper = mapper;
    }

    public Serializable insert(T entity) {
        return (URL)httpClient.doPostAndReturnLocation(
                resourceUrl(""), mapper.entityToWriter(entity));
    }

    public T retrieve(Serializable key) {
        return mapper.xmlToEntity(httpClient.doGet(resourceUrl(key)));
    }

    public void update(Serializable key, T entity) {
        httpClient.doPut(resourceUrl(key), mapper.entityToWriter(entity));
    }

    public void delete(Serializable key) {
        httpClient.doDelete(resourceUrl(key));
    }

    private URL resourceUrl(Serializable key) {
        try {
            return new URL(baseUrl, key.toString());
        } catch (MalformedURLException e) {
            throw new IllegalArgumentException("Invalid url " + key);
        }
    }

This code really shows off the beautiful simplicity of the REST scheme. Each method does an HTTP call to a URL that is determined by the ID, encodes the object as XML, and decodes the result as XML, too. The XML mapping can be as complex as you want, but it is independent of the REST stack. The only other complex part of this solution is the RESTHttpClient. I implemented this on top of Java’s poor HttpURLConnection in order to avoid dependencies. It’s a horribly API, so please don’t blame me for the horrible client code. :-(

There is one thing that should make you go “that’s funny” in the RESTRepostory code, namely the insert method. Here, we get a String back and convert it to a URL. What is going on here?

With REST web services, the normal way of creating a new resource is to POST the resource. The server will respond with 201 – created, and it will have the URL to the new resource in the “Location” header. I have taken a shortcut with regard to this in the RESTHttpClient.doPostAndReturnLocation. An interesting point here is that I can use the URL as key throughout the client side. As long as the code doesn’t get too curious with the actual class of the key, this works fine.

Now for the other side of the equation, the servlet:

public class RESTRepositoryServlet<T> extends HttpServlet {

    private Repository<T> repository;

    private EntityToXmlMapper<T> mapper;

    public RESTRepositoryServlet(Repository<T> repository,
			       EntityToXmlMapper<T> mapper) {
        this.repository = repository;
        this.mapper = mapper;
    }

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
			      throws IOException {
        T entity = getEntity(req);
        mapper.entityToWriter(entity).doWithWriter(resp.getWriter());
        resp.setContentType("application/xml");
    }

    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) {
        Serializable key = repository.insert(mapper.xmlToEntity(req.getReader()));
        String url = req.getScheme() + "://" + req.getServerName() + ":"
                  + req.getServerPort() + req.getContextPath()
                  + req.getServletPath() + "/" + key;
        resp.setHeader("Location", url);
        resp.setStatus(HttpServletResponse.SC_CREATED);
    }

    @Override
    protected void doPut(HttpServletRequest req, HttpServletResponse resp) {
        getEntity(req);
        repository.update(getKey(req), mapper.xmlToEntity(req.getReader()));
        resp.setStatus(HttpServletResponse.SC_OK);
    }

    @Override
    protected void doDelete(HttpServletRequest req, HttpServletResponse resp) {
        repository.delete(getKey(req));
    }

Again, REST maps one to one with the servlet specification. The doPost mirrors what we saw on the client side. It returns 201, and sets the Location header. There is a minor snafu, though. I have been unable to find the full path to the servlet, so I had to paste it together from all the different parts of the request properties. I would very much appreciate input on how this should be done.

I have one more trick up my sleeve when it comes to the servlet. The code for validating the content of the request is usually littered all over the Servlet. In order to fix this, I use an unchecked exception and override HttpServlet.service to deal with it:

@Override
protected void service(HttpServletRequest req, HttpServletResponse resp)
	      throws ServletException, IOException {
    try {
        super.service(req, resp);
    } catch (ServletRequestException e) {
        resp.sendError(e.getStatusCode(), e.getMessage());
    }
}

private Serializable getKey(HttpServletRequest req) {
    // Called from doDelete, doPut and getEntity
    String[] parts = req.getPathInfo().split("/");
    if (parts.length <= 1) {
        throw new ServletRequestException(400, "Missing id");
    }
    return Long.valueOf(parts[1]);
}

Who needs a web framework?! Hahaha!

Testing it all

I have created two implementations of the RESTHttpClient used by the RESTClientRepository. One is a bridge from the interface to HttpServletRequest and HttpServletResponse. This way, I can test the client and server connection in memory. This is very fast so it comes in quite handy. The code uses spring-mock MockHttpServletRequest and MockHttpServletResponse.

Here is the code that runs the http to servlet bridge test:

public class CategoryRESTRepositoryTest extends RepositoryTest<Category> {

    private final static String SERVLET_PATH = "/test";

    private Repository<Category> serverRepo=new MemoryRepository<Category>();

    private CategoryToXmlMapper entityToXmlMapper = new CategoryToXmlMapper();

    private Servlet restRepoServlet =
                new RESTRepositoryServlet<Category>(serverRepo, entityToXmlMapper);

    private Repository<Category> clientRepo = new SessionCachedRepository<Category>(
                new RESTClientRepository<Category>(
                      toUrl("http://foobar.com:83112", SERVLET_PATH),
                      new RESTServletClient(restRepoServlet, SERVLET_PATH),
                      entityToXmlMapper));

    @Override
    protected Repository<Category> getRepository() {
        return clientRepo;
    }

    private int index = 0;

    @Override
    protected Category createDistinctObject() {
        return new Category("foo " + index++, CategoryType.OPEN);
    }

Notice how I wrap the RESTClientRepository in a SessionCachedRepostory. This way, we can reuse the implementation of this code.

Of course, no test of Web Services would be complete without there at least being some traffic over the network! Following the recipe on how to run unit test with Jetty, I start the servlet in Jetty and run the unit tests with it. This code implements my RESTHttpClient using java.net.HttpURLConnection. It's a horrible class, but it's usable (barely).

Conclusion

RepositoryTest tests a generic Repository interface with three implementations: In-memory, simulated servlets, and real http traffic. The RepositoryTest has almost all the code, and there are only minimal overrides in the subclasses. This illustrates how to reuse tests in different contexts. The test is also a good example of how to test on a black-box level, focusing on the interesting behavior, and not the implementation.

RESTClientRepository and RESTRepositoryServlet implements the client and server-side parts of a full REST stack. I have done the XML binding as simple as possible in this article, but in a later post, I will explore how to do more interesting stuff with the XML binding, including implementing lazy relationships over REST.

The source of the article is available under the Apache Software License. Feel free to use it any way you see fit. However: I will be extremely grateful if you drop me a line if you find it useful, or if there's anything you'd like to see improved.

]]> http://johannesbrodwall.com/2007/03/03/a-brief-adventure-with-universal-repositories-and-rest-web-services/feed/ 14 http://johannesbrodwall.com/2007/02/27/crud-rest-rails/ http://johannesbrodwall.com/2007/02/27/crud-rest-rails/#comments Tue, 27 Feb 2007 23:27:53 +0000 Johannes Brodwall http://www.brodwall.com/johannes/blog/2007/02/27/crud-rest-rails/ Some time back, I watched a video David Heinemeier Hansson give a talk on ActiveResource on RailsConf. The thing that struck me is how much Rails’ ideas are connected to those of Domain-Driven Design. Watching DHH is like seeing a version of Eric Evans on speed.

The video is long, but very entertaining. And it is well worth watching even if you couldn’t care less about Rails. DHH explains in real concrete terms how to think in terms of Domain-Driven Design, even though from the sound of it, I don’t think he’s heard of the term. The subtitle of DHH’s talk is “How I stopped worrying and learned to love the CRUD”. Besides being a reference to one of the best movies ever, this title explains the views of DHH on CRUD, REST and DDD and how they fit together.

CRUD means Create, Retrieve, Update, Delete – the elementary operations on data. In many ways the POST, GET, PUT, DELETE http verbs in REST correspond to INSERT, SELECT, UPDATE, DELETE in SQL and Create, Retrieve, Update, Delete in the CRUD acronym. Thinking in REST means that you can only Create, Retrieve, Update or Delete stuff. But what about, say, making a Prospect Party into a Customer Party. This is a Business Operation on the Party object, or something, isn’t it? The Enterprisey thing to do is to create a method for it, right?

Not so fast, says DHH (and Evans!): We have a party entity. What if we instead insert a CustomerRelationship on the Party object? Or, in terms of REST:

PUT /party/135122/relationships
<customerRelationship>
  <keyAccountManager>http://my.company.com/employee/652312</keyAccountManager>
  <validUntil>2007/08/01</validUntil>
   <customerStatus>GOLD</customerStatus>
</customerRelationship>

For every enterprisey business operation, there is a new entity trying to break out.

For an excellent example of the applicability of these ideas, see The Beast Forum implemented in Rails. In not much more than 500 lines of code (yes, that is not a typo – five hundred! But there are about 1500 lines of template html code as well), they implement implement a fully featured forum, including RSS support up the wazzo (much better than PHPBB or JForum), and REST web services up the Wazzo. By following the CRUD conventions, you get it all for free!

CRUD is sooo goood! It taste like ice cream. With peanut butter! Yum!

]]> http://johannesbrodwall.com/2007/02/27/crud-rest-rails/feed/ 7 http://johannesbrodwall.com/2006/11/19/on-integration-consolidated-view/ http://johannesbrodwall.com/2006/11/19/on-integration-consolidated-view/#comments Sun, 19 Nov 2006 21:24:13 +0000 Johannes Brodwall http://www.brodwall.com/johannes/blog/2006/11/19/on-integration-consolidated-view/ In my last post, I wrote about four integration scenarios using databases: Reference data, Consolidated view, Subscription and Publishing. Of these, the Consolidated View scenario requires the most interaction between the server and the client roles. This post will examine how to make the pieces fit together.

Consolidated view joins the data of multiple clients into a consolidated view. This makes you able to create administrative applications that span a set of subapplications without having to change the central view when a new application joins the mix.



In order to work smoothly, a Consolidated View application uses two forms of integration: The client application write to their partitions of a table in the server application, and the view of the server application redirects to the view of the client applications. Here is an example:

1. A user wants to list all the contracts defined for a customer, regardless of what application manages that contract
2. The Consolidated View application lists some contracts from Client Application 1, some contracts from Client Application 2 and so on
3. The user clicks on the line associated with a Client Application 1 contract
4. The consolidated view application has registered the URL for this kind of contract to be handled by a another application. It redirects (or proxies) the user to (for example) http://a.server.name:3223/clientApp1/contracts/foo_contract/?id=3313
5. Client Application 1 lets the user interact with the data in any way the user desires
6. Client Application 2 updates the consolidated view application database to reflect any changes to the contract
7. The Contract has a link back to “View All Contracts for this Customer”. The user clicks this link and is brought back to the Consolidated View application
8. The user wants to add a new type of contract to this Customer. This particular type of contract is handled by Client Application 2. The user clicks “Add bar contract”
9. The consolidated view application has registered the URL for inserting new contracts like these for the customer. It redirects (or proxies) the user to (for example) http://b.server.name:3311/clientApp2/contracts/bar_contract/new?customer=1234
10. Client Application 2 lets the user enter the data to create the new contract
11. Client Application 2 updates the database for the consolidated application to include the new contract

In this scenario, the only thing the Consolidated View application knows about specific types of contract is where the user can be redirected in order to create or modify them. If a new application is added, a new row needs to by added to the contract_type table in the Consolidated View database. That is all. This means that despite the fact that the user is presented with a unified view of the data, the applications are very decoupled.

The downsides of using this solution is that there is still some coupling, and this coupling is now more implicit. If any of the client applications move to a new URL, someone has to remember to update the consolidated view application. Even worse: If the client applications link back to the server application, we have to remember to update all client applications if the server application moves. Of course, the last problem can be solved (if we really want to) by having the consolidated view server application URL be part of a Reference Data domain.

I hope this article has been concrete enough to show you, dear reader, how to solve problems elegantly using no application-level integration mechanisms. In my next post, I plan on examining how to get the various scenarios to scale.

]]> http://johannesbrodwall.com/2006/11/19/on-integration-consolidated-view/feed/ 0 http://johannesbrodwall.com/2006/11/11/on-integration-organizing-the-data/ http://johannesbrodwall.com/2006/11/11/on-integration-organizing-the-data/#comments Sat, 11 Nov 2006 17:35:56 +0000 Johannes Brodwall http://www.brodwall.com/johannes/blog/2006/11/11/on-integration-organizing-the-data/ In my last post on using the database for integration, I argued that the best metaphor for creating systems that are interconnected is that of One-Large Database. Carl-Henrik asked some very relevant questions about this, which I will interpret (for now) mainly as “how do you avoid drowning in complexity”. This post will address the issue of maintainability, especially when things grow to be large.

The On Large Database metaphor is mostly useful as a starting point. The first thing I notice when I look at our systems that have been organized this way is the fact that the data-structure is far from flat. Each application will deal with four categories of tables. First, an application has a large private domain, containing tables that the rest of the world has no business messing with. Second, it exports some tables that other applications use. Third, it imports tables exported by other applications. And finally, there are some tables that are shared by a number of applications with no clear owner. The last case is something we normally want to avoid.

In my experience, the integration scenarios can further be divided into four cases: Reference data, Consolidated data, Published data, and Subscription data.

Perhaps the most useful of the integration scenarios is that of Reference Data. Reference Data is exported by an application that is responsible for maintaining the data. Applications import Reference Data read-only. Reference data is very common in all domains. It often contains customer databases and other core data entities.



In some special cases, you might want shared data to be updated by a distributed set of applications. I call this scenario Consolidated View. One application is responsible for displaying a shared view of data that is being maintained by a number of applications. Each application will normally be responsible for reading and updating a subset of the shared data. This scenario is not as common as Reference data, but we see it for example in a situation where we want a consolidated view of all the contracts a particular customer (which is in the Reference data domain) holds. The different contracts are defined in the applications that operate on them (for example, a contract for a payment service is defined in the payment service system). A consolidated view allows us to display all the contracts a customer holds, even though the details are distributed in a number of applications.



These two scenarios are the ones used for sharing data. The last two scenarios are used to delegate responsibility. I call them Publishing and Subscription domains, respectively. Essentially, the subscribing party in these scenarios will poll the database at an appropriate interval. A little or a lot of sophistication can be put into the polling implementation as desired.

Publishing domains distribute data from one application to many. The domain is exported by the publishing application. Each subscribing application is usually responsible for a partition of the data. They will remove or update this data as they consume it. The most common example is an application that receives files that should be processed by one of many applications, depending on the type of file.



Subscription domains collect data from many applications into one. The domain is exported by the subscribing application, which also may remove and update data. The publishing applications insert data into the subscription domain. The most common example of subscription domain is an application collecting events for distribution or for billing.



Notably absent from the list is the one-to-many event publishing scenario. All the publish-subscribe scenarios I have talked about are point-to-point, not broadcasts. This is by design. All the examples we have seen so far have been broadcasting changes to a Reference domain. So far, it seems like these scenarios are better served by having a tighter integration with the Reference domain. I am still looking for counter examples.

In each of the four scenarios I have listed (Reference Data, Consolidated View, Publishing, Subscription) there is a clear definition of who is the client and who is the server. There is also a clear definition of which operations should be allowed by each party. For example, in the Consolidated View, the clients should be able to Created, Update and Delete data in their segment of the domain, and the server should only Read data. We currently implement these restrictions as GRANTS in the database.

In most domain, the manageability can be improved by simplifying the exported domains. This can be implemented by using VIEWS. VIEWS will also allow the server to change the internal structure without breaking clients, and allow clients that need different versions of the exported domain to coexist. It is important to note that for an application, it is transparent whether it is accessing a TABLE or a VIEW. This means that we can at a later time replace tables with views. I might write more on views in a later post.

I hope this post explains how to make the Single Database vision Manageable. Seen from one application, we will still maintain the illusion that it has direct access to all data. By using SYNONYMS, we can make the logical division into imported and exported domains transparent. This means that the application will be deployed in a single database schema in development and test environments, and the full glory of the integrating applications will only be revealed when we put the application into integration tests or production environments.

The fact that we have defined a limited number of integration scenarios mean that we can analyze how to make each scale separately. Stay tuned for my next post, where I discuss how to make the Single Database Vision scale to scenarios where different applications have to use different database instances.

]]> http://johannesbrodwall.com/2006/11/11/on-integration-organizing-the-data/feed/ 0 http://johannesbrodwall.com/2006/10/18/on-integration-why-i-enjoy-working-with-databases/ http://johannesbrodwall.com/2006/10/18/on-integration-why-i-enjoy-working-with-databases/#comments Wed, 18 Oct 2006 20:43:21 +0000 Johannes Brodwall http://www.brodwall.com/johannes/blog/2006/10/18/on-integration-why-i-enjoy-working-with-databases/ Status: This article is currently pretty dry. I’d like feedback on how to make it more eloquent.

In my previous blog post, I promised to write more about using databases as the main integration strategy. In the current post, I plan to cover maybe the most important question: “Why?”

Imagine an application where every time it wants to communicate with another system, it reads or writes to the database. For now, let’s ignore how this would work, and how it would evolve, which will be the subject of later posts. What advantages does this offer?

The alternative is usually to integrate with another system though a variety of means. In Java, the most common ones are Web Services, RMI, EJBs (which offers it own quirks in addition to those of RMI), Sockets, and various tricks using the file system.

The most important issue to me is invariably productivity. When I work with databases, I generally can use Object-Relation Mapping tools. This is a very productive way of accessing database data in an application. RMI offers similar advantages, but you will have to build lazy loading on top of the domain model if you want to have a rich model where the objects are interconnected. Web Services generally have some bindings to Java, but in my experience, these are really inadequate. Either the Java side suffers, for example by forcing you to have getters and setters, by forcing you to use arrays instead of collections, or by forcing you to use strings as the main data type. Alternatively, the XML-side suffers by having non-specific types (if you use collections). Sockets, of course are very unproductive. They give up productivity for simplicity.

The data that is managed by the remote service generally will come from a database anyway. This means that the data access code will be have to be developed somewhere anyway. A remoting layer will have to be developed in addition.

To maintain sustainable productivity, we need unit tests. Unit testing has for me proved to be hard to do well for both Web Services and RMI, and EJBs are of course out of the question. As my regular readers know, using a test database for standalone unit testing is quite simple. As an added bonus, tests that use the database will essentially have verified the integration. When I use a remoting protocol, I always run into strange problems very late in the test process.

Both unit testing and productivity benefits from the fact that dealing with databases is something we’ve done for a long time. The tools and techniques for doing so are very mature, compared to other methods of integration.

Secondly, there is the problem of reliability. If you use a single database, everything you do is within one transaction. Either all work will be committed, or it will be rolled back. This vastly simplifies your logic if you care about your correctness. For distributed systems, this will in theory be solved by the 2-phase commit protocol. However, my experience is that this adds so much complexity to a solution that the system can metaphorically collapse under its own weight. As a result, most solutions I’ve seen (and, I suspect, most solutions I haven’t) simply ignore this problem. This means that the odd resource error that occurs might very well have very unpredictable results.

A remote layer will also introduce another place where things can go wrong. Many developers end up coding recovery rutines for dealing with these kinds of errors. In my experience, this is some of the most error prone code you can write.

Third, performance-wise it is hard to beat the database. Most other methods will eventually hit the database anyway, and as a general rule, adding more steps to a solution seldom makes it faster. There are some issues with scalabilitity, however, that I will address in a later post.

Last, and maybe most importantly, I have never seen a standard interface for dealing with remote services. Solutions generally end up having half-a-dozen or more different policies for accessing different back end systems. There is one thing we will always be sure of, though: There’ll always be a database among these backend systems, no matter what else you have to talk to. Every extra communication mechanism you remove will reduce the shoestring-and-paperclip-factor of your system.

By using a single data source as the place for communicating with other systems, we will reduce complexity and improve testability, performance and reliabilty.

I hope that in this post, I have demonstrated why, in an ideal world, you would want to use a single database as your primary integration mechanism. However, the world is rarely ideal. Database schemas change, more load is added than what a single database can tackle, you have to understand a forest of database schemas, some applications should not be allowed to access all the data. In my next blog post, I will talk about how to solve these problems with database without giving up the single database vision. Stay tuned for evolution, scalability, security, reuse, and understandability.

]]> http://johannesbrodwall.com/2006/10/18/on-integration-why-i-enjoy-working-with-databases/feed/ 4