This test is in the actual war module of the project. This is what it does:
Configures the application to run towards a test database
Starts up the web server Jetty on an arbitrary port (port = 0) and deploys the current web application into Jetty
Fires up HtmlUnit which is a simulated web browser
Inserts an object into the database
Navigates to the appropriate location in the application
Verifies that inserted object is present on the appropriate page
This test requires org.eclipse.jetty:jetty-server, org.eclipse.jetty:jetty-webapp and org.seleniumhq.selenium:selenium-htmlunit-driver to be present in the classpath. When I use this technique, I often employ com.h2database:h2 as my database. H2 can run in-memory and so the database is fresh and empty for each test run. The test does not require you to install an application server, use some inane (sorry) Maven plugin or create any weird XML configuration. It doesn’t require that your application runs on Jetty in production or test environment – it work equally fine for Web applications that are deployed to Tomcat, JBoss or any other application server.
If you are developing a web application for any application server and you are using Maven, this trick has the potential to increase your productivity insanely. Stop what you’re doing and try it out:
Add Jetty and HtmlUnit to your pom.xml
Create a unit test that starts Jetty and navigates to the front page. Verify that the title is what you expect (assertEqual("My Web Application", browser.getTitle()))
Try and run the test
Feel free to contact me if you run into any trouble.
“Problemet med Java er at det krever så mange abstraksjoner. Factories, proxies, rammeverk…” Min samtalepartner gjenfortalte inntrykket han hadde av de Java-programmerende kollegene sine.
Jeg måtte innrømme at jeg kjente meg igjen. Kulturen rundt Java-programmering har noen sykdomstrekk. Kanskje det minst flatterende er fascinasjonen for komplekse teknologiske løsninger. Et gjennomsnittlig Java-prosjekt har rammeverk (Spring, Hibernate), tjenestebusser – gjerne flere (OSB, Camel, Mule), byggverktøy (Maven, Ant, Gradle), persisteringsverktøy (JPA, Hibernate), kodegeneratorer (JAXB, JAX-WS), meldingskøer (JMS), webrammeverk (JSF, Wicket, Spring-MVC) og applikasjonsservere (WebSphere, WebLogic, JBoss). Og det er bare starten. Hvor kommer denne impulsen fra?
Jeg har hørt to teorier om hvorfor Java-programmerere ender opp med en kompleks hverdag. Den ene skylder på sjefene, mens den andre skylder på programmererne. Jeg vil ta for meg begge og peke på en vei ut av jungelen.
Teori 1: Steak and strippers
Zed Shaw, mannen bak “The motherfucking manifesto for programming, motherfuckers” skylder på IT-sjefene. Han mener at selgerne fra teknologiske giganter tar med IT-sjefer ut på strippebuler og kjøper biffmiddager til dem. Og vips så kjøper IT-sjefen inn et ubrukelig verktøy som programmererne er tvunget til å bruke. (Zed påpeker at det vil være positivt med flere kvinnelige IT-sjefer, ettersom de i det minste ikke vil være like interessert i strippebulene)
Ref: Zed Shaw: Control and responsibility (http://zedshaw.com/essays/control_and_responsibility.html). Zed Shaw: Programming, motherfuckers (http://programming-motherfucker.com/). Zed Shaw: Video – Steak and Strippers (http://vimeo.com/2723800).
Argumentet er underholdende, men forklarer bare en del av problemet. Mange av verktøyene som står bak kompleksiteten i Java-prosjekter er åpen kildekode og selges typisk ikke av selgere med fete representasjonskontoer.
Teori 2: Blinkende lys
Jeg tror en viktigere teori er at programmerere er opptatt av fancy ting som blinker. Enkle ting som firmaet tjener penger på er kjedelige. Å sette seg ned og flytte data fra databasen og putte det i en webside er kjedelig for en programmerere. Å lære seg et nytt rammeverk som gjør den samme jobben på en fancy måte er spennende. Å fjerne teknologier og lage en enklere løsning er kjedelig. Å innføre et rammeverk som skal skru sammen teknologien er spennende.
Alle foretrekker å gjøre det de synes er spennende. Jeg vet det, for jeg har vært der selv.
Grunnleggende sett tror jeg at Java-programmere selv har skaffet seg de problemene de ofte klager over med komplekse teknologier.
Veien ut av villmarken
For meg var det viktigste skrittet ut av villmarken foredraget jeg hold på JavaZone for to år siden. I foredraget bygger jeg og Anders Karlsen en webapplikasjon i Java uten å bruke webrammeverk. (Innbild deg at du hører et ironisk gisp her)
Jeg har øvet meg på å løse de problemene jeg har i prosjekter med å bruke enklere teknologier. Slik vet jeg hva de komplekse løsningen gjør. Og så langt er det nesten ingen løsninger som ikke innfører mer problemer enn de fjerner. Det de gjør er at de flytter fokuset fra den oppgaven programmet egentlig skulle løse over til alle de teknologiske delene som man skal få til å fungere sammen.
Jeg tror ikke programmerere er bedre eller dårlige enn andre mennesker når det gjelder det. Men vi har alle en tendens til å bruke tiden vår til å finne spennende problemer å løse i stedet for å løse det vi egentlig skulle på en naiv og enkel måte. Min far sa det egentlig best: Du bør ikke bruke en kalkulator før du kan løse de samme regnestykkene på papir.
In HTML applications, jQuery has changed the way people thing about view rendering. Instead of an input or a text field in the view pulling data into it, the jQuery code pushes data into the view. How could this look in a server side situation like Java?
In this code example, I read an HTML template file from the classpath, set the value of an input field and append more data based on a template (also in the HTML file).
This is a third way from the alternatives of templated views like Velocity and JSP and from component models like JSF. In this model, the view, the model and the binding of the model variables to the view are all separate.
Disclaimer: In this example, I’ve used my still in pre-alpha XML library with the working name of Eaxy. You can get similiar results with libraries like jSoup and JOOX.
Caveat: I’ve never tried this on a grand scale. It’s an idea that compels me for three reasons: First, it’s very explicit. Nothing happens through @annotation, conventions or some special syntax in a template. Second, it’s very unit testable. There is nothing tying this code to running in a web application server. Finally, it’s easy to get to this code through incremental steps. I initially wrote the example application with code that embedded the HTML as strings in Java code and refactored to use the Java Query approach.
Sometimes, small questions lead to big answers. Sometimes these answers are controversial. One such question is “What does this warning about serialVersionUID mean”? All the advice out there basically is for developers who don’t know what’s going on to write code that will ignore errors when something unexpected happens. In my view – this is exactly the wrong approach. The safe way to act is to make sure that your program crashes if you don’t have control.
Java programmers usually get this warning when they write code that looks like this:
On stackoverflow this question has an answer that is extensive, well-written, accepted and, in my opinion, wrong. In short, the answer just recommends to implement something like the following:
Let’s dig deeper.
The reason we get this warning is that HttpServlet implements the interface Serializable. This was a design flaw in the first version of the servlet-api, and now we’re stuck with it. A serialized object can be written to and read from byte streams, such as a file. Here is an example:
In this case, everything is fine. But let’s imagine that some time passes between the write and the read. For example, what if we try to read the file with the next version of the program. A version in which the Person class is changed? For example, what if we changed the implementation of Person to store the name as a single field fullName, instead of two fields firstName and lastName?
In this case, we would get an error message like the following:
But wait, there’s another option. Let’s say we found this problem when we tested if our new version was backwards compatible. Now, we could just cut and paste the serialVersionUID from the stack trace. If we do test our software, this is just as easy as putting the serialVersionUID there in the first place. So, let’s fix the Person class:
Voila! Problem fixed. Our code will run again. Here’s the output of printing the object:
Whoops! By forcing different versions of class Person to have the same serialVersionUID, the code now has a serious bug. FullName should never be null (especially since it’s final!). And what’s worse, the bug is a silent one. If we don’t examine the contents of System.out (in this case), we might not catch it before it escapes into the wild.
When you’re not sure, the correct behavior should be to fail, not to silently do the wrong thing.
If you omit a serialVersionUID field from your class, many changes will cause serialized objects to no longer be readable. Even for trivial changes.
Sadly, classes like HttpServlet, AbstractAction and JFrame which are meant to be subclassed implements Serializable, even though they are almost never serialized in practice. Adding serialVersionUID to these classes would only be noise.
The serialVersionUID field can be added to a class afterward if you actually want to read old objects in a new version of the program. This leaves you no worse off than if you added the serialVersionUID field in the first place.
If the old and the new version of the class are deeply incompatible, giving the class a serialVersionUID when you first create it will cause silent faulty behavior.
I prefer loud, failing behavior that is easy to detect during testing over quiet, faulty behavior that may escape into production. I think you do, too.
I’m working on this idea, and I don’t know if it appeals to you guys. I’d like your input on whether this is something to explore further.
Here’s the deal: I’ve encountered teams who, when working with SOA technologies have been dragged into the mud by the sheer complexity of their tools. I’ve only seen this in Java, but I’ve heard from some C# developers that they recognize the phenomenon there as well. I’d like to explore an alternative approach.
This approach requires more hard work than adding a WSDL (web service definition language. Hocus pocus) file to your project and automatically generating stuff. But it comes with added understanding and increased testability. In the end, I’ve experienced that this has made me able to complete my tasks quicker, despite the extra manual labor.
The purpose of this blog post (and if you like it, it’s expansions) is to explore a more bare-bones approach to SOA in general and to web services specifically. I’m illustrating these principles by using a concrete example: Let users be notified when their currency drops below a threshold relative to the US dollar. In order to make the service technologically interesting, I will be using the IP address of the subscriber to determine their currency.
Step 1: Create your active services by mocking external interactions
Mocking the activity of your own services can help you construct the interfaces that define your interaction with external services.
Spoiler: I’ve recently started using random test data generation for my tests with great effect.
The Publisher has a number of Services that it uses. Let us focus on one service for now: The GeoLocationService.
Step 2: Create a test and a stub for each service – starting with GeoLocationService
The top level test shows what we need from each external service. Informed by this and reading (yeah!) the WSDL for a service, we can test drive a stub for a service. In this example, we actually run the test using HTTP by starting Jetty embedded inside the test.
Validate and create the XML payload
This is the first “bare-knuckled” bit. Here, I create the XML payload without using a framework (the groovy “$”-syntax is courtesy of the JOOX library, a thin wrapper on top of the built-in JAXP classes):
I add the XSD (more hocus pocus) for the actual service to the project and code to validate the message. Then I start building the XML payload by following the validation errors.
In this example, I get a little help (and a little pain) from the JOOX library for XML manipulation in Java. As XML libaries for Java are insane, I’m giving up with the checked exceptions, too.
Spoiler: I’m generally very unhappy with the handling of namespaces, validation, XPath and checked exceptions in all XML libraries that I’ve found so far. So I’m thinking about creating my own.
Of course, you can use the same approach with classes that are automatically generated from the XSD, but I’m not convinced that it really would help much.
Stream the XML over HTTP
Java’s built in HttpURLConnection is a clunky, but serviceable way to get the XML to the server (As long as you’re not doing advanced HTTP authentication).
Spoiler: This code should be expanded with logging and error handling and the validation should be moved into a decorator. By taking control of the HTTP handling, we can solve most of what people buy an ESB to solve.
Create the stub and parse the XML
The stub uses xpath to find the location in the request. It generates the response in much the same way as the ws client generated the request (not shown).
Spoiler: The stubs can be expanded to have a web page that lets me test my system without real integration to any external service.
Validate and parse the response
The ws client can now validate that the response from the stub complies with the XSD and parse the response. Again, this done using XPath. I’m not showing the code, as it’s just more of the same.
The real thing!
The code now verifies that the XML payload conforms to the XSD. This means that the ws client should now be usable with the real thing. Let’s write a separate test to check it:
Yay! It works! Actually, it failed the first time I tried it, as I didn’t have the correct country name for the IP address that I tested with.
This sort of point-to-point integration test is slower and less robust than my other unit tests. However, I don’t find make too big of a deal out of that fact. I filter the test from my Infinitest config and I don’t care much beyond that.
Fleshing out all the services
The SubscriptionRepository, CurrencyService and EmailService need to be fleshed out in the same way as the GeolocationService. However, since we know that we only need very specific interaction with each of these services, we don’t need to worry about everything that could possibly be sent or received as part of the SOAP services. As long as we can do the job that the business logic (CurrencyPublisher) needs, we’re good to go!
Demonstration and value chain testing
If we create web UI for the stubs, we can now demonstrate the whole value chain of this service to our customers. In my SOA projects, some of the services we depend on will only come online late in the project. In this case, we can use our stubs to show that our service works.
Spoiler: As I get tired of verifying that the manual value chain test works, I may end up creating a test that uses WebDriver to set up the stubs and verify that the test ran okay, just like I would in the manual test.
Taking the gloves off when fighting in an SOA arena
In this article, I’ve showed and hinted at more than half a dozen techniques to work with tests, http, xml and validation that don’t involve frameworks, ESBs or code generation. The approach gives the programmer 100% control over their place in the SOA ecosystem. Each of the areas have a lot more depth to explore. Let me know if you’d like to see it be explored.
Oh, and I’d also like ideas for better web services to use, as the Geolocated currency email is pretty hokey.
Changing your perspective even a small amount can have profound effects on how you approach your system.
Let’s say you’re writing a web application in Java. In the system you deal with orders, customers and products. As a web application, your classes include staples like PersonController, PersonRepository, CustomerController and OrderService. How do you organize your classes into packages?
There are two fundamental ways to structure your packages. Either you can focus on the logical tiers, like com.brodwall.myapp.controllers, com.brodwall.myapp.domain or perhaps com.brodwall.myapp.services.customer. Or you can focus on the domain contexts, like com.brodwall.myapp.customer, com.brodwall.myapp.orders and com.brodwall.myapp.products. The first approach is by far the most prevalent. In my view, it’s also the least helpful.
Here are some ways your thinking changes if you structure your packages around domain concepts, rather than technological tiers:
First, and most fundamentally, your mental model will now be aligned with that of the users of your system. If you’re asked to implement a typical feature, it is now more likely to be focused around a strict subset of the packages of your system. For example, adding a new field to a form will at least affect the presentation logic, entity and persistence layer for the corresponding domain concept. If your packages are organized around tiers, this change will hit all over your system. In a word: A system organized around features, rather than technologies, have higher coherence. This technical term means that a large percentage of a the dependencies of a class are located close to that class.
Secondly, organizing around domain concepts will give you more options when your software grows. When a package contains tens of classes, you may want to split it up in several packages. The discussion can itself be enlightening. “Maybe we should separate out the customer address classes into a com.brodwall.myapp.customer.address package. It seems to have a bit of a life on its own.” “Yeah, and maybe we can use the same classes for other places we need addresses, such as suppliers?” “Cool, so com.brodwall.myapp.address, then?” Or maybe you decide that order status codes and payment status codes deserve to be in the “com.brodwall.myapp.order.codes” package.
On the other hand, what options do you have for splitting up com.brodwall.myapp.controllers? You could create subpackages for customer, orders and products, but these subpackages may only have one or possibly two classes each.
Finally, and perhaps most intriguingly, using domain concepts for packages allows you to vary the design according on a case by case basis. Maybe you really need a OrderService which coordinates the payment and shipping of an order, while ProductController only needs basic create-retrieve-update-delete functionality with a repository. A ProductService would just get in the way. If ProductService is missing from the com.brodwall.myapp.services package, this may be confusing or at the very least give you a nagging feeling that something is wrong. On the other hand, if there’s no Controller in the com.brodwall.myapp.product package, it doesn’t matter much.
Also, most systems have some good parts and some not-so-good parts. If your Services package is not working for you, there’s not much you can do. But if the Products package is rotten, you can throw it out and reimplement it without the whole system being thrown into a state of chaos.
By putting the classes needed to implement a feature together with each other and apart from the classes needed to implement other features, developers can be pragmatic and innovative when developing one feature without negatively affecting other features.
The flip side of this is that most developers are more comfortable with some technologies in the application and less comfortable with other technologies. Organizing around features instead of technologies force each developer to consider a larger set of technological challenges. Some programmers take this as a motivating challenge to learn, while others, it seems, would rather not have to learn something new.
If it were my money being spend to create features, I know what kind of developer I would want.
Trivial changes can have large effects. By organizing your software around features, you get a more coherent system that allows for growth. It may challenge your developers, but it drives down the number of hand-offs needed to implement a feature and it challenges the developers to improve the parts of the application they are working on.
Do you know how to apply coding practices the technology stack that you use on a daily basis? Do you know how the technology stack works? For many programmers, it’s easy enough to use test-driven development with a trivial example, but it can be very hard to know how to apply it to the problems you face every day in your job.
A similar approach is likely useful for programmers using any technology. Therefore, I give you: The rules of the Architecture Spike Kata.
Create a web application that lets users register a Person with names and search for people. The Person objects should be saved in a data store that is similar to the technology you use daily (probably a relational database). The goal is to get a spike working as quickly as possible, so in the first iteration, the Person entity should probably only contain one field. You can add more fields and refactor the application later.
The most important rules are Robert Martin‘s three rules of Test-driven development:
No code without test (that is, the code should never do something that isn’t required in order to get a test to pass)
Only enough test to get to red (that is, the tests should run, give an error message and that error message should correct)
Only enough code to get to green (that is, the tests should run and not give an error)
(My addition: Refactor on green without adding functionality)
Secondly, application should be driven from the outside in. That is, your first test should be a top-level acceptance test that tests through http and html. It’s okay to comment out or @Ignore this test after it has run red for the first time.
Lastly, you should not introduce any technology before the pain of not doing so is blinding. The first time you do the kata in a language, don’t use a web framework beyond the language minimum (in Java, this means Servlets, in node.js it’s require('http'), in Ruby it means Rack). Don’t use a Object-Relational Mapping framework. Don’t use a dependency injection framework. Most definitely don’t use an application generator like Rails scaffold, Spring Roo or Lift. These frameworks can be real time savers, but this kata is about understanding how the underlying technology works.
As a second iteration, use the technologies you use on a daily basis, but this time set up from scratch. For example, if your project uses Hibernate, try configuring the session factory by hand. By using frameworks in simplest way possible, you’ll both learn more about what they bring to the table and how to use them properly. For complex technology like Hibernate, there’s no substitute for deeper understanding.
This is what I’ve learned about working with web applications in Java:
Most Java web frameworks seem to harm more than they help
Hibernate is a bitch to set up, but once it’s working, it saves a lot of hassle
Using TDD with Hibernate helped me understand how to use Hibernate more effectively
I’ve stopped using dependency injection frameworks (but kept on using dependency injection as a pattern)
I have learned several ways to test web applications and database access independently and integrated
I no longer have to expend mental energy to write tests for full stack application
The first time I write this kata with another developer, it takes around 3 to 5 hours, depending on the experience level of my pair. After running through it a few times, most developers can complete the task in less than an hour.
We get better through practice, and the Architecture Spike Kata is a way to practice TDD with the technologies that you use daily and get a better understanding of what’s going on.
On larger projects, I’ve always ended up resorting to writing down a lot of detailed specifications, many of which are wrong, irrelevant or we might not be ready to answer them yet. On small projects, the dialogue between the customer and the developers can flow easy, and good things happen.
The quick analysis
Developer: … so we’re going to complete the current task tomorrow or the day after. Could we discuss what to do next before you’re off to your next meeting?
Customer: Sure. The next task on the backlog is to send the payment request to the accounting system
Developer: Yeah. Um… well, we’re not really ready for the integration tasks yet. I was thinking that we might perhaps pick another user interface task. Would that be okay?
Customer: No problem. Let’s see…. how about “road project submits funding request”?
Developer: Great. What’s that?
Customer: Well, today, the owner of the road project sends in a paper form. Let’s see, the form is available as a PDF on our web site. Let’s take a look.
Developer: Let’s see…. We know how to deal with social security number, names, addresses. This field “municipality,” that’s the usual stuff, right? (customer nods) What about “request id”? What’s that?
Customer: It’s just a sequence number per municipality. It would be really nice if you could suggest a request number and then let the user override.
Developer: That should be doable.
They work out a few more details and get on their way. Total time, 30 minutes
The low ambition implementation
Developer: So, we worked on the funding request UI for the last two days. I think we’ve got something that looks pretty good.
Customer: Great, let’s see it! Let’s see… There’s a new “apply for funding request” menu item. Nice. But it should be “Fund road project” (developer notes feedback). Let me try and press it… Hmm… “select municipality”? Why do I get that?
Developer: Well, we’re still a bit shaky on AJAX. So in order to find the next sequence number, we would like to have the user select municipality first.
Customer: Oh, AJAX. That’s interactive web pages, right? Yeah, I guess that’s fine, but we need to fix up the text a little. So, what happens when I select one…? Nice, it sort of looks like the paper form. That will help us get learn it. But let’s move the fields around a little (he sketches a few change on a piece of paper).
Developer: We were a bit uncertain about this “sum” field. Does the user have to fill in the sum himself? He’s already filled in all the numbers.
Customer: That doesn’t make too much sense. How about if the web pages updates the sum field dynamically? Oh, you’ve got that ‘don’t make me do dynamic webpages’ face again.
Developer: Yeah, sorry. The last time it was a big mess. Can we hold off on that for now? Do you really need it?
Customer: (laughs) No, that’s okay. I guess we can just remove the field. That should be fine.
Developer: Yay! What about validating the bank account number? We left that out, because the documentation is…. well, insane. That alone will take at least another day.
Customer: No, we really need that. People often type it wrong, and then it’s a nightmare to fix all the errors that follow. You just have to fight though it.
Developer: (sigh) I guess it can’t be helped. We’re on it.
Customer: All in all, it looks pretty good. I’ll go over it by myself after lunch and bring you back some issues tomorrow.
The quick completion
Customer: Here’s a list of corrections. Nothing major, just a few misplaced fields and a few extra validation rules.
Developer: Great. We spent all of yesterday doing the bank account number validation. Man, that’s some bad doc! But we should be done before lunch.
Customer: I have an opening in my calendar at the end of the day, let’s take a look then and call it done. Great work, guys!
Have you ever been on a small project where everyone has a pretty good understand of what’s going on and the communication works well? Developers can make a pretty good guess at the details and develop a pretty good first version. There may be some more work that’s needed, but these can be negotiated when the developers have a better idea of what’s going to be easy and what’s going to be hard.
When I’ve been on larger projects, this communication seems to break down. Instead the developers require a detailed specification in writing. And if something is missing from the specification, they reject it and ask for further details. And they won’t show the product to the customer before all the kinks are worked out.
When we do it well it’s not so bad, getting a detailed specification and finishing the task before we get real feedback. But it requires the customer to decide on things that may not be important, but that may impact cost substantially. It requires back and forth to get the specification written, reviewed, amended etc.
On a small project, a developer can understand the purpose, build something quickly and adjust it based on feedback. Is the reality of larger projects that this will just remain a pipe dream? If so, why? If not, how do you get the quick and informal feedback?
Oracle released Java 7 on July 28, 2011. This is nearly 5 years after the release of Java 6 in December 2006. The release received a lot of bad press, both because it is very meager on features, and because it shipped with a severe bug. Nevertheless, once the most serious bugs have been fixed, you might think about starting to use Java 7. What will this mean?
New language features
Java 7 has a few new language features. Sadly, the most exciting ones have been postponed until Java 8. The following 3 features may show up in your pretty quickly, though:
This is the try-with-resources or Automatic Resource Management block. If you declare a variable in the try() statement, Java automatically calls close on it, like you would in a finally block. This is a small improvement, but nice. You can use try-with-resources on your own object by implementing the new interface java.lang.AutoCloseable.
This is the multi-catch statement. It’s useful because of the load of checked exceptions on the sanity of your average Java-developer. It’s nice, but hardly revolutionary. It makes me really wish we got rid of checked exceptions, though.
This is type inference for Generic Instance Creation. Saves a few keystrokes without removing any type safety. Again, a nice, but very small improvement.
There are a few more language features, but I expect they will see very little use.
The Java virtual machine gets a new instruction: invokedynamic. Using invokedynamic, the JVM can invoke a method on an object without having to know on which class or interface the method is declared. If it walks like a duck and talks like a duck…
Invokedynamic will be very helpful for implementors of dynamic languages in the JVM, so it’s great. But the average developer will never encounter it in the wild.
Looking at the release notes for Java 7, you may first suspect that there are some interesting library changes here. However, when examining the list more thoroughly, I couldn’t find a single change that I expect I will actually use. The library changes are mostly low-level, behind the scenes fixes of small problems.
So there it is: try-with-resources, multi-catch and a very limited type inference. Hopefully, Java 8 will be released as planned in late 2012 with all the stuff we’ve been waiting for. If so, I expect most shops will skip Java 7. But if Java 8 follows the pattern of delays from Java 7, these slim pickings may be all the crumbs the Java community gets for another five years.