This article is a summary of a seminar I had on the topic. If it seems like it’s a continuation of an existing discussion that’s because, to some extent, it is. If you haven’t been discussing exchanging your app server, this article probably isn’t very interesting to you.

By putting the application server inside my application instead of the other way around, I was able to leap tall buildings in a single bound.

The embedded application server

This is how I build and deploy my sample application to a new test environment (or to production):

1. mvn install
2. scp someapp-server/target/someapp-1.0.onejar.jar appuser@appserver:/home/appuser/test-env1/
3. ssh appuser@appserver "cd /home/appuser/test-env1/ && java -jar someapp-1.0.onejar.jar&"

This require no prior installed software on the appserver (with the exception of the JVM). It requires no prior configuration. Rolling back is a matter of replacing one jar-file with another. Clustering is a matter of deploying the same application several times.

In order to make this work in a real environment, there are a many details you as a developer need to take care of. As a matter of fact, you will have to take responsibility for your operational environment. The good news is that creating a good operational environment is not more time-consuming than trying to cope with the feed and care of a big-a Application Server.

In this scheme every application comes with its own application server in the form of jetty’s jar-files embedded in the deployed jar-file.

The advantages

Why would you want to do something like this?

• Independent application: If you’ve ever been told that you can’t use Java 1.5 because that would require an upgrade of the application server. And if we upgrade the application server, that could affect someone else adversely. So we need to start a huge undertaking to find out who could possibly be affected.
• Developer managed libraries: Similar problems can occur with libraries. Especially those that come with the application server. For example: Oracle OC4J helpfully places a preview version of JPA 1.0 first in your classpath. If you want to use Hibernate with JPA 1.0-FINAL, it will mostly work. Until you try to use a annotation that was changed after the preview version (@Discriminator, for example). The general rule is: If an API comes with your app server, you’re better served by staying away from it. A rather bizarre state of affairs.
• Deployment, configuration and upgrades: Each version of the application, including all its dependencies is packaged into a single jar-file that can be deployed on several application server, or several times on the same application server (with different ports). The configuration is read from a properties-file in the current working directory. On the minus side, there’s no fancy web UI where you can step through a wizard to deploy the application or change the configuration. On the plus side, there is no fancy web UI …. If you’ve used one such web UI, you know what I mean.
• Continuous deployment: As your maven-repository will contain stand alone applications, creating a continuous deployment scheme is very easy. In my previous environment, a cron job running wget periodically was all that was needed to connect the dots. Having each server environment PULL the latest version gives a bit more flexibility if you want many test environments. (However, if you’re doing automated PUSH deployment, it’s probably just as practical for you).
• Same code in test and production: The fact that you can start Jetty inside a plain old JUnit test means that it is ideal for taking your automated tests one step further. However, if you test with Jetty and deploy on a different Application Server, the difference will occasionally trip you. It’s not a big deal. You have to test in the server environment anyway. But why not eliminate the extra source of pain if you can?
• Licenses: Sure, you can afford to pay a few million $ for an application server. You probably don’t have any better use for that money, anyway, right? However, if you have to pay licenses for each test-server in addition, it will probably mean that you will test less. We don’t want that.
• Operations: In my experience, operations people don’t like to mess around with the internals of an Application Server. An executable jar file plus a script that can be run with [start|status|stop] may be a much better match.

The missing bits

Taking control of the application server takes away a lot of complex technology. This simplifies and makes a lot of stuff cheaper. It also puts you back in control of the environment. However, it forces you to think about some things that might’ve been solved for you before:

• Monitoring: The first step of monitoring is simple: Just make sure you write to a log file that is being monitored by your operations department. The second step requires some work: Create a servlet (or a Jetty Handler) that a monitoring tool can ping to check that everything is okay. Taking control of this means that you can improve it: Check if your data sources can connect, if your file share is visible, if that service answers. Maybe add application-calibrated load reporting. Beyond that, Jetty has good JMX support, but I’ve never needed it myself.
• Load balancing: My setup supports no load balancing or failover out of the box. However, this is normally something that the web server or routers in front of the application server anyway. You might want to look into Jetty’s options for session affinity, if you need that.
• Security: Jetty supports JAAS, of course. Also: In all the environments I’ve been working with (CA SiteMinder, Sun OpenSSO, Oracle SSO), the SSO server sends the user name of the currently logged in user as an HTTP header. You can get far by just using that.
• Consistency: If you deploy more than one application as an embedded application server, the file structure used by an application (if any) should be standardized. As should the commands to start and stop the application. And the location of logs. Beyond that, reuse what you like, recreate what you don’t.

Taking control of your destiny

Using an embedded application server means using the application server as a library instead of a framework. It means taking control of your “main” method. There’s a surprisingly small number of things you need to work out yourself. In exchange, you get the control to do many things that are impossible with a big-A Application Server.

Thanks to Dicksen, Eivind, Terje, Kristian and Kristian for a fun discussion on Jetty as a production app server

For some reason, most projects I’ve seen with little experience in iterative development prefer three week iterations. Personally, I prefer two week iterations. Here is the breakdown:

• Three week iterations: After three months, you’ve spent about 7% of your time on iteration meetings. You’ve had 4 opportunities to improve.
• Two week iterations: After three months, you’ve spent about 10% of your time on iteration meetings. You’ve had 6 opportunities to improve.
• One week iterations: After three months, you’ve spent about 20% of your time on iteration meetings. You’ve had 12 opportunities to improve.

Going from 93% to 90% efficiency for a 50% increase in learning seems like a good deal. Going from 90% to 80% efficiency for a 100% increase in learning, not so much.

These numbers are of course greatly simplified. You might also consider:

• With shorter iterations, the planning time may go down. But this takes practice – it doesn’t happen automatically.
• With very short iterations, you may not have experienced enough to learn much from the retrospective. However, if you find that you do a timeline, and most of the things people remember happened the last week, it may not be because that’s the only time something significant happened.
• You may consider different frequencies for different ceremonies. For example, on my current project we want to have demos with our power users. But they have to travel far to visit us. So we only have a full demo every other four weeks. We plan every two weeks and have an internal review and retrospective every two weeks.

What’s the right iteration length for your project?

I’ve not yet decided what posts to publish here and what posts to publish on Sterk Blanding. My present idea is that I’ll publish most of my English articles here at Thinking inside a Bigger Box and Norwegian language articles at Sterk Blanding. But I can be persuaded to change my mind.

For my Norwegian readers, enjoy my articles on Sterk Blanding:

• Hvordan komme i gang med blogging: Some tips for people who’re considering to write on a blog
• Hemmligheten bak gode spesifikasjoner: How to change your tests from specifying behavior to specify intentions
• “Det var dyrt”-øyeblikket: The sobering moment in my project when both I and my customer realized that the transparency of Scrum meant that we could see how much the project was really costing.
• Hvordan endre en statisk klasse til en dynamisk singleton: A step by step guide to an important strategic level refactoring in legacy systems: Changing static calls to use a singleton that can be mocked.
• Med Ram Yoga: Smidig brukervennlighet: Some practical tips on how to use agile software development and user centric design to complement each other

Top five shortcuts

There are some keyboard short cuts that everyone who uses Eclipse should know:

• ctrl-1 (quick fix): You hopefully use this shortcut to get quick fix support for compiler errors and warnings. Did you know that you can also use it to assign parameters to new fields, rename variables, and invert if-statements and equals-checks? Learn to think of ctrl-1 as asking Eclipse “what can you do make the code better (or just different)?”
• ctrl-space (complete): Again, you hopefully know that you can use ctrl-space to complete the name of variables and method names. But did you know that you can type “equa<ctrl-space>” in the class body and have Eclipse override the equals-method for you? Or that you can type “getNam<ctrl-space>” and have Eclipse create the whole implementation of a getter for name (if there’s a name-field in the class). Or that you can type “Test<ctrl-space>” and have Eclipse fill in the Test code template. Think of ctrl-space as asking Eclipse “guess what I’m about to write”
• ctrl-f6 (next editor): Use this to cycle between open files. It really should’ve been bound to ctrl-tab, but you can do this yourself.
• f3 (go to definition): Place the cursor on a method call or variable usage and press f3 to go to it’s definition.
• ctrl-shift-t (open type): A nifty dialog to go to any class in your project. Did you know that typing PerCoT will take you to PersonControllerTest?

Top ten runner ups

Here are some eye-openers that people enjoy learning:

• alt-ctrl-down (copy current line): Creates a new copy of the line under the cursor on the next line. Without wiping the clipboard! Try it while selecting several lines, too
• alt-down (move current line): Moves the line under the cursor down one line. Works with alt-up, too. And with a number of lines selected. A quick way to move code around with the keyboard.
• alt-shift-left (extend selection): Progressively selects a larger syntactic element in the editor. Hard to explain. Try it out!
• ctrl-shift-m (static import): Replace a call to Assert.assertEquals with a static import of the org.junit.Assert.assertEquals and a call to assertEquals.
• ctrl-F11 (rerun latest command): To run for example the same test again, you can usually press ctrl-f11. Sadly, a few years back the Eclipse team tried to improve this and failed. Fix it under Windows->Preferences, Run/Debug -> Launching. Change “Launch operation” to “Always launch previous”.
• f12 (activate editor): When you perform an operation where some other pane got the focus, use f12 to return to the editor again.
• ctrl-N (new <something>): Create a new class, XML file or whatever. Be sure to use the filter
• alt-shift-l (extract local variable): My favorite refactoring. Select an expression and press alt-shift-l to assign it to a local variable and replace all uses of the expression with that variable.
• alt-shift-m (extract method): Your bread and butter refactoring to split up complex logic in understandable units.
• alt-shift-i (inline method/inline variable): The inverse of both alt-shift-l and alt-shift-m. Together, these three refactorings let you resculpt your code while being certain that the behavior is unaltered.

Top three properties to change

When I sit down with new programmers, I almost always help them make the following changes in the preferences. Find the preferences under Window->Preferences:

• Use ctrl-tab (and ctrl-shift-tab) to switch between open editors: Go to General->Keys, type in “next editor” in the filter. Select “Copy command” and type “ctrl-tab” in the Binding field. Do the same for “previous editor” and ctrl-shift-tab.
• Type filter: Do you wonder why Eclipse can’t understand that when you say List, you mean java.util.List, not org.hibernate.mapping.List or (ye gods!) java.awt.List. Well, you can make Eclipse understand. Put classes and packages you don’t like under Java->Appearance->Type filters. If your project is like mine, putting org.hibernate.mapping.*, antlr.*, java.awt.List, and com.sun.* in the list makes List unique to java.util.List. Then “organize imports” and completion works as you want.
• Static import favorites: Do you find yourself using static imports with the same few classes again and again? The preference Java->Editor->Content Assist->Favorites lets you list up classes which will have their static methods checked when you press ctrl-space to complete a method call. org.junit.Assert.* is a good first candidate.

Learning your tool

All IDEs are rich and powerful tools. Spending some time to learn a few new tricks is well worth the effort.

Up into the start of last year, I only worked sporadically with pair programming. Last year, I was lucky enough to be part of a team that used pair programming all the time. Since I’ve experienced real pair programming, I never want to give it up.

Pair programming offers benefits to many stakeholders:

• As a developer, you will have more fun at work. You will get to know your colleagues better and experience flow practically the whole day. You will be tired by the end of the day, but you will also feel like you’ve accomplished good work.
• The team will have a higher quality code base that everyone is comfortable with.
• As an architect or team lead, you will have a good way to contribute even if you only have a little time before a meeting. You will also have a better chance to influence the rest of the team, instead of just issuing edicts that nobody follows.
• As the project manager, you will have a more flexible team. If someone gets sick, goes on vacation or moves to another project, there won’t be a big problem.
• As the customer, you will get better quality code faster.

With these benefits in mind, why doesn’t everybody pair program? Well, it is unfamiliar, a little scary, and exhausting when you start out. Most developers are not used to having other watch them code. Or to focus on the task at hand the whole day.

Here are some techniques I’ve seen have effect for teams transitioning to pair programming:

• Code dojos: Everyone on the team gets together and programs a sample program or a spike together. Two people sit at the keyboard, while the rest watch on a projector. Rotate pairs frequently. This lets everyone get comfortable with coding as a social activity.
• Pair programming should be the norm, but allow for exceptions. If people only pair program occasionally, they end up not pair programming at all. If people are forced to pair program when they just need some time by themselves to think, they will not be happy pair programming.
• The pair programming star: Write the names of the team members in a circle. Every time two people pair program, draw a line between their names. Keep the pair programming star in a visible location.
• Facilities: The furniture can make it harder to get started pair programming. Consider using two mice, two keyboards and perhaps two monitors per PC to make it easier. Or use VNC for desktop sharing.
• Give it time: Pair programming is exhausting when you first start doing it. It will take a while before people are comfortable with the new pace. But once they switch, they will never want to go back.

Resources

For more inspiration, see these presentations from the Smidig 2009 conference (in Norwegian):

• Jøran Lillesand: Derre e itj smidi!: On why pair programming is the very foundation of successful agile projects
• Ørjan Lillevik og Kari Røssland: Parprogrammering gir driv: On what it feels like to be on a team that adopts pair programming. From reluctance to joy.

But I thought I’d post the video here, to see what you all think. Comments are welcome!

I hope the video will demonstrate how to use refactoring effectively to drive the design of a program.

I chose the FizzBuzz kata – that is, to generate a sequence of numbers where every number divisible by three is replaced by “fizz” and every number divisible by five is replaced by “five”. The music changes to be more aggressive just as I induce a new requirement into the kata: The FizzBuzz generator should be programmable, so, in the kata, numbers divisible by two are replaced by “coconut” and numbers divisible by seven are replaced by “banana”.

Thanks to Emily Bache for the inspiration for the kata.

Enjoy!

Fizz buzz code kata from Johannes Brodwall on Vimeo.

The video was made with IntelliJ IDEA Community Edition on Windows Vista (!) with BB FlashBack Express (free screen recorder), converted to AVI with Windows Media 1 codec and uploaded to Vimeo.

• Java EE Spike: An application that stores People with names to a database and lets me search for them. I’ve repeated this pair programming with several different programmers.
• Programmable Fizz Buzz: Create a sequence of numbers 1,2,fizz,4,buzz,fizz,… you know the one. And the twist: Make it programmable, so that for example numbers divisible by 7 should be replaced with “coconut”.

I’ve learned a lot from repeating these exercises:

• Using test-driven development it takes me longer to get to something that “should work in principle”, but shorter to get to something that works correctly.
• When refactoring to a new data structure, add the new structure while keeping the old one, make switching between them as simple as changing a single line. Delete the old when it works.
• There’s always an automated refactoring you still want to help you out. Extract Parameter Object was my big one.
• Writing for example a method invocation and then using quickfix to have the IDE generate the method is the quickest way of writing code available to you.
• After 8 iterations, the Java EE Spike takes me 80 minutes solo. Pair programming with another programmer who had practiced: 65 minutes. I don’t know why!
• Pair programming a moderately complex kata like the Java EE Spike is fun. It’s also a good chance to discuss different roles of different tests.
• There is a huge difference between a test that takes 3 seconds to run and one that takes 0.5 seconds when you’re test driving. More surprisingly, there’s a big difference between a test that takes 0.5 seconds and one that takes 0.01 seconds
• If you think test-driven development is not for you or that it’s bunk, you probably write really slow tests.

What are your latest coding observations?

The first years I was working in this business, I didn’t see any distinction between “the user” and “the customer”. Once I started seeing the distinction, I started to understand that the person who is going to use the system we’re developing is not the person who defines what the system should do and neither of these is usually the person that pays me to develop the system. So I starting distinguishing between the product owner, that is, the customer and the end user. But the product owner often calls the person I call “end user” his “customer”. What’s going on here? Let’s check the dictionary:

CUSTOMER
Main Entry: cus·tom·er
Pronunciation: \ˈkəs-tə-mər\
Function: noun
1: one that purchases a commodity or service
2: an individual usually having some specified distinctive trait

CLIENT
Main Entry: cli·ent
Pronunciation: \ˈklī-ənt\
Function: noun
1: one that is under the protection of another : dependent
2a: a person who engages the professional advice or services of another
2b: customer
2c: a person served by or utilizing the services of a social agency
2d: a computer in a network that uses the services (as access to files or shared peripherals) provided by a server

I’ve seen suppliers approach their work by asking for a specification of a product to deliver and then trying to deliver something to that specification for payment. The mental model is that of a customer going to the grocery story asking for “eight pounds of CRM software”. My experience with organizations with this sort of mindset has always been unsatisfactory.

On the other hand, I’ve seen suppliers approach their work as an agent of the organization that pays them. “Our job is to enable someone else do their job better.” This totally changes the way an organization deals with this relationship. The word “customer” may not be conductive to this sort of thinking. Instead, we should think of ourselves as agents acting on behalf of a client. As an agent, your responsibility is to enable your client. This includes helping your client to find better means of reaching their goal.

By the way, wikipedia defines the word “agent” as “a person who is authorized to act on behalf of another (called the Principal or client) to create a legal relationship with a Third Party”. If the “third party” is the computer, then a good developer is an agent acting on their clients behalf in dealings with the computer software.

Why doesn’t the software industry use the word “client” instead of “customer”?

The results of the experiment were interesting, but far from conclusive.

During the experiment, we gave the same set of requirements to three teams, each consisting of three estimators. Each team was told to use a different technique. We decided on the following techniques:

• Planning poker: The purpose is to give all requirements a relative number (the meaning of these numbers will later be measured based on the output of the iterations). Each estimator has a deck of cards and choose a card with the number he feels is appropriate for the current requirement. Everyone reveals the numbers at the same time to avoid anchoring. The team discusses and reestimates a requirement until their estimates converge.
• Table spread estimation: This is one of the new techniques we proposed. Each requirement is written on a card. The estimators spread the cards along a large table according to the relative effort required per requirement. Numbers can be imposed later if desired.
• Goldilocks estimation: The purpose is to restructure requirements until they all have roughly equal size. Instead of assigning a number to a requirement, the estimators pick one of three options: Too big (split up and estimate the parts again), Too small (merge with other requirements), or Just right. When all requirements have been split or merged into “Just Right” size, the estimation is complete.

All teams found their estimation techniques to be motivating, but the Table Spread and Goldilocks groups managed to complete the estimation much faster. The Table Spread estimation would obviously need more space if we had a lot of requirements, while the Goldilocks estimation would generate a large number of requirements.

Based on these experiences, we propose the following experiment in a project:

• Use Table Spread Estimation for Release planning. This will encourage the team to keep the number of requirements low instead of trying to plan too detailed too far ahead. Since the table spread is quick it can be redone every iteration.
• Use Goldilocks Estimation for the next few upcoming iterations to split up the requirements into equal sized items. This will generate a better set of work items. The shorter planning window will ensure that we won’t have an unmanageable number of requirements.

These are currently very rough ideas and we have no idea of whether it will work as we expect. Let me know if you have any relevant experience or if you want more information.

1. Effect Enterprise Java architecture in 2009

Since the Effective Enterprise Java book was written, many of the topics regarding transactions, concurrency and shared state have been resolved. Here are the basic guidelines of an enterprise application in Java as of 2009:

1. All processing is triggered by an event, such as an http-request, a timer or an incoming message
2. Each processing event is handled in an isolated scope, never touching the data of another processing event. All coordination of data happens through the data layer. This means that objects are either stateful and short-lived or stateless and immortal.
3. Each processing event is either completed or aborted totally. Very few applications will benefit from trying to automatically recover from most problems.
4. Inconsistent updates are resolved when transactions are committed, usually through optimistic locking.

Some things I told the attendants to consider: First, today most people consider EJBs to be more trouble than value (with the exception of Entity beans 3.0 which is JPA which is really mostly Hibernate, which really isn’t very much EJB). Second, all triggers can be forged. We return to the second issue when we discuss security.

2. Web integration testing

I showed a practical demo using WebDriver and Jetty to perform web integration tests as JUnit test. The remarkable things about this example is that it requires no installation of an app server (Jetty is installed as a Maven dependency), it requires no separate starting of an application server (Jetty can run embedded in the test) and it is very fast (Jetty starts up in about 200 milliseconds).

3. Hibernate integration testing

I showed a practical example of how to test a DAO implemented with Hibernate. The remarkable things about this demonstration was that, again, no installation or startup is required (I use H2 as an in-memory database).

Hibernate is a power tool. I use the following analogy: If you’re building a tunnel and need to mine through a mountain, you want to use dynamite. If you want to remove a rock from you back yard, you may want to use dynamite. But if you don’t know what you’re doing, chances are you may blow your foot off.

Hibernate is like that dynamite. You need knowledge and safety measures to deal with it correctly, but when you do, it can save you a lot of effort. Creating JUnit tests for your Hibernate code is one such safety measure.

4. Security

Almost all the threats an application developer should be concerned with are in the same class, namely that of Injection attacks. An injection attack is when a client tricks another process into treating data as instructions. For example by using SQL meta-characters:

An important source of injection vulnerabilities is HTML injection, also known as Cross-Site Scripting (XSS).

In both situations, and in all others, there’s one important guideline: Data from the outside world should be considered “tainted”. Never use tainted data in unsafe ways. When reading input parameters, validate against malicious characters (but please don’t make poor “O’Reiley” unable to use your system). When writing HTML pages, always escape tainted data. When using tainted data during access to the database or with HSQL or JPAQL, always use PreparedStatement and send in data as parameters.

Another often overlooked exploit is request forgery, often used in combination with phishing attacks. To protect your users from request forgery, supply an authentication token as a hidden field with all forms. Or if you’re lazy: Make sure all operations have confirmation dialogs.

5. Continuous Deployment

Continuous Deployment is the practice of rolling out a deployment to a server after every successful build on your Continuous Integration server. I described two ways of doing Continuous Deployment during the tutorial, but I will restrict this discussion to the more modern one.

Most teams doing continuous deployment use Maven or Ant to invoke the deployment tools of their respective application servers. Many application servers make this pretty hard, but the hardest part of the battle if finding out what command needs to be invoked. The Continuous Integration server can be configured to run this task.

After doing deployment, it is a good idea to run some sort of system level integration tests. Teams use replay of production data, load generators like JMeter and webcrawlers that validate HTML and CSS to do automated non-functional integration tests. If you keep your logs clean, you can actually gain quite a bit of confidence just by looking at the logs after applying simulated load to your system.

Some projects take this even further, by continuously deploying to production. Both IMVU and Flickr are known to practice this.

At any rate, the practice of doing continuous deployment should lead you to consider how to simplify your deployment and runtime configuration, which will result in an easier installation procedure into production, even if it’s not automated.

Summary

Effective Enterprise Java development has progressed a lot since 2004. Much of the emphasis now is on how to improve testing in enterprise Java applications. The way applications usually process data has stabilized as well, with most application preferring each event to be processed in an isolated, transactional context with very little automated recovery.

In the end, Effective Enterprise Java is a lot simpler in 2009 than it was in 2004.

Material

• My slides, including topics that we didn’t discuss as well as code for all the examples
• The complete source code for one iteration of my Enterprise Java Kata, including a pom.xml file with all dependencies needed to get the tests running