Java web+database applications are usually filled to the brim with technologies. Many of these are hard to test and many of these may not add value. In order to explore TDD and Java applications, I practiced the Java EE Spike Kata in 2010. Here’s a video of me and Anders Karlsen doing this kata at JavaZone 2010.

A similar approach is likely useful for programmers using any technology. Therefore, I give you: The rules of the Architecture Spike Kata.

The problem

Create a web application that lets users register a Person with names and search for people. The Person objects should be saved in a data store that is similar to the technology you use daily (probably a relational database). The goal is to get a spike working as quickly as possible, so in the first iteration, the Person entity should probably only contain one field. You can add more fields and refactor the application later.

The rules

The most important rules are Robert Martin‘s three rules of Test-driven development:

• No code without test (that is, the code should never do something that isn’t required in order to get a test to pass)
• Only enough test to get to red (that is, the tests should run, give an error message and that error message should correct)
• Only enough code to get to green (that is, the tests should run and not give an error)
• (My addition: Refactor on green without adding functionality)

Secondly, application should be driven from the outside in. That is, your first test should be a top-level acceptance test that tests through http and html. It’s okay to comment out or @Ignore this test after it has run red for the first time.

Lastly, you should not introduce any technology before the pain of not doing so is blinding. The first time you do the kata in a language, don’t use a web framework beyond the language minimum (in Java, this means Servlets, in node.js it’s require('http'), in Ruby it means Rack). Don’t use a Object-Relational Mapping framework. Don’t use a dependency injection framework. Most definitely don’t use an application generator like Rails scaffold, Spring Roo or Lift. These frameworks can be real time savers, but this kata is about understanding how the underlying technology works.

As a second iteration, use the technologies you use on a daily basis, but this time set up from scratch. For example, if your project uses Hibernate, try configuring the session factory by hand. By using frameworks in simplest way possible, you’ll both learn more about what they bring to the table and how to use them properly. For complex technology like Hibernate, there’s no substitute for deeper understanding.

What to expect

So far, I’ve only done the Architecture Spike Kata in Java. But on the other hand, I’ve done it around 50 times together with more than ten other developers. I’ve written about how to get started with the Java EE Spike Kata (in Norwegian) on my blog before.

This is what I’ve learned about working with web applications in Java:

• Most Java web frameworks seem to harm more than they help
• Hibernate is a bitch to set up, but once it’s working, it saves a lot of hassle
• Using TDD with Hibernate helped me understand how to use Hibernate more effectively
• I’ve stopped using dependency injection frameworks (but kept on using dependency injection as a pattern)
• I have learned several ways to test web applications and database access independently and integrated
• I no longer have to expend mental energy to write tests for full stack application

The first time I write this kata with another developer, it takes around 3 to 5 hours, depending on the experience level of my pair. After running through it a few times, most developers can complete the task in less than an hour.

We get better through practice, and the Architecture Spike Kata is a way to practice TDD with the technologies that you use daily and get a better understanding of what’s going on.

This is my MyScrum:

• I want to measure velocity every week
• I want to demostrate the product with a cadence that makes sure users show up to the demo
• I don’t want to have story point estimates, I want to have story point budgets
• I want the product owner, not the team, to own the budget/estimate (but they team may veto)
• I don’t want commitments or forecasts from the team, I want measured historical progress
• I want to plan per story, not per sprint
• I want the developers who will develop a story to follow it (or pass the baton to other developers) from detailing to deployment.

What does your MyScrum look like?

The quick analysis

Developer: … so we’re going to complete the current task tomorrow or the day after. Could we discuss what to do next before you’re off to your next meeting?

Customer: Sure. The next task on the backlog is to send the payment request to the accounting system

Developer: Yeah. Um… well, we’re not really ready for the integration tasks yet. I was thinking that we might perhaps pick another user interface task. Would that be okay?

Customer: No problem. Let’s see…. how about “road project submits funding request”?

Developer: Great. What’s that?

Customer: Well, today, the owner of the road project sends in a paper form. Let’s see, the form is available as a PDF on our web site. Let’s take a look.

Developer: Let’s see…. We know how to deal with social security number, names, addresses. This field “municipality,” that’s the usual stuff, right? (customer nods) What about “request id”? What’s that?

Customer: It’s just a sequence number per municipality. It would be really nice if you could suggest a request number and then let the user override.

Developer: That should be doable.

They work out a few more details and get on their way. Total time, 30 minutes

The low ambition implementation

Developer: So, we worked on the funding request UI for the last two days. I think we’ve got something that looks pretty good.

Customer: Great, let’s see it! Let’s see… There’s a new “apply for funding request” menu item. Nice. But it should be “Fund road project” (developer notes feedback). Let me try and press it… Hmm… “select municipality”? Why do I get that?

Developer: Well, we’re still a bit shaky on AJAX. So in order to find the next sequence number, we would like to have the user select municipality first.

Customer: Oh, AJAX. That’s interactive web pages, right? Yeah, I guess that’s fine, but we need to fix up the text a little. So, what happens when I select one…? Nice, it sort of looks like the paper form. That will help us get learn it. But let’s move the fields around a little (he sketches a few change on a piece of paper).

Developer: We were a bit uncertain about this “sum” field. Does the user have to fill in the sum himself? He’s already filled in all the numbers.

Customer: That doesn’t make too much sense. How about if the web pages updates the sum field dynamically? Oh, you’ve got that ‘don’t make me do dynamic webpages’ face again.

Developer: Yeah, sorry. The last time it was a big mess. Can we hold off on that for now? Do you really need it?

Customer: (laughs) No, that’s okay. I guess we can just remove the field. That should be fine.

Developer: Yay! What about validating the bank account number? We left that out, because the documentation is…. well, insane. That alone will take at least another day.

Customer: No, we really need that. People often type it wrong, and then it’s a nightmare to fix all the errors that follow. You just have to fight though it.

Developer: (sigh) I guess it can’t be helped. We’re on it.

Customer: All in all, it looks pretty good. I’ll go over it by myself after lunch and bring you back some issues tomorrow.

The quick completion

Customer: Here’s a list of corrections. Nothing major, just a few misplaced fields and a few extra validation rules.

Developer: Great. We spent all of yesterday doing the bank account number validation. Man, that’s some bad doc! But we should be done before lunch.

Customer: I have an opening in my calendar at the end of the day, let’s take a look then and call it done. Great work, guys!

The contrast

Have you ever been on a small project where everyone has a pretty good understand of what’s going on and the communication works well? Developers can make a pretty good guess at the details and develop a pretty good first version. There may be some more work that’s needed, but these can be negotiated when the developers have a better idea of what’s going to be easy and what’s going to be hard.

When I’ve been on larger projects, this communication seems to break down. Instead the developers require a detailed specification in writing. And if something is missing from the specification, they reject it and ask for further details. And they won’t show the product to the customer before all the kinks are worked out.

When we do it well it’s not so bad, getting a detailed specification and finishing the task before we get real feedback. But it requires the customer to decide on things that may not be important, but that may impact cost substantially. It requires back and forth to get the specification written, reviewed, amended etc.

On a small project, a developer can understand the purpose, build something quickly and adjust it based on feedback. Is the reality of larger projects that this will just remain a pipe dream? If so, why? If not, how do you get the quick and informal feedback?

What is the goal of your current project? I currently work on a project for the transmission system operator for the Norwegian electrical grid. The value of the system we’re building is that my beer stays cool.

Skill

If you’re not skilled at what you’re doing, you may put in a lot of hours and end up having nothing to show for it. Maybe you ended up writing code that didn’t compile and you had to throw it away. Maybe you ended up staring at an empty screen for hours waiting for inspiration to strike. Maybe you ended up writing a long document, only to forget to save it and then your computer crashed.

The first trick to making sure that your work is worthwhile is to make sure that the effort you put in actually transforms to some effect. In order to achieve this, you need experience and practice with your craft.

Purpose

If you don’t know why you’re doing what you’re doing, you may produce a lot, but it ends up not being valuable to anyone. Maybe the code you write solved the wrong problem. Maybe your long document didn’t answer the questions of you readers.

The second trick to making sure that your work is worthwhile is to make sure that the effect you have actually produces value. In order to achieve this, you need to move in the right direction.

This is the model of all human industry, whether you’re making cars, books, food, presentations or software.

In software development, effort is the time you put in, skill is your skill as a developer, effect is the working software, purpose is the architecture and value is the business goal.

The fast track to success is to understand the last step: Value.

“The other me”

Here’s an example. Find the value:

My current project creates a system which presents in user interface electricity reserves and planned electricity production that it has received from a message gateway, as well as measurements that it has received from an electricity management system.

This is gobbledygook. If this is how I see the system, I will fail.

In order to see the value, we have to take a step back. The system receives planned production and available reserve capacity from power producers. It presents this, along with measurements of the current balance between consumption and production on the electricity grid to an operator. The operator uses the reserves to regulate production to match consumption.

Hopefully, this is much more understandable. You’re probably aware that there are companies that produce electrical power, and you probably understand that electricity consumption should match production.

But why?

The ultimate value of the system is to the consumer who want to plug his appliances into the grid and have them work. They should neither blow up (too much production) or lose power (too little production).

Who is this mysterious “consumer”, then. Why, it’s me! It’s not Johannes-the-software-developer, it’s the other me. The “me” who has his fridge connected to the power grid so I can keep my beer cool.

The purpose of my project is to keep my beer cool.

The other you

To understand the purpose of your system, you should try and see how it contributes to your own life. Not the life of the “you” who develops software, but the “you” who uses electricity, the “you” who drives a car, the “you” who breathes fresh air, the “you” who wants to retire at some point in the future.

If you can’t see “the other you” in your system – take a step back and look again. If you still can’t see “the other you”, maybe you should be working on a different project.

Project development lifecycles

• Agile: The project creates a roadmap for a year or two, commits to the scope a delivery in a few weeks to a few months, adapts their commitment as they learn more and irons out the details of each task just prior to when it’s performed.
• Incremental: The project commits to the scope the whole roadmap at the beginning of the project and delivers bits every few months according to this plan. The details of a task are ironed a few months ahead of time. Changes to the full project scope are handled as exceptions.
• Waterfall: The project commits to the whole scope of the roadmap at the beginning of the project and postpones delivery of all functionality until the end of the project. The project avoids spending time to verify details after any implementation has started. Learning is handled as exceptions.

Agile described in terms of waterfall (“do a full project every few weeks”) sounds chaotic. Waterfall explained in terms of agile sounds insane.

We practice pair programming to get redundancy. This manifests itself in many ways:

• When someone is sick, on vacation or leaves the project (to become a doctor, say), the project doesn’t suffer. Or rather, it doesn’t suffer as much. We do get less work done if fewer people are present, of course. And many of our developers have areas where everyone loves to get their help.
• We can add new people to the team. Learning how the system works is just a matter of pair programming as part of the normal rotation.
• We are very adaptive to varying focus at different times in the project. If we had someone who could only work with, say, databases, he wouldn’t have much to do at some times, and too much to do at other times. With pair programming, we still have people who’re especially good in some areas, but everyone can help out with whatever needs doing.
• No parts of the code are off-limits. As every commit of the code is written by two people, no one person has the right to block changes to “his” code. Anyone can make the changes that they need to complete their current task as good as they can.
• The team is working with fewer tasks at a time. As teams grow in size, there can be a lot of different things happening at the same time. This both increases the chances that people will step on each other’s toes (that is, work on the same source code files) and increases the amount of “Work in progress” at any one time. Pair programming effectively halves the amount of work in progress.
• Everyone has a understanding of the whole system. During planning and design discussions, everyone can contribute. Instead of focusing on technical components and subsystems, the discussion is about how to solve the user story as a whole.

Some studies focuses on the impact of pair programming on efficiency, quality, progress, or even focus. I think it’s much more interesting to see how pair programming changes the dynamic of the team. From a group of individuals, we form teams where everyone is appreciated as a contributor. Everyone has their individual strengths, but more importantly, everyone has the same goal in mind: The success of the team.

Sadly, the practice of software engineering research has not come very far in this area, so the questioner is left unsatisfied. In this blogpost, I will summarize the current state of research into pair programming, and describe my hopes for the future.

Lately, I’ve been talking with researchers at SINTEF about project in-situ studies into different aspects of agile development, including pair programming. As we have several teams that practice pair programming on a daily basis in Steria, I’m hoping that this may finally shine some scientific light on what I believe to be a really good practice.

The current situation is not satisfactory. The body of research into Pair Programming is summarized in the meta-study Are Two Heads Better than One? On the effectiveness of pair programming by Dybå, Arisholm, Sjøberg, Hannay, and Shull in IEEE Software November, 2007.

The meta-analysis looks at twelve papers on pair programming in their assessment of pair programming on three project metrics: Quality, Duration and Effort.

The studies are mostly pointing to the same results: Pair programming seems to improve quality significantly, decrease duration slightly, and increase effort slightly. That is: Pair programming will make your project ship a little earlier, with significantly better quality, but at a higher cost.

Sadly, all the research so far miss some significant factors:

• First: Better quality in the short term may transform into lower cost in the long term. As the studies are all short term, this effect cannot be assessed.
• Second: One of the main reasons we do pair programming is to decrease the effect of absence (vacations, illness, turnover) and suboptimization due to over-specialization. As far as I know, there are no studies that include these effects.
• Third: I’ve found that pair programming is a skill in itself. It takes up to a few months to learn it. All studies of pair programming have been with people who are inexperienced in the practice.

Finally, studies on pair programming seem to conclude that pair programming may be appropriate for juniors or mid-level developers, but that it seldom makes sense for senior developers. I don’t know if I should take that as an insult or not, as I practice pair programming in almost all my programming time. And I would like to consider myself a senior developer. Feel free to disagree.

Good: In order to really get the benefit from refactoring, I think we have to do it all the time. I resolve to always leave a code file in a state where I could be satisfied if I never got to improve it in the future. This works best on the micro-level: Before I jump to a different file, I check the naming, method lengths and formatting of my code. When done well, making the code better is part of writing the code in the first place.

Sad: When I forget to follow this practice, or follow it poorly, I end up with codes in need of constant cleaning up. Creating new functionality feels like a drag, because there’s a lot of old junk I want to clean up before going on. It’s a sad day when I find myself cleaning up code that I really shouldn’t have to touch.

Ugly: But if I skirt the sadness of improving code that I come across on my journeys, after a while, I’ll have a mess on my hands. If I’m lucky, I can fix this with a few checkins. Some projects seem to close down for weeks to “deal with technical debt”. In the worst situations, the code will be in a non-functioning state in the meantime. Or you may hear those dreaded words: “Rewrite from scratch”.

If there’s one think I’ve learned, it’s to never leave the code in a non-functioning state. Once I depart from what I know work, getting back on track is many times harder than I ever imagined.

When a project schedules “refactoring tasks”, I find myself sad. The need for scheduled refactorings is a symptom that we don’t do the necessary good or even sad refactorings as we should. And scheduling refactoring tasks is an excuse to perpetuate our bad habits.

I often find myself in a situation where I have a method call that’s starting to accumulate parameters. Something like this:

showPersonCreateForm(writer, firstName, firstNameErrorMessage, lastName, lastNameErrorMessage,....);

After three or four parameters, the need to refactor is starting to become evident. I would rather have something like this:

CreatePersonForm form = new CreatePersonForm();
form.setFirstName(firstName);
form.setFirstNameErrorMessage(firstName);
form.setLastName(firstName);
form.setLastNameErrorMessage(firstName);
form.show(writer);

This is one of the more complex simple refactorings you can make, and it requires several steps. In this five minute video, I show how to perform such a refactoring without any steps that break my code:

The screencast was created using the free BB FlashBack Express on Windows. All the magic you see happening while I program is either ctrl-space (complete) or ctrl-1 (quick fix).

Can you modify your code without going thought long stages of nothing working? I think you can!