Thinking Inside a Bigger Box

I hate giving promises for things I can’t control. I can promise that I will attend a party or that I will set aside time to help you with your problem. I cannot promise that the party will be fun or that your problem will be solved. Giving promises on effort is honest, giving promises on outcomes is dishonest.

A team that commits to an estimate is promising something they cannot control. A team that is blamed for giving an estimate that is too low can easily avoid that particular mistake next time around. And given the law that work expands to fill available time, the estimate will never be too high. The result is cost spiraling up.

What if we assumed that estimates would never be particularly reliable. And if we assumed that forcing a team to commit to an estimate is unreasonable behavior? How would we act in a world where that’s true?

Let’s imagine we live in a world where the product owner cannot ask for an estimate. What can the product owner do instead?

The product owner can say “stop working on this story if you’ve spend more than 40 hours. Or if you think you will end up spending more than 40 hours.”

Such a limit can be viewed as a budget.

The product owner can make a bet on how many user stories the team will complete by looking at what they have done before. If the consequence of the betting wrong are severe, the product owner can be cautious about the number of user stories. (Bonus question: Is the team is a better or worse position to know the consequences of betting wrong?)

Such a bet can be viewed as a forecast.

The team, on the other hand can commit to working according to the agreed-upon rules. They can commit to do things in the order set by the product owner. They can commit to doing the best job they can within the time budget the product owner has allocated.

We commit to effort, not to outcomes.

As a user, I can add social security number, so patient logs have social security numbers

As a developer, how would you react if you were given this user story? Would you throw it back in the face of the product owner, or would you try and understand it?

How about the following dialogue?

• Developer: “What are we hoping to achieve with this story?”
• Customer: “We hope that the patient logs will have social security numbers. Duh.”
• Developer: “Sorry, I was unclear: What problem do you experience now that we don’t have the social security number?”
• Customer: “Oh! We need the social security number when we bill the customer.”
• Developer: “I see. So what happens now?”
• Customer: “Well, since the patient has left the hospital, the billing department has to phone or send postal mail to get the information. That’s a lot of work.”
• Developer: “What places in the application can we add this information?”
• Customer: “We’ll add it to the patient journal.”
• Developer: “Who updates the patient journal, and when?”
• Customer: “Whops. The doctor updated the journal after the patient has left. We better add it when secretary checks the patient out. So that would be in the appointment system”
• Developer: “What other places could we enter this information?”
• Customer: “Well, we could have a field for the social security number when the patient first requests the appointment”
• Developer: “So, we have considered two options: The appointment system at checkout or the appointment system when the appointment is booked. What other ways could we get the same information?”
• Customer: “We could collect it from the governmental web service, I suppose.”
• Developer: “What would you like us to try first?”
• Customer: “Since the patient normally fills in the appointment request themselves, let’s try that out first.”
• Developer: “Let’s see if I get you correctly….”

In order to have the required information to send an invoice, as the billing department, I want the patient to enter their social security number when they request an appointment

What you’ve just witnessed is a series of questions that can be organized like this:

• Goal: What are we hoping to achieve? How will the user story change the world?
• Reality: How are things working now?
• Options: What else could we do instead? In addition?
• Will: What will we do? What is our plan of action

The GROW mnemonic is a tool from professional coaching. That is: Talking to people about the problems and ambitions in their professional and private lives. It helps guide a novice coach through a coaching conversation and focus on the problems of the person being coached. Or, in this case: On the business value we want to achieve.

At the end of the day, as a developer you will be judged not on whether you followed the orders you were given, but whether you understood and delivered what was really needed. By asking your users, your product owner or your domain expert what they really, really want (Goal), it will be easier for you achieve. And if you don’t have access to the relevant people, these questions can still help you guide your thinking about the requirements.

You can read more about the GROW model at What-is-coaching.com and many other websites devoted to helping people get to the best of their ability.

A big thank you goes to Antti Kirjavainen for suggesting the patient journal example and for coaching me in the process of writing this article.

Do you know how to apply coding practices the technology stack that you use on a daily basis? Do you know how the technology stack works? For many programmers, it’s easy enough to use test-driven development with a trivial example, but it can be very hard to know how to apply it to the problems you face every day in your job.

Java web+database applications are usually filled to the brim with technologies. Many of these are hard to test and many of these may not add value. In order to explore TDD and Java applications, I practiced the Java EE Spike Kata in 2010. Here’s a video of me and Anders Karlsen doing this kata at JavaZone 2010.

A similar approach is likely useful for programmers using any technology. Therefore, I give you: The rules of the Architecture Spike Kata.

The problem

Create a web application that lets users register a Person with names and search for people. The Person objects should be saved in a data store that is similar to the technology you use daily (probably a relational database). The goal is to get a spike working as quickly as possible, so in the first iteration, the Person entity should probably only contain one field. You can add more fields and refactor the application later.

The rules

The most important rules are Robert Martin‘s three rules of Test-driven development:

• No code without test (that is, the code should never do something that isn’t required in order to get a test to pass)
• Only enough test to get to red (that is, the tests should run, give an error message and that error message should correct)
• Only enough code to get to green (that is, the tests should run and not give an error)
• (My addition: Refactor on green without adding functionality)

Secondly, application should be driven from the outside in. That is, your first test should be a top-level acceptance test that tests through http and html. It’s okay to comment out or @Ignore this test after it has run red for the first time.

Lastly, you should not introduce any technology before the pain of not doing so is blinding. The first time you do the kata in a language, don’t use a web framework beyond the language minimum (in Java, this means Servlets, in node.js it’s require('http'), in Ruby it means Rack). Don’t use a Object-Relational Mapping framework. Don’t use a dependency injection framework. Most definitely don’t use an application generator like Rails scaffold, Spring Roo or Lift. These frameworks can be real time savers, but this kata is about understanding how the underlying technology works.

As a second iteration, use the technologies you use on a daily basis, but this time set up from scratch. For example, if your project uses Hibernate, try configuring the session factory by hand. By using frameworks in simplest way possible, you’ll both learn more about what they bring to the table and how to use them properly. For complex technology like Hibernate, there’s no substitute for deeper understanding.

What to expect

So far, I’ve only done the Architecture Spike Kata in Java. But on the other hand, I’ve done it around 50 times together with more than ten other developers. I’ve written about how to get started with the Java EE Spike Kata (in Norwegian) on my blog before.

This is what I’ve learned about working with web applications in Java:

• Most Java web frameworks seem to harm more than they help
• Hibernate is a bitch to set up, but once it’s working, it saves a lot of hassle
• Using TDD with Hibernate helped me understand how to use Hibernate more effectively
• I’ve stopped using dependency injection frameworks (but kept on using dependency injection as a pattern)
• I have learned several ways to test web applications and database access independently and integrated
• I no longer have to expend mental energy to write tests for full stack application

The first time I write this kata with another developer, it takes around 3 to 5 hours, depending on the experience level of my pair. After running through it a few times, most developers can complete the task in less than an hour.

We get better through practice, and the Architecture Spike Kata is a way to practice TDD with the technologies that you use daily and get a better understanding of what’s going on.

Instead of using Scrum, maybe we should use MyScrum. It’s like Scrum, with the stuff added that you think will super charge your MyScrum.

This is my MyScrum:

• I want to measure velocity every week
• I want to demostrate the product with a cadence that makes sure users show up to the demo
• I don’t want to have story point estimates, I want to have story point budgets
• I want the product owner, not the team, to own the budget/estimate (but they team may veto)
• I don’t want commitments or forecasts from the team, I want measured historical progress
• I want to plan per story, not per sprint
• I want the developers who will develop a story to follow it (or pass the baton to other developers) from detailing to deployment.

What does your MyScrum look like?

On larger projects, I’ve always ended up resorting to writing down a lot of detailed specifications, many of which are wrong, irrelevant or we might not be ready to answer them yet. On small projects, the dialogue between the customer and the developers can flow easy, and good things happen.

The quick analysis

Developer: … so we’re going to complete the current task tomorrow or the day after. Could we discuss what to do next before you’re off to your next meeting?

Customer: Sure. The next task on the backlog is to send the payment request to the accounting system

Developer: Yeah. Um… well, we’re not really ready for the integration tasks yet. I was thinking that we might perhaps pick another user interface task. Would that be okay?

Customer: No problem. Let’s see…. how about “road project submits funding request”?

Developer: Great. What’s that?

Customer: Well, today, the owner of the road project sends in a paper form. Let’s see, the form is available as a PDF on our web site. Let’s take a look.

Developer: Let’s see…. We know how to deal with social security number, names, addresses. This field “municipality,” that’s the usual stuff, right? (customer nods) What about “request id”? What’s that?

Customer: It’s just a sequence number per municipality. It would be really nice if you could suggest a request number and then let the user override.

Developer: That should be doable.

They work out a few more details and get on their way. Total time, 30 minutes

The low ambition implementation

Developer: So, we worked on the funding request UI for the last two days. I think we’ve got something that looks pretty good.

Customer: Great, let’s see it! Let’s see… There’s a new “apply for funding request” menu item. Nice. But it should be “Fund road project” (developer notes feedback). Let me try and press it… Hmm… “select municipality”? Why do I get that?

Developer: Well, we’re still a bit shaky on AJAX. So in order to find the next sequence number, we would like to have the user select municipality first.

Customer: Oh, AJAX. That’s interactive web pages, right? Yeah, I guess that’s fine, but we need to fix up the text a little. So, what happens when I select one…? Nice, it sort of looks like the paper form. That will help us get learn it. But let’s move the fields around a little (he sketches a few change on a piece of paper).

Developer: We were a bit uncertain about this “sum” field. Does the user have to fill in the sum himself? He’s already filled in all the numbers.

Customer: That doesn’t make too much sense. How about if the web pages updates the sum field dynamically? Oh, you’ve got that ‘don’t make me do dynamic webpages’ face again.

Developer: Yeah, sorry. The last time it was a big mess. Can we hold off on that for now? Do you really need it?

Customer: (laughs) No, that’s okay. I guess we can just remove the field. That should be fine.

Developer: Yay! What about validating the bank account number? We left that out, because the documentation is…. well, insane. That alone will take at least another day.

Customer: No, we really need that. People often type it wrong, and then it’s a nightmare to fix all the errors that follow. You just have to fight though it.

Developer: (sigh) I guess it can’t be helped. We’re on it.

Customer: All in all, it looks pretty good. I’ll go over it by myself after lunch and bring you back some issues tomorrow.

The quick completion

Customer: Here’s a list of corrections. Nothing major, just a few misplaced fields and a few extra validation rules.

Developer: Great. We spent all of yesterday doing the bank account number validation. Man, that’s some bad doc! But we should be done before lunch.

Customer: I have an opening in my calendar at the end of the day, let’s take a look then and call it done. Great work, guys!

The contrast

Have you ever been on a small project where everyone has a pretty good understand of what’s going on and the communication works well? Developers can make a pretty good guess at the details and develop a pretty good first version. There may be some more work that’s needed, but these can be negotiated when the developers have a better idea of what’s going to be easy and what’s going to be hard.

When I’ve been on larger projects, this communication seems to break down. Instead the developers require a detailed specification in writing. And if something is missing from the specification, they reject it and ask for further details. And they won’t show the product to the customer before all the kinks are worked out.

When we do it well it’s not so bad, getting a detailed specification and finishing the task before we get real feedback. But it requires the customer to decide on things that may not be important, but that may impact cost substantially. It requires back and forth to get the specification written, reviewed, amended etc.

On a small project, a developer can understand the purpose, build something quickly and adjust it based on feedback. Is the reality of larger projects that this will just remain a pipe dream? If so, why? If not, how do you get the quick and informal feedback?

This blogpost is a summary of my ScanDev 2011 talk: “Fearless Improvement”

What is the goal of your current project? I currently work on a project for the transmission system operator for the Norwegian electrical grid. The value of the system we’re building is that my beer stays cool.

Skill

If you’re not skilled at what you’re doing, you may put in a lot of hours and end up having nothing to show for it. Maybe you ended up writing code that didn’t compile and you had to throw it away. Maybe you ended up staring at an empty screen for hours waiting for inspiration to strike. Maybe you ended up writing a long document, only to forget to save it and then your computer crashed.

The first trick to making sure that your work is worthwhile is to make sure that the effort you put in actually transforms to some effect. In order to achieve this, you need experience and practice with your craft.

Purpose

If you don’t know why you’re doing what you’re doing, you may produce a lot, but it ends up not being valuable to anyone. Maybe the code you write solved the wrong problem. Maybe your long document didn’t answer the questions of you readers.

The second trick to making sure that your work is worthwhile is to make sure that the effect you have actually produces value. In order to achieve this, you need to move in the right direction.

This is the model of all human industry, whether you’re making cars, books, food, presentations or software.

In software development, effort is the time you put in, skill is your skill as a developer, effect is the working software, purpose is the architecture and value is the business goal.

The fast track to success is to understand the last step: Value.

“The other me”

Here’s an example. Find the value:

My current project creates a system which presents in user interface electricity reserves and planned electricity production that it has received from a message gateway, as well as measurements that it has received from an electricity management system.

This is gobbledygook. If this is how I see the system, I will fail.

In order to see the value, we have to take a step back. The system receives planned production and available reserve capacity from power producers. It presents this, along with measurements of the current balance between consumption and production on the electricity grid to an operator. The operator uses the reserves to regulate production to match consumption.

Hopefully, this is much more understandable. You’re probably aware that there are companies that produce electrical power, and you probably understand that electricity consumption should match production.

But why?

The ultimate value of the system is to the consumer who want to plug his appliances into the grid and have them work. They should neither blow up (too much production) or lose power (too little production).

Who is this mysterious “consumer”, then. Why, it’s me! It’s not Johannes-the-software-developer, it’s the other me. The “me” who has his fridge connected to the power grid so I can keep my beer cool.

The purpose of my project is to keep my beer cool.

The other you

To understand the purpose of your system, you should try and see how it contributes to your own life. Not the life of the “you” who develops software, but the “you” who uses electricity, the “you” who drives a car, the “you” who breathes fresh air, the “you” who wants to retire at some point in the future.

If you can’t see “the other you” in your system – take a step back and look again. If you still can’t see “the other you”, maybe you should be working on a different project.

I’m getting a little fed up with descriptions of project development lifecycles starting with waterfall, and then describing iterative and agile development in waterfall terms. What happens if we start on the other side?

Project development lifecycles

• Agile: The project creates a roadmap for a year or two, commits to the scope a delivery in a few weeks to a few months, adapts their commitment as they learn more and irons out the details of each task just prior to when it’s performed.
• Incremental: The project commits to the scope the whole roadmap at the beginning of the project and delivers bits every few months according to this plan. The details of a task are ironed a few months ahead of time. Changes to the full project scope are handled as exceptions.
• Waterfall: The project commits to the whole scope of the roadmap at the beginning of the project and postpones delivery of all functionality until the end of the project. The project avoids spending time to verify details after any implementation has started. Learning is handled as exceptions.

Agile described in terms of waterfall (“do a full project every few weeks”) sounds chaotic. Waterfall explained in terms of agile sounds insane.

What do you do if you want to have a reliable system? You make sure you have redundancy: More than one component can do a certain job. You back up you valuable data to a separate system. You have two servers to provide your critical service, in case on fails. Yet, many projects have no plan for the inevitable absence of people with critical knowledge.

We practice pair programming to get redundancy. This manifests itself in many ways:

• When someone is sick, on vacation or leaves the project (to become a doctor, say), the project doesn’t suffer. Or rather, it doesn’t suffer as much. We do get less work done if fewer people are present, of course. And many of our developers have areas where everyone loves to get their help.
• We can add new people to the team. Learning how the system works is just a matter of pair programming as part of the normal rotation.
• We are very adaptive to varying focus at different times in the project. If we had someone who could only work with, say, databases, he wouldn’t have much to do at some times, and too much to do at other times. With pair programming, we still have people who’re especially good in some areas, but everyone can help out with whatever needs doing.
• No parts of the code are off-limits. As every commit of the code is written by two people, no one person has the right to block changes to “his” code. Anyone can make the changes that they need to complete their current task as good as they can.
• The team is working with fewer tasks at a time. As teams grow in size, there can be a lot of different things happening at the same time. This both increases the chances that people will step on each other’s toes (that is, work on the same source code files) and increases the amount of “Work in progress” at any one time. Pair programming effectively halves the amount of work in progress.
• Everyone has a understanding of the whole system. During planning and design discussions, everyone can contribute. Instead of focusing on technical components and subsystems, the discussion is about how to solve the user story as a whole.

Some studies focuses on the impact of pair programming on efficiency, quality, progress, or even focus. I think it’s much more interesting to see how pair programming changes the dynamic of the team. From a group of individuals, we form teams where everyone is appreciated as a contributor. Everyone has their individual strengths, but more importantly, everyone has the same goal in mind: The success of the team.

When I started pair programming daily, it changed my life and my projects for the better. I often point to pair programming as one of the most influential possible interventions on any projects. Very often, an audience member with a healthy degree skepticism will inquire about the research into pair programming.

Sadly, the practice of software engineering research has not come very far in this area, so the questioner is left unsatisfied. In this blogpost, I will summarize the current state of research into pair programming, and describe my hopes for the future.

Read the rest of this entry »

“Refactoring” is the practice of “improving the design of existing code without changing its behavior”. It’s an essential part of software maintenance. Done well, refactoring will make sure your code base is easy to maintain. Done poorly, refactoring will lead you into a dangerous swamp where you’ll be stuck forever.

Read the rest of this entry »