Thinking Inside a Bigger Box

When picking iteration length for an agile project, there are mainly two forces that you have to balance: The rate of learning is proportional with the number of iterations, rather than the length of the project. This means that shorter iterations help you get better faster. But each iteration has some overhead with sprint reviews, retrospectives and planning. You don’t want this overhead to dominate the effort spent on the project.

For some reason, most projects I’ve seen with little experience in iterative development prefer three week iterations. Personally, I prefer two week iterations. Here is the breakdown:

• Three week iterations: After three months, you’ve spent about 7% of your time on iteration meetings. You’ve had 4 opportunities to improve.
• Two week iterations: After three months, you’ve spent about 10% of your time on iteration meetings. You’ve had 6 opportunities to improve.
• One week iterations: After three months, you’ve spent about 20% of your time on iteration meetings. You’ve had 12 opportunities to improve.

Going from 93% to 90% efficiency for a 50% increase in learning seems like a good deal. Going from 90% to 80% efficiency for a 100% increase in learning, not so much.

These numbers are of course greatly simplified. You might also consider:

• With shorter iterations, the planning time may go down. But this takes practice – it doesn’t happen automatically.
• With very short iterations, you may not have experienced enough to learn much from the retrospective. However, if you find that you do a timeline, and most of the things people remember happened the last week, it may not be because that’s the only time something significant happened.
• You may consider different frequencies for different ceremonies. For example, on my current project we want to have demos with our power users. But they have to travel far to visit us. So we only have a full demo every other four weeks. We plan every two weeks and have an internal review and retrospective every two weeks.

What’s the right iteration length for your project?

As it turns out, one of the least used practices of agile development is also one of the most powerful.

Up into the start of last year, I only worked sporadically with pair programming. Last year, I was lucky enough to be part of a team that used pair programming all the time. Since I’ve experienced real pair programming, I never want to give it up.

Pair programming offers benefits to many stakeholders:

• As a developer, you will have more fun at work. You will get to know your colleagues better and experience flow practically the whole day. You will be tired by the end of the day, but you will also feel like you’ve accomplished good work.
• The team will have a higher quality code base that everyone is comfortable with.
• As an architect or team lead, you will have a good way to contribute even if you only have a little time before a meeting. You will also have a better chance to influence the rest of the team, instead of just issuing edicts that nobody follows.
• As the project manager, you will have a more flexible team. If someone gets sick, goes on vacation or moves to another project, there won’t be a big problem.
• As the customer, you will get better quality code faster.

With these benefits in mind, why doesn’t everybody pair program? Well, it is unfamiliar, a little scary, and exhausting when you start out. Most developers are not used to having other watch them code. Or to focus on the task at hand the whole day.

Here are some techniques I’ve seen have effect for teams transitioning to pair programming:

• Code dojos: Everyone on the team gets together and programs a sample program or a spike together. Two people sit at the keyboard, while the rest watch on a projector. Rotate pairs frequently. This lets everyone get comfortable with coding as a social activity.
• Pair programming should be the norm, but allow for exceptions. If people only pair program occasionally, they end up not pair programming at all. If people are forced to pair program when they just need some time by themselves to think, they will not be happy pair programming.
• The pair programming star: Write the names of the team members in a circle. Every time two people pair program, draw a line between their names. Keep the pair programming star in a visible location.
• Facilities: The furniture can make it harder to get started pair programming. Consider using two mice, two keyboards and perhaps two monitors per PC to make it easier. Or use VNC for desktop sharing.
• Give it time: Pair programming is exhausting when you first start doing it. It will take a while before people are comfortable with the new pace. But once they switch, they will never want to go back.

Resources

For more inspiration, see these presentations from the Smidig 2009 conference (in Norwegian):

• Jøran Lillesand: Derre e itj smidi!: On why pair programming is the very foundation of successful agile projects
• Ørjan Lillevik og Kari Røssland: Parprogrammering gir driv: On what it feels like to be on a team that adopts pair programming. From reluctance to joy.

After seeing some of the great examples of coders working on practiced problems on KataCasts, I decided to try make my own. I am not happy with the pacing of the video. I’m about a minute too early relative to the music.

But I thought I’d post the video here, to see what you all think. Comments are welcome!

I hope the video will demonstrate how to use refactoring effectively to drive the design of a program.

I chose the FizzBuzz kata – that is, to generate a sequence of numbers where every number divisible by three is replaced by “fizz” and every number divisible by five is replaced by “five”. The music changes to be more aggressive just as I induce a new requirement into the kata: The FizzBuzz generator should be programmable, so, in the kata, numbers divisible by two are replaced by “coconut” and numbers divisible by seven are replaced by “banana”.

Thanks to Emily Bache for the inspiration for the kata.

Enjoy!

Fizz buzz code kata from Johannes Brodwall on Vimeo.

The video was made with IntelliJ IDEA Community Edition on Windows Vista (!) with BB FlashBack Express (free screen recorder), converted to AVI with Windows Media 1 codec and uploaded to Vimeo.

For å bli bedre må man trene. For å bli bedre med avanserte ting, må man forstå de grunnleggende tingene bra. For å vite hvorfor man bruker avanserte verktøy, må man prøve å jobbe uten dem. Derfor har jeg de siste ukene trent mange ganger på å lage en veldig enkel webapplikasjon i Java. For hele applikasjonen har jeg startet med å skrive testene før koden som implementerer funksjonaliteten.

Dersom du vil prøve deg på samme øvelse, inneholder denne artikkelen litt informasjon for å komme i gang. Start med koden under og følg feilmeldingene. Send en kommentar dersom du ikke kommer videre fra en feilmelding, så får vi en FAQ.

Oppgaven

Løs et så enkelt som mulig problem som involverer websider og database med så enkel teknologi om mulig.

Oppgaven jeg har laget går ut på å opprette personer med fullt navn og søke etter personer basert på navnet deres. For å gjøre oppgaven så lite som mulig har jeg valgt å la personer kun ha ett informasjonsfelt: Fullt navn. Denne oppgaven tar cirka 2-3 timer uten øvelse og du kan få den ned i 60-90 minutter med trening.

Du kan naturligvis velge en annen oppgave, men uansett hva du velger: Det er mer lærerikt å gjenta den samme oppgaven flere ganger enn å utføre en avansert oppgave.

Når jeg utfører oppgaven er det viktigste jeg lærer meg å forstå feilmeldingene som guider meg gjennom utviklingen. Dersom du trenger hjelp til å komme til de første feilmeldingene kan du se resten av artikkelen.

Steg for steg: Startpunktet

Selv om jeg valgte veldig enkel teknologi for implementasjonen, har jeg valgt et større sett med biblioteker for å skrive testene. Jeg bruker følgende når jeg skriver testene:

• JUnit 4.6
• Jetty 6.1.22
• HSqlDb 1.8.0.10
• WebDriver-HtmlUnit 0.6.1039
• Mockito 1.8.0
• FEST-assert 1.2 (ikke påkrevd, men gjør testene søtere)

Den eneste teknologien jeg har valgt for implementasjonen er Servlet-API 2.5 og Hibernate-Annotations 3.4.0.GA.

For at du skal slippe å plundre så mye med avhengigheter før du kommer i gang har jeg laget en pom.xml-fil som du kan ta utgangspunkt i.

Web-tester

For å starte utviklingen, er det lurt med en test som starter på utsiden av applikasjonen. Noe slikt:

1. Start opp miljøet
2. Legg inn en person
3. Søk etter personen

Slik kommer du i gang med en test som går mot en web applikasjon:

int SERVER_PICKS_PORT = 0;
org.mortbay.jetty.Server server = 
       new org.mortbay.jetty.Server(SERVER_PICKS_PORT);
server.addHandler(
       new org.mortbay.jetty.webapp.WebAppContext("src/main/webapp", "/"));
server.start();
 
int serverPort = server.getConnectors()[0].getLocalPort();
 
org.openqa.selenium.WebDriver browser =
       new org.openqa.selenium.htmlunit.HtmlUnitDriver();
browser.get("http://localhost:" + serverPort + "/");
browser.findElement(By.linkText("Create person"));

Dette oppsettet forventer å finne web.xml-fila på src/main/webapp/WEB-INF/web.xml.

Funksjonell test

En funksjonell test definerer kravene i applikasjonen. Det er lurt å gjøre funksjonelle tester så raske som overhode mulig, samtidig som de går gjennom alle kravene. En funksjonell test trenger ikke være en ende-til-ende test, slik som eksempelet over. Dette er viktig, fordi ende-til-ende tester er ofte veldig trege. Her er noen eksempler på funksjonelle tester:

• Vis en siden for å opprette nye personer
• Opprett en ny person
• Verifiser at personens navn er oppgitt og ikke inneholder ulovlige tegn
• Vis en side for å søke etter personer
• Vis alle personer dersom søkestreng ikke er angitt
• Søk etter angitt søkestreng

En funksjonell test kan se slik ut:

PersonServlet servlet = new PersonServlet();
 
HttpServletRequest req =
    org.mockito.Mockito.mock(HttpServletRequest.class);
HttpServletResponse resp =
    org.mockito.Mockito.mock(HttpServletResponse.class);
 
PersonDao personDao =
    org.mockito.Mockito.mock(PersonDao.class);
servlet.setPersonDao(personDao);
 
org.mockito.Mockito.when(req.getMethod())
    .thenReturn("POST");
org.mockito.Mockito.when(req.getPathInfo())
    .thenReturn("/create.html");
org.mockito.Mockito.when(req.getParameter("full_name"))
    .thenReturn("Johannes Brodwall");
 
StringWriter pageSource = new StringWriter();
org.mockito.Mockito.when(resp.getWriter())
    .thenReturn(new PrintWriter(pageSource));
 
servlet.service(req, resp);
 
org.mockito.Mockito.verify(personDao)
    .create(Person.byName("Johannes Brodwall"));
 
org.fest.assertions.Assertions.assertThat(pageSource.toString())
    .contains("Personen er opprettet");

Data-aksess-test

Hibernate forenkler databasebruken mye. Men Hibernate er selv komplekst og når man bruker det på mer avanserte måter fortjener det egne tester. En typisk test med Hibernate kan være:

1. Legg i tre personer i database
2. Søk etter en del av navnet på en av dem
3. Sjekk at du får tilbake akkurat den du forventet

Når jeg starter med Hibernate, lager jeg en test som dette, og følger feilmeldingene. Pass på å både følge feilmeldinger i loggen og stack tracer.

AnnotationConfiguration conf = new AnnotationConfiguration()
    .setProperty(Environment.URL, "jdbc:hsqldb:mem:persondaotest");
PersonDao dao = new HibernatePersonDao(conf.buildSessionFactory());
 
dao.create(Person.withName("foo"));
 
org.fest.assertions.Assertions.assertThat(dao.find(null))
    .containsExactly(Person.withName("foo"));

Følg feilmeldingene herfra.

Integrasjon

En veldig vanlig måte for web serveren å overlevere spesielt ting som DataSources til applikasjonen er via JNDI. I Jetty kan du gjøre dette i Web-testen på følgende måte:

org.hsqldb.jdbc.jdbcDataSource ds = new org.hsqldb.jdbc.jdbcDataSource();
ds.setDatabase("jdbc:hsqldb:mem:personwebtest");
ds.setUser("sa");
new org.mortbay.jetty.plus.naming.Resource("jdbc/primaryDs", ds);
 
// Oppstart av Jetty som vist over

Konklusjon

Å gjøre en liten øvelse som dette er en god måte å bli bevisst hvilke vaner du har og hvor lang tid det egentlig tar for deg å gjøre oppgavene dine. Du vil oppleve at det å skrive tester før koden føles som om det går saktere enn du tror du er vant til.

Men dersom du er som meg, vil du også oppleve noe annet: Når du tester ut applikasjonen første gang (du kan gjøre dette med Jetty, naturligvis) så er sjansene gode for at den vil være nokså feilfri og at debugging i stor grad er overflødig. Jeg vet ikke med deg, men debugging er en aktivitet jeg gjerne blir kvitt.

Lately I’ve been thinking a lot about how easy it is to lose sight of the goal of the project and instead focus on whatever means someone first thought was a good starting point when the project was first conceived of. And I think it all comes down to words.

The first years I was working in this business, I didn’t see any distinction between “the user” and “the customer”. Once I started seeing the distinction, I started to understand that the person who is going to use the system we’re developing is not the person who defines what the system should do and neither of these is usually the person that pays me to develop the system. So I starting distinguishing between the product owner, that is, the customer and the end user. But the product owner often calls the person I call “end user” his “customer”. What’s going on here? Let’s check the dictionary:

CUSTOMER
Main Entry: cus·tom·er
Pronunciation: \ˈkəs-tə-mər\
Function: noun
1: one that purchases a commodity or service
2: an individual usually having some specified distinctive trait

CLIENT
Main Entry: cli·ent
Pronunciation: \ˈklī-ənt\
Function: noun
1: one that is under the protection of another : dependent
2a: a person who engages the professional advice or services of another
2b: customer
2c: a person served by or utilizing the services of a social agency
2d: a computer in a network that uses the services (as access to files or shared peripherals) provided by a server

I’ve seen suppliers approach their work by asking for a specification of a product to deliver and then trying to deliver something to that specification for payment. The mental model is that of a customer going to the grocery story asking for “eight pounds of CRM software”. My experience with organizations with this sort of mindset has always been unsatisfactory.

On the other hand, I’ve seen suppliers approach their work as an agent of the organization that pays them. “Our job is to enable someone else do their job better.” This totally changes the way an organization deals with this relationship. The word “customer” may not be conductive to this sort of thinking. Instead, we should think of ourselves as agents acting on behalf of a client. As an agent, your responsibility is to enable your client. This includes helping your client to find better means of reaching their goal.

By the way, wikipedia defines the word “agent” as “a person who is authorized to act on behalf of another (called the Principal or client) to create a legal relationship with a Third Party”. If the “third party” is the computer, then a good developer is an agent acting on their clients behalf in dealings with the computer software.

Why doesn’t the software industry use the word “client” instead of “customer”?

At ØreDev I ran into Lasse Koskela. We started talking about estimation techniques, and we both felt that the dominant estimation technique of relative estimation with planning poker has been unchallenged for a very long time. We found ourselves wondering what the next big idea about estimation will be. After throwing a couple of ideas back and forth, we decided to invite to a workshop comparing a few estimation techniques. We decided to call the workshop “The Malmö Experiment.”

The results of the experiment were interesting, but far from conclusive.

During the experiment, we gave the same set of requirements to three teams, each consisting of three estimators. Each team was told to use a different technique. We decided on the following techniques:

• Planning poker: The purpose is to give all requirements a relative number (the meaning of these numbers will later be measured based on the output of the iterations). Each estimator has a deck of cards and choose a card with the number he feels is appropriate for the current requirement. Everyone reveals the numbers at the same time to avoid anchoring. The team discusses and reestimates a requirement until their estimates converge.
• Table spread estimation: This is one of the new techniques we proposed. Each requirement is written on a card. The estimators spread the cards along a large table according to the relative effort required per requirement. Numbers can be imposed later if desired.
• Goldilocks estimation: The purpose is to restructure requirements until they all have roughly equal size. Instead of assigning a number to a requirement, the estimators pick one of three options: Too big (split up and estimate the parts again), Too small (merge with other requirements), or Just right. When all requirements have been split or merged into “Just Right” size, the estimation is complete.

All teams found their estimation techniques to be motivating, but the Table Spread and Goldilocks groups managed to complete the estimation much faster. The Table Spread estimation would obviously need more space if we had a lot of requirements, while the Goldilocks estimation would generate a large number of requirements.

Based on these experiences, we propose the following experiment in a project:

• Use Table Spread Estimation for Release planning. This will encourage the team to keep the number of requirements low instead of trying to plan too detailed too far ahead. Since the table spread is quick it can be redone every iteration.
• Use Goldilocks Estimation for the next few upcoming iterations to split up the requirements into equal sized items. This will generate a better set of work items. The shorter planning window will ensure that we won’t have an unmanageable number of requirements.

These are currently very rough ideas and we have no idea of whether it will work as we expect. Let me know if you have any relevant experience or if you want more information.

I’m working on a collection of patterns for early releases with Niklas Bjørnerstedt. Here are some of my thoughts based on this work.

In a few different projects, I’ve noticed that the idea of “where are we going” seems to go though a familiar pattern:

1. “The old system is the requirement document, just make the new one do the same things”. After a while, someone will realize that it’s rather pointless to replace a system with a new one that does the same thing, which leads to…
2. “Analyze the business processes and make the new system automate all decisions that a human used to make.” After a while, people start realizing that business rules are interpreted slightly different by different users and finding a consensus approach is hard. Besides, some of the decisions require human judgment. On top of this, progress towards implementing the business processes is much slower than expected. As a matter of fact, people are panicking as the project gets increasingly delayed, which leads to…
3. “Just do whatever the old system did, with whatever improvements are dead easy. Just get this damned thing out the door.” Even reducing the scope to just the “bare bones of the current system with minimal improvements” doesn’t seem to give sufficient progress. Or sufficient value to justify the project. So, finally, we arrive at…
4. “Can we just add a new piece of software that makes an existing business process easier. And repeat until the budget is spent.”

The sad conclusion is that the original goal of replacing the old system begins to appear further into the future. At the same time, the new system will realize some value to some stakeholder pretty soon thereafter. Maybe the first step towards a successful replacement project is giving up replacing the old system?

The good news is that with an iterative approach to the requirement process, my current project was able to go through all these steps in a couple of months. Which beats my previous record of a year of full burn rate in stage 1.

This Norwegian language article introduces a short two-page guide I’ve written to explain Scrum to people who’ve only just heard of it.

I samarbeid med våre dyktige redaksjonelle medarbeidere på Steria, har jeg forfattet en “3 minutters guide” til Scrum. Denne tar for seg spørsmålene som “hva er egentlig Scrum”.

Dette er Scrum

3-minutterguidene kan lastes ned fra Sterias hjemmesider.

Jeg planlegger å følge opp denne guiden med en guide som beskriver hva som skal til for å faktisk lykkes med Scrum. Har du noen ideer?

Every Scrum-team should use their taskboard to support their particular way of working. I’d like to share the way we use our taskboard at my current project for your inspiration.

Colored ink, paper and makers support team process

When I started my current project I went to pick up sticky notes and marker pens for the taskboard. I grabbed, more or less at random three colors of notes (red, green, yellow), four colors of pens (black, red, blue, green) and five color sticky bookmarks (yellow, green, blue, orange, red). Over the first weeks of the project, we evolved a system that uses combinations of all the colors:

• For tasks that we plan at the start of an iteration, we use the black pen. Tasks that contribute directly to the project goals go on green notes, meetings go on yellow notes and administrative tasks go on red notes. (I’m still tweaking this)
• For tasks that we discover during the iteration, we use the red pen. The color codes for the tasks are the same.
• We don’t write names on tasks. Instead, each of the three team members have three sticky bookmarks of a color (green, blue, orange) that they can place on stuff they work on. We limit the number of bookmarks per person to limit multitasking.
• We don’t have a column for tasks that are to be verified. Instead, we use the yellow sticky bookmarks to mark things that should be verified. The yellow sticky is addressed at latest at the next stand-up meeting. Like with concurrent work, we’ve limited the number of tasks to be verified to three. (So far we’ve never needed more than one.)
• We use the red sticky bookmarks for tasks that are blocked. That is, tasks where the team needs outside help. We’ve limited ourselves to three red bookmarks. When we’d like to place the fourth, we’d rather spend our time following up the current blockers.
• Since we had some spare area on our taskboard, we decided to use this to look for improvements to our process. We believe that improvements shouldn’t wait until the retrospectives. Instead, we want to think of how to get better all the time. We use the green pen for improvements. Impediments that cause extra work or defects go on red notes, things that would make us work better go on green, and neat ideas that would be fun to try go on yellow notes.
• Finally, we have an area for impediments. We haven’t come up with a system for this yet, so we just use arbitrary color notes and pens.

Markers support pair programming

We’re always looking for ways to improve the taskboard further. In particular, I’d like to use the blue pen for something. I’m also not totally happy with the division into different sort of tasks. Finally, I’d like to have a special sort of sticky for things that don’t really take time, but that we need to remember to do, like sending out meeting invitations.

How do you customize your taskboard to our team’s process?

What will the daily experience of software development look like, say, five years from now? Have our current processes reached their peak, or will the world continue to change? Alan Kay said “the easiest way to predict the future is to invent it.” Here are some ideas of the future I want to invent: I hope it will be dramatically better than what we currently do.

Steel pipes (by monkeyc.net)

The term “Continuous Integration” was first discussed when Extreme Programming was starting to garner interest in the late 90s. From then, it has gone from being a manual process that top-notch team used, to being an automated, nearly ubiquitous process. The tools have gone from being home made through demanding tools like CruiseControl to user friendly tools like Hudson. After Hudson, is there still any radical change in store for us?

Somewhat independent of the evolution of Continuous Integration tools, there have been four trends that have developed in the last few years:

• Continuous testing: Tools like autotest for Ruby and Kent Beck’s JUnit Max for Java execute your tests after every change you make to the source code. Autotest is widely used within the Rails community, and even though JUnit Max did not get the takeoff Kent was hoping for, I think there’s still great potential in this sphere. I’ve used both tools, and they transform the way I work for the better.
• Distributed source control greatly increases our flexibility in terms of multiple sources and stages of source code. Especially Git has seen growing interest in the last two years. Github is quickly becoming one of the large project hosting providers.
• Continuous deployment: Organizations have started pushing the result of their continuous integration process further towards production. In the last three years, I’ve worked on two large projects, both of which deploy every build to a test server. The company IMVU, with it’s large customer base, deploys automatically into production roughly 50 times per day.
• Smaller checkins: In the last issue of The Agile Toolkit George (no last name given in podcast or notes) suggest checking in every time your build is green. I’ve never worked on a project like that, but I’ve experienced a gradual increase in how frequently we check in.

Complexity (by nerovivo)

If we extrapolate from these trends, where do they lead? Here is what I think will be the development experience of advanced teams in the future:

1. Whenever I save a file, my (fast running) tests are run in the background.
2. When all the tests run successfully, my changes are pushed up to my personal clone of the repository.
   A first stage continuous integration server listens to changes from all the developers repositories. When it verifies the tests, it pushes the changes to the integrated repository.
3. Every few minutes, my workspace is updated to reflect new changes from other developers in the integrated repository.
4. After the integrated repository, similar build processes propagate code changes through slower, and possibly even manual tests. The verified result is stored in the staged repository.
5. At the push of a button, I can roll the code from the staged repository into any test or production environment.

Sounds far fetched? Vincent Massol wrote about unbreakable builds five years ago. Distributed version control is being adopted quickly and will greatly simplify the implementation of such processes. Despite Kent Beck’s regretful decision to stop active development of JUnit Max, I believe the time for continuous testing is near. The process I outline can include as enough verification steps to make the organization comfortable. As the trend of improving test quality continues, this process will be gradually more automated.

The strange thing is that we’ve almost made a complete circle: Before the widespread use of revision control, many developers would edit the code directly in their production environment. Extreme Integration will feel almost like this, but with enough non-intrusive verification to make even the most paranoid test manager happy.

Thanks to Martin Eggen for digging up the information on IMVU’s Continuous Deployment. Thanks to Sarah Brodwall, Trond Pedersen and Finn-Robert Kristensen for helpful comments.