Category Archives: Software Development

Can we learn to restrict our work to a budget?

I’ve previously talked about the idea of shifting from estimates to budgets.

The fundamental point of this article is that it’s more useful to control costs than to predict costs.

The problem of this argument is whether it’s possible to develop software in that way. How will the relationship between the developer (or supplier organization) and the customer (or the customer organization) have to change? Is this a chance we’re able to make?

In my popular style, here is a typical dialogue:

  • Customer “… So, we’re going to change the patient appointment request form to include the social security number of the user and use this field in all the relevant processes. Can it be done in two weeks?”
  • Developer “I believe so. But we probably can’t automate everything in this case.”
  • Customer “That’s all right, just do what you can within that timeframe”
  • A week passes
  • Developer “I’ve done a lot of work, but I haven’t got anything to show you yet.”
  • Customer “You’ve spend half the budget, and we’ve got nothing to show for it? That’s it, I’m pulling the plug. Throw away what you’ve done and let’s work on something else.”

Meanwhile – in another universe

  • Developer “Let me show you: I’ve got the field in the appointment request form and it’s also displayed when looking at the appointment details. I’ve tested this automatically and manually, and deployed it on the acceptance test environment.”
  • Customer “Great work so far. These are the next improvements we want (in order of importance): Make sure that non-privileged users never see the number, include the field in the invoicing process and make the input form validate the social security number. Then integrate the field in other processing as well.”
  • Developer “Sure thing. I’ll talk to the users for details.”
  • Another week passes
  • Developer “I’ve fixed the security issue by checking the user privilege before displaying the field. I’ve also included the field in the invoice process. But I didn’t have time to add the validation. Everything is tested, of course.”
  • Customer “Great! We’ve spent the budget, so we’ll stop there. The validation is actually pretty important, so I’ll considering adding a new user story to the budget for that.”

In this story the developer had to produce a minimum story before spending half the budget. This ensures that we get something when the budget runs out. The developer had to finish one expansion of the user story at a time until the budget was expended. And then stop! When the customer wanted one of these sub-features bad enough, he had to create a new user story, thus expanding the scope of the project.

Can we learn to work in this way? As a developer, can I learn to organize my work so that I have something to show for it after just half the budget? Can I learn to keep track of the time I spend so well that I know when I’m over budget? And can I have the discipline to stop once the budget has been spent? As a customer, can I learn to accept that the developer will do the best he can within my budget and accept that?

Posted in English, Extreme Programming, Software Development | Leave a comment

Lean architecture

Lean thinking describes seven classical sources of waste 無駄:

  • Overproduction: Making stuff that nobody buys
  • Over-processing: Creating stuff that is more fancy than what the customer wants
  • Transport: Moving stuff from one place to another in order to create it or give it to a customer
  • Motion: Expending unneeded effort while creating stuff
  • Defects: Having to redo work because it wasn’t done right the first time
  • Inventory: Storing stuff waiting to be worked on more
  • Waiting: The time stuff is just sitting there, waiting to be worked on

Often an eight is added: Unused talent

It’s always fun (but sometimes risky) to draw analogies with other fields. In an enterprise information application (that is: what most people are working on these days), “stuff” is the data that your application shuffles between users, databases and other systems.

Here are the seven wastes of enterprise information application architecture:

  • Overproduction: Any code that is not executed by an end user. In the extreme case, many applications contain code that is not “reachable” and so can never be executed. In other cases, it may be functionality that is available, but never used
  • Over-processing: Do you really need to scale to a million concurrent users for a company internal application? Do you really need an application that is used get data for monthly reports to have 98% uptime?
  • Transport: How many classes in the code touches a single piece of data before it’s displayed to the user (or stored in the database (or sent to another system)). How many systems does a piece of data touch before it’s done processing?
  • Motion: How many times do you transform data into another representation?
  • Defects: Do you accept data that may be in error? How do you deal with old data that contains errors?
  • Inventory: I don’t know… maybe are you deleting obsolete data?
  • Waiting: Do you have batch processes that run daily? ‘Nuff said

(As an aside about transport. Did you know: In C#, it’s requires substantially less work to save and find data with EntityFramework than it does to call a web service? If system A “helps” system B by provide a data access service, system A is actually causing more work for system B. It’s not just an investment with low ROI, it’s got negative ROI!)

Remember: Not all production is overproduction and not all transport is wasted transport. But much is.

Posted in English, Software Development | Leave a comment

Humble architects

Humility is not a very common trait with software architects. After having worked with a few awful architects and recently with a very pleasant one, I’ve compiled a few of my experiences in the way every architect loves: As a set of rules.

Rule 0: Don’t assume stupidity

It seems like some architects assume that developers, if left to their own devices, would behave like monkeys. In my experience, this is very rarely the case. The only situations where I’ve seen developers do something stupid is when silently protesting against an architect. If you follow this rule, the rest are details.

Rule 1: You may be wrong

When reviewing someone’s design idea, I prefer to try and ask questions that are honestly open. Maybe I think the developer have overlooked a critical fact, for example concurrency. There are a few different approaches to the situation:

  1. Architect: “You can’t do it that way, because it breaks the code guidelines”
  2. Architect: “You can’t do it that way, because it won’t be safe when there are several users”
  3. Architect: “Have you thought of how it will work with several users?”
  4. Architect: “How does your solution address the situation of several users?”

Dear architect: Please rate these approaches from the one least to the one most likely to get a best possible system. (Tip: This is an easy task, even though many architects fail it routinely)

Rule 2: Be careful with technology

Every technology comes with a cost. Many technologies come with an infinitesimal benefit.

Here’s a list of technologies that I’ve experienced as having consistently higher cost than benefits, and thus will never use (if you don’t know them, don’t worry. The point is the number): JavaServer Pages, Java Server Faces, JAX-WS, Hibernate, Spring, EJB, Oracle SOA Server, IBM WebSphere, Wicket, Google Web Toolkit, Adobe Flex, JBoss jBPM, JMS (all implementations), JBoss.

Here’s a list of technologies that I happily use: JUnit, Jetty, Joda-time, Java Standard Edition.

Here’s a humble exchange that you may want to try and copy:

  • Architect: You should use technology X
  • Me: I’ve looked at technology X, and I don’t see how it will help me solve the business problem
  • Architect: What do you mean?
  • Me: Well, this is what we need to do: ….. And this is what technology X assumes: …. I don’t see how they match.
  • Architect: So what do you suggest using instead?
  • Me: Um… I think we can solve this with plain Java. As a matter of fact, I made a pretty good proof-of-concept yesterday evening.
  • Awesome architect: Cool. Let’s use it.

Rule 3: Consistency isn’t as important as you think

If I’d have a penny for every time I’d hear this….

  • Architect: “Yes, I know this way may seem clumsy, but you have to do it. You see, if you don’t, the system becomes inconsistent and impossible to maintain”

Granted, I don’t often work with maintenance, but I know that when I deal with any system, the most difficult part is understanding the business logic for that system. Whether system X (which has one set of business logic) and system Y (which has another) are consistent is very low on the list of things that make me lose sleep.

The fact that system X is horribly complex because it has a dozen layer to be consistent with system Y – now this is something that does make me want to pull out my hair. Different contexts have different trade-offs.

Oh, yes: Remember Rule 0? Assume that the developers in a given context are trying to create a good solution for that context.

Oh, yes, another thing: I’ve never seen something that was incomprehensibly complex in the small become maintainable just because we grew to become big.

Oh, yes, yet another thing: If a programmer really runs screaming away from the code because some of it had one style of curly braces and some had curly braces in another style, I’m going to lose all faith in humanity.

Rule 4: Bottom-up consistency beats top-down consistency

There is one way I’ve been able to create more consistency inside a system:

  • Create a reference application with an architecture that is easier to follow than to break. If you do this well, the developers will shake their head at the very idea of deviating from the architecture. Unless they really need to. In which case it’s okay.
  • Foster a culture of cross-pollination: Developers who see each other’s code have more consistent code than developers who just see their own code. Pair programming, code reviews and tech sharing sessions all foster cross-pollination.

Rule 5: Tactical reuse in across systems is suboptimization

Reuse creates coupling. If system X and system Y reuse some functionality and system X needs that functionality to be modified, this will affect system Y. At the very least, the team working on system X must decide to make a private fork of the reused functionality, which means that it’s no longer really reused. At worst, system Y will get a bug because of a change in the reused functionality.

When you reuse across systems, what you reuse should either be stable (say: the Java SE platform, or something so stable that you definitely didn’t make it yourself) or strategic. By strategic reuse, I mean services that integrate information and not just duplicate functionality.

In other words: Reuse should either be use or integration. Duplication is your friend.

Rule 6: Separate between rules and dogma

There are three reasons to have a rule in any coding standard:

  • Unsafe: The code has a bug that will manifest under some (non-theoretical) circumstance
  • Incomprehensible: “I” don’t understand what’s going on
  • Heresy: The code is written in a style some person doesn’t like

Pop quiz: If you have a rule that says “all fields must have a JavaDoc comment”, is that a safety issue, a comprehensibility issue or a heresy issue? What if the standard uses this example:

What about the rule that says “no newline before an opening curly brace”? What about the rule: “the style used for curly braces should be consistent”? Is it addressing Unsafe code, Incomprehensible code or Heresy?

We should be more concerned with writing appropriate code for the situation and less concerned with appeasing the Gods of Consistency.

Rule Omega: Be humble

In the years I’ve worked with software development, I’ve seen more harm done by software architects than help. As a professional role, I think we’d save money if we got rid of them (us). Even if we still paid their salaries.

When you’re in a profession that causes more harm than they prevent, you have two options: You can try and improve, or you can pray that nobody notices.

Posted in English, Java, Software Development | Leave a comment

Micro-Scrum: A stamp-sized version of Scrum

“Show frequently what you’ve done to someone who cares”

Are you working in the way you are because it’s a good idea, or just because someone told you to do it? I increasingly hear experienced professionals at Agile conference bemoan the blind adherence to the techniques of Scrum without understanding the principles and values that make it work. I also encounter many software professionals who are overwhelmed by the amount of things that they are asked to do. The result is the questions “are we ‘Agile’?” and “should we be doing ‘Agile’ or not?”

I die a little bit inside when people say “should we do Agile or not”. The assumptions behind this question are all wrong: That there is one way to “be Agile”, that learning from Agile methods like Scrum requires that you use everything in those methods, and that there are good reasons to be “not Agile”. All of this is wrong.

To me, the most essential lesson that the Agile manifesto tries to communicate is that of Feedback. And frankly, if you’re ignoring feedback on your project, you’re stupid. There are different constraints around the feedback on different project, but essentially, delaying feedback is delaying critical learning.

The most essential manifestation of feedback in Scrum is the Sprint review, or demo. No matter if you’re calling what you do “Scrum”, “Kanban”, “Cowboy coding”, “Waterfall” (which is more like Cowboy coding than a real process) or just “the way we do stuff here”, you will get benefit from showing what you’ve done to someone who cares frequently. For some projects, “someone who cares” may be an end user, while for other projects it’s not feasible to involve end-users frequently. On some projects, “frequently” may mean every day, other projects may not be able to do more than once every month. You may even find that nobody cares about what you do. If that’s the case, surely you can find more relaxing ways of doing it.

When you are showing what you’ve done to someone who cares frequently, you can improve. You can think about how to show it better, how to have the show more accurately reflect what you’ve done, how to involve more people or people how care more and how you can do it even more frequently. Some of the techniques of Scrum may help you do that, but use whatever source of inspiration you like.

No matter if you’re excited about the word “Agile” or not, if you’re not getting feedback, you’re not only non-Agile, you’re non-smart.

Posted in English, Extreme Programming, Software Development, Technology | Leave a comment

A canonical web test

In order to smoke test web applications, I like to run-to-end smoke tests that start the web server and drives a web browser to interact with the application. Here is how this may look:

This test is in the actual war module of the project. This is what it does:

  1. Configures the application to run towards a test database
  2. Starts up the web server Jetty on an arbitrary port (port = 0) and deploys the current web application into Jetty
  3. Fires up HtmlUnit which is a simulated web browser
  4. Inserts an object into the database
  5. Navigates to the appropriate location in the application
  6. Verifies that inserted object is present on the appropriate page

This test requires org.eclipse.jetty:jetty-server, org.eclipse.jetty:jetty-webapp and org.seleniumhq.selenium:selenium-htmlunit-driver to be present in the classpath. When I use this technique, I often employ com.h2database:h2 as my database. H2 can run in-memory and so the database is fresh and empty for each test run. The test does not require you to install an application server, use some inane (sorry) Maven plugin or create any weird XML configuration. It doesn’t require that your application runs on Jetty in production or test environment – it work equally fine for Web applications that are deployed to Tomcat, JBoss or any other application server.


If you are developing a web application for any application server and you are using Maven, this trick has the potential to increase your productivity insanely. Stop what you’re doing and try it out:

  1. Add Jetty and HtmlUnit to your pom.xml
  2. Create a unit test that starts Jetty and navigates to the front page. Verify that the title is what you expect (assertEqual("My Web Application", browser.getTitle()))
  3. Try and run the test

Feel free to contact me if you run into any trouble.

Posted in English, Extreme Programming, Java, Software Development, Unit testing | Leave a comment

Om å løse alt bortsett fra det egentlige problemet

“Problemet med Java er at det krever så mange abstraksjoner. Factories, proxies, rammeverk…” Min samtalepartner gjenfortalte inntrykket han hadde av de Java-programmerende kollegene sine.

Jeg måtte innrømme at jeg kjente meg igjen. Kulturen rundt Java-programmering har noen sykdomstrekk. Kanskje det minst flatterende er fascinasjonen for komplekse teknologiske løsninger. Et gjennomsnittlig Java-prosjekt har rammeverk (Spring, Hibernate), tjenestebusser – gjerne flere (OSB, Camel, Mule), byggverktøy (Maven, Ant, Gradle), persisteringsverktøy (JPA, Hibernate), kodegeneratorer (JAXB, JAX-WS), meldingskøer (JMS), webrammeverk (JSF, Wicket, Spring-MVC) og applikasjonsservere (WebSphere, WebLogic, JBoss). Og det er bare starten. Hvor kommer denne impulsen fra?

Jeg har hørt to teorier om hvorfor Java-programmerere ender opp med en kompleks hverdag. Den ene skylder på sjefene, mens den andre skylder på programmererne. Jeg vil ta for meg begge og peke på en vei ut av jungelen.

Teori 1: Steak and strippers

Zed Shaw, mannen bak “The motherfucking manifesto for programming, motherfuckers” skylder på IT-sjefene. Han mener at selgerne fra teknologiske giganter tar med IT-sjefer ut på strippebuler og kjøper biffmiddager til dem. Og vips så kjøper IT-sjefen inn et ubrukelig verktøy som programmererne er tvunget til å bruke. (Zed påpeker at det vil være positivt med flere kvinnelige IT-sjefer, ettersom de i det minste ikke vil være like interessert i strippebulene)

Ref: Zed Shaw: Control and responsibility ( Zed Shaw: Programming, motherfuckers ( Zed Shaw: Video – Steak and Strippers (

Argumentet er underholdende, men forklarer bare en del av problemet. Mange av verktøyene som står bak kompleksiteten i Java-prosjekter er åpen kildekode og selges typisk ikke av selgere med fete representasjonskontoer.

Teori 2: Blinkende lys

Jeg tror en viktigere teori er at programmerere er opptatt av fancy ting som blinker. Enkle ting som firmaet tjener penger på er kjedelige. Å sette seg ned og flytte data fra databasen og putte det i en webside er kjedelig for en programmerere. Å lære seg et nytt rammeverk som gjør den samme jobben på en fancy måte er spennende. Å fjerne teknologier og lage en enklere løsning er kjedelig. Å innføre et rammeverk som skal skru sammen teknologien er spennende.

Alle foretrekker å gjøre det de synes er spennende. Jeg vet det, for jeg har vært der selv.

Grunnleggende sett tror jeg at Java-programmere selv har skaffet seg de problemene de ofte klager over med komplekse teknologier.

Veien ut av villmarken

For meg var det viktigste skrittet ut av villmarken foredraget jeg hold på JavaZone for to år siden. I foredraget bygger jeg og Anders Karlsen en webapplikasjon i Java uten å bruke webrammeverk. (Innbild deg at du hører et ironisk gisp her)

Jeg har øvet meg på å løse de problemene jeg har i prosjekter med å bruke enklere teknologier. Slik vet jeg hva de komplekse løsningen gjør. Og så langt er det nesten ingen løsninger som ikke innfører mer problemer enn de fjerner. Det de gjør er at de flytter fokuset fra den oppgaven programmet egentlig skulle løse over til alle de teknologiske delene som man skal få til å fungere sammen.

Jeg tror ikke programmerere er bedre eller dårlige enn andre mennesker når det gjelder det. Men vi har alle en tendens til å bruke tiden vår til å finne spennende problemer å løse i stedet for å løse det vi egentlig skulle på en naiv og enkel måte. Min far sa det egentlig best: Du bør ikke bruke en kalkulator før du kan løse de samme regnestykkene på papir.

Posted in Java, Non-technical, Norsk, Software Development | 27 Comments

Only four roles

Many sources of stress on projects come from forgetting what our roles are. Scrum championed a simple set of roles with the development team, the Scrum Master, and the Product Owner. The first problem is the people affected by agile projects who fall into any of these categories, many of which are important. The second problem comes from forgetting that the only roles with authority, the Scrum Master and the Product Owner are the least important people on the whole project.

When creating something of value, the first people we care about are those who will get value from the product we create. I call these Consumers. These are the users and those who are affected by the work of the users. For a call center application, it would be the customer service representative as well as the person calling who has to wait for the service rep to look up information in the slow system.

Without caring about the Consumers, the product has no value.

The second category are those whose work goes into creating the product. It may be people who create layouts and graphics, people who develop the application, people who examine the application to make sure it performs as needed and people who train the consumers in using the application. I call these people the Creators.

Without the Creators, there will be no product.

But in order to create the product, someone usually has to put money on the line. I call this the Sponsor(s). The sponsor is the person who can really decide that, “yes, we will let five people work on this for a year”. If the Creators work for free, they are also the Sponsors. Otherwise, the sponsor is the person who signs their paycheck.

Without the Sponsor, the Creators will starve.

It’s worth noting that many Product Owners, Scrum Masters, Architects and Project Managers fall into none of these roles. The product owner is seldom an actual Consumer of the product, and in very few cases does he pay the salary of the Creators. Instead, he talks to the Consumers and helps the Creators understand what to create. In the same way, a good Scrum Master can ask good questions of the Creators that will help them avoid impediments and work better.

I call everyone who doesn’t Consume the product, Create the product or Pay for the product a Helper. When you facilitate a meeting, write a report or take the requirements from the Consumers to the Creator, you are helping. If you’re doing your job right.

The funny thing is this: Most people with authority in most organizations have Helper roles. But nothing is worse than a “Helper” you don’t need, but who insists that you do what they say.

I am a Helper, and this makes me nervous. If everybody is a Helper, nothing gets done. At best, I can make others better able to do their job. At worst, I distract from real progress.

Helpers must be humble

Posted in English, Extreme Programming, Software Development | 3 Comments

Scrum as an impediment to Agility

As I’m working with smaller and more agile projects, I’m increasingly seeing the classic way that Scrum is executed as more of an impediment to agility than a helper.

This is especially the case when it comes to the classic Sprint Planning as described in the Scrum Guide:

  • “For example, two-week Sprints have four-hour Sprint Planning Meetings”
  • In the Sprint Planning Meeting part 1: “Development Team works to forecast the functionality that will be developed during the Sprint.”
  • In the Sprint Planning Meeting part 1: “Product Owner presents ordered Product Backlog items”
  • “Work planned for the first days of the Sprint by the Development Team is decomposed to units of one day or less by the end of this meeting”

I’ve seen many sprint planning meetings struggle for the same reasons again and again:

  • The user stories described by the product owner doesn’t fit the team’s way of working
  • The team dives into too many details on each user story to be able to break it down to the level required
  • The team blames the product owner for not providing enough details to the user stories
  • Most of the design discussions are considered to be over once the sprint starts
  • The forecasting/commitment to future velocity becomes a heated negotiation

If your project experienced these sorts of Sprint planning meetings, I would expect that the reaction of the project was to add meetings (“backlog grooming”), documentation and checkpoint prior to starting a new sprint. These activities would probably resulted in the product owner (team) spending less amount of time with the development team.

Scrum’s Sprint planning is assuming a situation where the product backlog is detailed for a considerable amount of time and where the ideal is that the product owner spends their time adding more details to the product backlog all the time.

The resulting projects have huge rigid backlogs describing the details for several months into the future. They communication between the users and developers is limited to the acceptance criteria that the product owner writes down before each sprint planning. They spend a considerable amount of the sprint planning the rest of the sprint. Deviations from the sprint backlog are considered problematic.

I think this is misguided. I think this is why we left waterfall in the first place.

In order for Scrum to work better, we have to abandon the idea that the product owner comes to the planning with a perfect set of stories, we have to abandon the sprint backlog detailing the work and design for several weeks and we probably should be very careful with what estimates we ask for.

Instead I would suggest the following approach to planning a sprint:

  • The product owner and the team comes into the room informed by their current understanding of the value the system can deliver
  • The product owner describes the current most important gaps in the value available to stakeholders
  • The team already knows their current trajectory and together with the product owner, they can describe “what’s the next meaningful thing we could demonstrate to closing these gaps” as a script for the next demonstration
  • The team isn’t asked to estimate their work, but the product owner, project managers and others are free to make qualified guesses based on the team’s past performance
  • Keep it short and frequent!

Scrum was developed in the time where it had to match the perception of projects that did huge batches of planning and design. In response, it does smaller batches of planning and design. But “give a man an inch and he’ll take a yard”. The smaller batches leads to frustration over lack of details and the sprints become more and more plan driven and the connection between the users and the developers more and more document-driven.

A new approach is needed.

Posted in English, Extreme Programming, Software Development | 8 Comments

How to start an agile project

During the recent panel debate in Colombo Agile Meetup my colleague Lasantha Bandara asked the following question:

How do you start an agile project and ensure room for future enhancements? How can we achieve flexibity at the beginning?

This is my answer:

Flexibility is about having an acceptable cost of change. Sometimes, the best cost of change is to create something and throw it away early to try something else if it doesn’t work out.

The first technical decision I make on projects is what I call the deployment model. That is: How will the users access the application and where will the data reside. The most common categories of deployment model include web, disconnected client, rich client with server, and mobile client. There are other less common as well, for example an email responder. A mobile web client could also be considered a category of its own and so may perhaps a rich JavaScript web application.

As Sabri Sawaad said during in the panel: You must make decisions, but consider the cost of changing them.

You have to choose a deployment model and client-server communication protocol before you write any code that you can demo. Often, the project comes with an assumed deployment model, but it’s not always set in stone. For example, a project where the customer was aiming for a traditional web application, we suggested a JavaScript application due to the skillset of the developers. In another project, we challenged the customer to try a rich client instead of a web application, but after the first sprint, we discarded the code and did the web application instead. In a final example, discussing the requirements further with the client it turned out that a web client was more appropriate than the initial idea of a mobile application, so we had to replace team members to get the correct skillset.

As an architect, it’s part of your job to help the customer find a deployment model that matches the requirements of the users and the skillset of the team.

On the panel, Subuki Shihabdeen and Buddhima Wickramasinghe both pointed out the importance of early feedback. In my opinion, this implies that the beginning of a project is a bad time to learn new technologies. When you know the deployment model, use the technologies and frameworks you are familiar with to quickly show something to the customer. If this means delaying using a database, as Hasith Yaga suggested in the panel, do this. If it means using Entity Framework and SQL Server, because the team is familiar with this, do that. But as Sabri said: Invest in strategies to minimize the cost of changing these decisions, such as encapsulating the data layer.

In order to quickly choose and roll out a first demo, the only think I’ve found to work is to practice. I’ve created applications repeatedly from scratch in many frameworks and languages for practice. Each time I do it faster.

So here is my process, summarized:

  1. Spend time to practice technologies so you can choose good options with confidence
  2. Help the customer choose an appropriate deployment model based on the user needs and developer skills (including your own)
  3. Put something in front of the customer quickly by using your existing skills
  4. Invest to reduce the cost of changing decisions on frameworks and technologies
  5. Change course if you find a better technology or even deployment model
Posted in English, Software Development | 4 Comments