Category Archives: English

All my non-Norwegian posts

Dirty Code Monday!

Lately I’ve been thinking about how easy it is to fall into the trap of not challenging our ideas about the code we’re working on. In order to challenge the default mindset of Clean Code, I recently proposed to institute Dirty Code Monday (a proposal that sort of got me into a bit of a big discussion).

Anyway, here is the report from the first successful Dirty Code Monday one week ago:

An unpromising start: I had done a code review with “Programmer A”‘s code on Thursday where “Programmer H” and I had pretty much rewritten all of it. Now we had to go through the results together. We all admitted that code review is not ideal for a Dirty Code Monday, but rules are rules.

The first bit of code we came across was this:

Notice the comment? // TODO: Find out if SSLContext.getSocketFactory is expensive an if so, cache. I sighed. “We really have to remember not to do premature optimization, or we’ll never find out if it is really slow or not.” “Programmer A” responded “What? I thought you said this was Dirty Code Monday! Let’s just do it!” And so we did. We cached the result of createSocketFactory in a static variable. Dirty Code Monday!

By now, I was starting to get my spirits up that we could actually pull this off. We glanced up at the on-wall monitor: Someone had broken the build. “Sorry,” said ‘Programmer K’, “I forgot that my commit also changed the backend.” ‘Programmer R’ chimed in: “Yay! Your turn to get cake, K”. “Not today,” said A, “Dirty Code Monday!”

‘Project manager K’ just mumbles “The cake is a lie”.

I continued the code review with A. Next up was this beauty:

“Hmm…” I said, “I guess that really should deal with DELETE and PUT requests, too. But it’s Dirty Code Monday.” We discussed a little and A mused “a switch-case could work here”. “Ugh! I say, I don’t like switch-statements. They kinda violate the OCP. Maybe. Maybe not here. But I still don’t like it.” He just looked at me and we both laughed.

“Ooooh,” I said. “Look at that repetition of the complicated stream() expression. Perfect!” ‘Programmer A’ was pleased as well “Dirty Code Monday is pretty okay.”

Perhaps you think our code wasn’t that dirty. I’m actually not that embarrassed about it. But changing our perspective helped us see other ways of working with the code than we were used to. Having a phrase like “Dirty Code Monday” was a fun thing for the team to talk about. I’d like to try it again today.

Happy Dirty Code Monday, everybody!

PS: The code in this article is actual production code. The whole team has approved the quotes attributed to them

Posted in Code, English, Java, Pair programming | Leave a comment

A canonical XML test

I recently did a few days of TDD training for a client. They asked me to help them test and refactor a class that created XML from an internal domain model. This gave me the opportunity to examine a bigger pattern.

I wondered where the domain model came from. Looking through the code base, I found that the same or similar data structures were dealt with many places. As often is the case, I also found a bit of code that parsed an XML structure and output the domain model. This made it possible to use my favorite way of testing mapping code: Round-tripping.

The general pattern: To test translation code, you can test the encoding and decoding as one. These tests will often give you a lot of bang for your buck, both in terms of readability and in terms of error detection rates. Their main limitation is that they may not work to exercise all paths of the code well. If this is a problem, you should supplement them with more fine-grained test.

As I have dealt with this sort of problem a few times before, I’ve decided to create my own XML library, Eaxy (as you do). I introduced the library in the tests, but the production code remained using a combination of DOM and JAXB. Here’s a reasonable reproduction of the test:

When I introduced this test to the existing code base, we discovered a few interesting things: 1. There were internal dependencies in the XML file that the developers were unaware of as all the canned test data consisted of huge files that nobody would read. 2. A field was decoded from base64, but treated internally as if it was still encoded, leading to doubly encoding it in the output. 3. The output structure was slightly different from the input structure.

The test, combined with coverage measurements, gave us enough confidence to refactor some pretty crufty code that the team relies on in the future. Round-trip testing can give you a lot of bang for your buck.

Posted in English, Java, Unit testing | Leave a comment

How to write better code

My previous blog post took off on Twitter. It pointed out a problem: Insisting on the obligation to follow certain rules at all times isn’t actually helping people work better. The most common question (or objection) I got to the blog post was: So how do we teach new coders how to code well. This blog post is about that topic.

First learn to collaborate

The most important skill we should teach is how to work well with others on a shared code base. In my career, it took me many years before I started getting good at this, but it had the most impact on my effectiveness. So this is where I want to start with developers who are getting ready to join the work force. On almost all projects we work on, our success will depend on others.

Working with others can be taught. Here are some techniques that have worked for me:

  • Pair programming. Good pair programming requires training, but there exists training styles like coding dojos and code retreats. Some of the best programming instructors in the world use pair programming actively as part of their training.
  • Code reviews. I work so much with pair programming that I personally give code reviews a lower priority. But many teams find great value in them.
  • Patience. When working with pair programming, it’s easy to get stuck in arguments. Our profession says almost nothing about interpersonal skills, yet we depend on them all the time. Try this: When pair programming and you see your pair making a mistake, wait to see if she spots it herself before breaking in.
  • Humility. I often get asked “how do I convince others to write clean code”. Of course, the question assumes that you’re right and they are wrong. Most of us are absolutely rubbish at the skill of changing our minds. When pair programming, if your pair want do something you think is wrong, try indulging them. At worst, you get the chance to teach them something. At best, you learn something yourself.

Unless you are in a situation where you write all the code yourself, collaboration skills are paramount. It doesn’t matter if you “know” what the code should look like if you can’t agree with your team.

Now, let’s talk about code!

If you are one of the people who skipped ahead to this section, please go back and read about collaboration. Seriously. Nothing I say about code will help you if you can’t work well with others.

Go ahead, read it again. I’ll wait here.

Good to see you back! Now let’s talk about learning to write good code. Of course good code is crucial to professional success. If the code base is better, you spend less time fixing bugs, you have an easier time changing the code in the future and your team will have a better time working with you. This is understood.

Learning to write good code is quite simple: You have to read code, you have to write code, you have to change code, you have to observe what is easy and what’s hard, and you have to delete code and start over again.

For the first of these points, great books help you to read code: Clean Code, Implementation Patterns, Refactoring, the Art of Agile, the Pragmatic Programmer, Practices of an Agile Developer, Design Patterns. I’ve enjoyed reading all these books immensely. These books will teach you considerations such as low coupling, high cohesion and simple design. They will teach you useful principles like Single Responsibility Principle and the Open-Closed Principle (although I find I almost never refer to the rest of SOLID). The patterns and principles teaches you words to discuss code with your team.

For the second of these points, Test-Driven Development is one great way to learn how to write code. I enjoy doing coding katas myself and often use them for teaching. But the most valuable skill when writing code is one I learned at code retreats (thank you, Corey Haines!). Learn to delete code you write. I don’t just mean refactor it to be smaller. I mean that for coding exercises, highlight all the files and press the delete button. I mean for production code, after spending a few hours working on a task, (occasionally) git reset --hard HEAD.

The hallmark of great code is how easy it is to change. Can you navigate in the code? Can you spot errors quickly? Do you know where to make a specific change? You can learn by reading about principles, but the most safe teacher is to expose yourself to more experience. (I’m planning to write a blog post on how principles sometimes, but not always coincide with good code). When you write and change code, reflect on how limitations of the code and your current skillset impedes the changes you want to make.

In short: Code more, listen to the code and listen to your peers.

Can you release smoothly

I almost forgot the #1 quality of good code: It has to be used! Release cycles of teams can vary from several times per day until a few times per year. If you can release your code at will, you can learn both about your code and your users. When the whole team learns that you will release again tomorrow, or next week, priorities of “now or later” gets easy. When you release only a few times a year, people get stressed and discussions get heated.

I’ve almost always find that team under-appreciate the value of spending time on build and deployment tools and scripts.

“Don’t do stupid stuff on purpose”

If you find code you’re unhappy with, it bears remembering that it is that way because it got that way. People did what they did because of reasons. Those reasons are valid, whether it was because the surrounding needs changed, because the developer had insufficient experience, because of pressure to finish, because they wanted to go home to their family instead of sitting late in the office or maybe they just had a difference in opinion on what’s good code.

Not everyone agrees with this, “what about those who just don’t care?” After 20 years of working as a programmer, I have never encountered anyone who just didn’t care. I have encountered programmers who had different preferences from my own, I’ve encountered programmers who are delivering under short deadlines, I’ve encountered programmers who get praised for making appearant progress at the expense of writing good code. And I’ve encountered many programmers who don’t care as much as me about the code. Having other interests in our life is an excellent reason to spend less time polishing code.

But I’ve never encountered a programmer who just doesn’t care. Have you? I’d love to learn from your story.

Posted in Code, English, Pair programming | Leave a comment

Forget about Clean Code, let’s embrace Compassionate Code

When your heroes start acting weird, you reexamine their influence on your life. I’ve long been learning, demonstrating and teaching clean code through TDD, patterns and so on. But when I look back, I am now worried that the ideas negatively influence my life and my work and that of others.

Many who know me consider me an exceptionally skilled programmer. I got that way because I have often spent my evenings practicing programming techniques and technologies. I often leave the office 1-2 hours later than my co-workers after polishing some piece of ultimately meaningless code. This is time I don’t spend with my family. Because I’ve learned to care about Clean Code.

Some of the most unpleasant experience in my professional life have been due to disagreements of the Clean way to write some piece of Code. I know people that I respect professionally that could have been good friends if one of us hadn’t insisted on purity of code in some situation or another.

And I’m not alone. I’ve seen teams that are at conflict with each other because of different expectations of a programmers obligations. Often fuelled by the idea of Clean Code.

As I’ve grown older, I learned to stop getting upset about “Unclean” Code. In any real world code base there will be an uncountable number of unfortunate quirks, odd and ends. This is okay. Things are the way they are because they got that way. People did what they did because of reasons. Those reasons are valid, whether it was because the surrounding needs changed, because the developer had insufficient experience, because they wanted to go home to their family instead of sitting late in the office or if they just had a difference in opinion on what’s good code.

I sometimes train teams in TDD, incremental design, refactoring and pair programming. I almost always have to spend a lot of time helping the team bridge their disagreements in a more constructive way. And often the ones who have read Clean Code are the most susceptible to this sort of conflicts.

Why do I bring this up now? Because the reason that the idea of Clean Code is unhelpful as a guiding principle came in stark relief last weekend. The reason is simply this: We get led astray when we follow a principle or a rule without reflecting on what it does to ourselves and others. Robert (Uncle Bob) Martin is the author of the Clean Code book and a prominent developer trainer. He often writes on obligations, the oath to defend and preserve the honor of the profession, and shames excuses for not doing TDD. For a long time, there has been a nagging feeling at the back of my head about this language of “honor”, “obligation” and “professionalism”. But I enjoy test-driven development, and I have experienced it as a more fun way to develop. I believe that developers should reject illegal orders. I agree with Uncle Bob on all of these specific points.

Only when he writes something that I strongly disagree with, does the hunch about Clean Code became clear. The last days on Twitter we watched Uncle Bob implicitly decry the violation of Godwin’s law rather than the internment of thousand children under conditions that Amnesty International compare with torture. In the following days, Uncle Bob fought back against his critics also stressing that he thinks the situation is horrible, “but…” not as important as “unhonorably” comparing others to Nazis. I think his priorities are horribly wrong. I responded “In light of @unclebobmartin’s recent tweets, ideas like Clean Code have started creating a bad taste in my mouth. Let’s just say “code”, eh? I’m officially instituting “dirty code Monday’s” to remember to question dogma, tribalism and things-before-people mentality.” Bob asked me to explain “why you decided to lead a boycott against the concept of Clean Code”. Thus this blog post.

I deeply disagree with his stance on this political issue. I respect that some people value rules and principles higher than individual fates. I do not value rules for themselves. Bringing this back to code: I don’t believe we should use TDD because it’s a professional obligation. Instead I use TDD when it makes my work more enjoyable. I don’t think we should refactor our code because it violates a SOLID-principle. Instead I sometimes reach to a principle to understand why some piece of code is hard to change or understand. I don’t want to shame people for writing Unclean Code. Instead I believe in having an honest dialog among equals about how we want our code to look. I don’t believe that professionalism should compel us to introduce tests for all untested code. Instead I believe we should prioritize which deficiencies we fix and which code monsters we allow to live out their lives in their part of the code base.

I want to accept my unclean code as battle scars to be proud of and to be humble about, not as failings to be ashamed of.

My friend Thorbjørn Sigberg writes If your agile transformation has only one goal, it should be “Do less boring stuff”. When Clean Code becomes a source of less boring work, I’m for it. When it becomes a source of frustration and guilt, I’m against it.

For me, the ideas of Extreme Programming that bring the greatest joy to my professional life and that of my team are ideas about the whole team, about pair programming and about focusing on the users. Uncle Bob acknowledges these elements of the programming practice, but hardly ever talk about how to do it well. I don’t agree with these priorities.

As Uncle Bob nominated me as leader of “the boycott of clean code”, I guess I should try to end with something profound. How about this: Your most valuable skill is to know what’s important. Code is not important. Principles are not important. Rules are not important. People are important. That means your users, your team, your family, yourself. To quote Joshua Kerievsky’s Modern Agile: Make people Awesome. Clean Code may help or hurt that goal. Learn to see the difference.

Posted in English, Extreme Programming, Non-technical | Leave a comment

A wicked Java trick to make the JVM forget to check exceptions

I’ve long time been a critic of the mechanism of compiler checked exceptions in Java. Whether you love them or hate then, one thing is sure: There’s situations where you don’t want to have to deal with them. The solution in Java is to wrap a checked exception in new RuntimeException(e) but this gives long stack traces without adding useful information. Sometimes, we just want to tell the compiler to chill.

As it turns out this is possible, through some wicked abuse of the type erasure misfeature of Java-generics. Seeing this in action is instructive for understanding the inner workings for Java. Let’s go!

Here is what we want:

Notice that the businessLogic() neither catches IOException or declares that it throws IOException. Instead, the softenException() method removes the checkedness of the exception. When we run it, we get the following stack trace:

Huh! The exception being thrown in the main method is a NoSuchFileException, which is a subclass of IOException – a checked exception! How can that be? Why didn’t any of the methods in the program have to declare throws IOException?

Here’s the trick:

The checkednessRemover method uses a trick that reveals a few things about the inner workings of Java. First, the generic type argument T is bound to RuntimeException in order to fulfill the contract of softenException. This means that the expression throws T becomes throws RuntimeException, which the compiler interprets as if there was no exceptions thrown.

But the statement throw (T)e; theoretically should evaluate to throw (RuntimeException)e;. Since e is a NoSuchFileException, you would expect this statement to result in a ClassCastException. But the way generics work in Java, the type information is removed by the compiler. So instead, the bytecode reads as throw (Exception)e;, which is fine.

So this strange trick shows that the Java compiler removes generic information from the compiled code and that checked exceptions is purely a feature of the compiler. There are no runtime verifications of checked exceptions.

Would I recommend using this trick in production code? I don’t know. It’s pretty weird and may not be that useful, but I do use it myself when I’m feeling wicked. If nothing else, I hope learning about has given you some insights into the inner workings of Java.

Disclaimers: (1) I read about this trick somewhere else, but I cannot find the source any longer. I thought it was Heinz Kabutz’ excellent Java Specialist newsletter, but I can’t find the source. (2) This is also implemented in Project Lombok as @SneakyThrows. If you are using Lombok, you should under no circumstance reimplement the trick from this blog. Use @SneakyThrows instead.

Posted in Code, English, Java | 3 Comments

The same risks in every projects

To avoid the big problems with projects, everybody recommends risk management. At the same time, I’ve rarely seen risk management practiced effectively. Do we identify the same risks and do we actually prepare to handle them? The ironic thing is that I think most projects have the same top four risks. In this blogpost, I explore these common risks. To avoid exposing my customers and colleagues, the examples given is based on hearsay and not actual experience.

The biggest risk: Is the problem even worth solving?

The biggest risk of a project is that the problem we’re trying to solve is not worth solving. The project may have uncertain costs and will only save the company an insignificant amount of money, or we may be creating a product that nobody is interested in paying for. If what we’re doing is not worth doing, it doesn’t matter how well we’re doing it.

Lean startup recognizes this risk at the center of the process. This is why the Minimum Viable Product is such a strong concept. This is a great idea if the project is about developing a product for a market.

What if the project is about delivering something that a customer ordered? What if your customer ordered something that they don’t need? Your contract may protect you financially, but you will still have wasted your time working on something insignificant.

As an example: Everyone wants to develop chatbots today, but do you know how much you are currently spending on your human customer services representatives? Is it even worth reducing this cost?

Will the proposed solutions solve the problem in a reasonable way?

Even if we know the problem is real, will our solution actually address it? In the example of the chatbot, we expect that our customers will get their answers from the chatbot, so they won’t bother our (expensive) human operators. But what if our customers always end up transferring to a human after all? Many chatbots are nothing more than fancy, less usable, FAQ databases, which may be cheaper to implement anyway.

Do we have the means to execute our solution?

Even if the solution would solve the problem if properly used, do we have the skills, data, customer relationships and market conditions to execute it well and without extreme expenses? In the case of our chatbot, the chatbot won’t do us any good unless we have the people who can train it and the data to train it with. As technologies are becoming increasingly specialized, it’s often much less available skills than demand.

Do we have the processes to deal with fundamental risks?

When a fundamental risk materializes on a project that has been started, it may be very hard to actually deal with it. We had a project at my company where it turned out that after putting in a few person-years worth of effort, we realized that there was no value to the project. It took us two weeks from the realization until we had discussed the discovery sufficiently that everyone saw the same situation and that we had described the situation in a way that made the top management able to make an informed decision to terminate the project and cut our losses.

We’re not always so lucky. In many organizations, the inertia of a project is so large that there is no understood decision process that makes us able to terminate the project once we realize it’s a waste of effort.

Dealing with the fundamental risks

The three biggest risks to the value of any undertaking are:

  1. The problem may not actually be worth solving
  2. The solutions we select may not actually solve the problem
  3. We may be unable to actually master the solution effectively
  4. We may be unable to terminate out-of-control projects

If one or both of the first two risks actually materialize for your project, your current undertaking is a waste of effort. If the third occurs, it costs more than it’s worth. If the last occurs, you will be unable to stop the bleeding. All projects have these risks, but they may materialize differently. What are your fundamental project risks?

Posted in English, Non-technical | Leave a comment

Privacy concerns everywhere, but don’t panic!

Recently, my organization reached a big goal and we decided to celebrate by taking everyone out for dinner. What happened was a GDPR nightmare. Or was it?

We sent out a Google Form with three simple questions: 1. What’s your name, 2. Are you coming to the dinner? 3. Do you have any dietary constraints?

But wait!

According to article 4 of the GDPR, “‘personal data’ means any information relating to an identified or identifiable natural person”. Is the dietary constraints personal data? It certainly can be!

Julie wrote: “I’m a vegetarian”. That’s certainly personal information. Is it lawful to collect it? Jason wrote: “I’m allergic to shellfish”. That’s actually medical information. Fareed wrote: “I don’t eat pork”. Did we just collect data about his religious belief? Both medical information and religious beliefs are considered special categories of personal data, requiring stricter basis for processing.

But it gets worse: Google Forms doesn’t guarantee that the data is stored outside the US. As long as the US is a rogue nation, especially when it comes to privacy, article 45 will probably never allow for worryless transfer to the US. And I have to admit that when it comes to transfer to rogue nations, I really don’t know what to do.

So, what the heck, right? What’s going on here? Is the GDPR making everything impossible?

Not at all, we just have to start thinking in terms of common curtesy in a time when massive amounts of personal data is in danger of being misused. Let’s fix these problems!

The biggest problem is Google, and we’re all holding our breath on this one. Is Google going to provide sufficient control to comply with the GDPR after May 25th? We all hope so. If not, perhaps we need to find another place to collect the data. (Hey, Google! Norway is a great place for data centers – why don’t you leave the sinking ship that is the US?)

Second, collecting the data is lawful if we get the https://gdpr-info.eu/art-7-gdpr/consent of the data subject. Just add a checkbox saying “I consent that this information is used to pre-order the food”. We also have to provide transparent information to our hungry team members. “This information will be aggregated with the rest of the responses and used to place our order. It will only be read by Johannes. After the order is placed, the information will be deleted”. Remember to restrict access to the data as you promise and delete the data when you promise.

Yes, this means that you have to ask again for the next party. Don’t cry – that’s not that big of a hassle.

Finally, and most importantly, perhaps we should take a lesson from Data protection by design. Instead of asking if our members have dietary constraints, perhaps just ask the restaurant for the menu and ask everyone what they would like to order. A food order is (probably!) not personal data and then we sidestep the whole problem. Or we could just anonymize the dietary question.

What’s the point of this story?

Data protection is an everyday problem. It doesn’t just affect big IT systems. It affects all the small information streams within an organization.

But at the same time, data protection doesn’t require a project to carry out. It just requires you to follow a reasonable set of principles. It means you have to think about whether you have the consent of the person subject, whether you’re upfront about how the data will be used, whether you protect the data and that you consider whether you need personal information at all.

A huge disclaimer: I’m not a legal expert in any sense of the word. I may be grossly wrong in this article and I’d love to hear about it! Please comment!

Posted in English, Non-technical | 2 Comments

Using Trello and Google Forms to organize a conference

We’re running Mobile Era for the second year on October 5th-6th and I’d like to share some experience on how we’re using scripts and Trello to help with the organization effort. If you’d like automating simple tools, this is the article for you.

If you haven’t signed up for Mobile Era yet, you can leave a comment in this blog post for a discount code!

Google Forms is great! You can simply design pretty advanced forms and easily get responses. But after you get the responses, it’s a bit trickier. Similarly, Trello is great: Once you have your tasks in place, you can work on them easily by dragging and dropping. We decided to combine them with a little bit of script magic.

For a conference, one of the big tasks is to collect talk submissions and select who should get to speak at the conference. Here’s how we did it:

  1. A potential speaker submits a talk in our Google Forms call-for-presentation.
  2. Our magical script creates a card in a Trello board for the talk
  3. If the speaker updates the talk, the Trello card gets automatically updated (and marked with a special label)
  4. The program committee can comment on the card and move it around, safe in the knowledge that any work they do will be preserved even if the speaker updates the card

I started doing this with Trello’s email integration, but this fell apart when I wanted to update existing cards. Now I use the Trello API instead (there is a Zapier integration that has the same limitation, too). Here is the source code for my Google Forms script.

This is how you use it:

  1. Create a Trello board with an “Inbox” list and a “Updated” label. You will need to find Trello’s ID’s for these, which I leave as an exercise to the reader.
  2. Create a Trello API key and token at https://trello.com/app-key
  3. Create a new Google Form and set it up with the fields that you want
  4. In the Forms editor, open the “…” menu and select “Script Editor…”
  5. Copy the text from my gist into the Script Editor
  6. Update the autentication, idBoard, inboxList and updatedLabel values with the IDs from your Trello account
  7. Update “getCardName” and “getCardLabelIds” to include relevant fields from your form
  8. Run the function “sendAll” to send any existing form submissions to Trello. This will make Google Forms ask for permission to store the links between Trello cards and Form submissions.
  9. Add a trigger to “saveToTrello” on “On form submit”

The clunkiest part of the code in the gist is probably finding the values of a defined form field. This is also brittle if the form is changed, but what can you do, eh?

I’ve also started using information Trello to merge emails and save drafts for easily editing and sending. But that’s a story for another day.

A little bit of script can save a lot of tedium and help you extend tools like Trello and Google Forms to work together. I welcome your comments on how the script could be made even better!

Posted in Communities, English, Software Development | Leave a comment

How I learned to love GDPR and so can you

If you are working with software development as a developer, manager or tester, then you will be impacted by the General Data Protection Regulation (GDPR) – the new EU laws regarding data privacy. In many ways, the regulation is likely to have as big of an impact as the Y2K problem. But this time it’s because of a good cause! And you cannot ignore it, as the fines for doing so can be crippling. But for most people who find themselves face-to-face with GDPR it’s quite intimidating.

I hope this article can do something about that.

First, let’s say a few words about how to approach the regulation itself and then let’s look at some actions you should consider. I find the text of the regulation to be surprisingly well-written, but I wish I had a reading guide when I started out.

The actual text is officially published on http://ec.europa.eu. The text starts with 173 “recitals”. These are background considerations for the regulation. Even though they are well thought out and well written, they are not very much to the point. They are also often quite heavy to read, even if it’s for good reason (a perfect example is Recital 38 which talks about the data protection concerns of children). After the recitals are the actual 99 articles of the law, which are much easier to read. These are divided into chapters and many of the later sections are about the structure for enforcement, not for the actual regulation as it impacts most organizations.

Instead of reading the PDF, I recommend looking at https://gdpr-info.eu, which has organized the regulation in an easy-to-use structure. If you are responsible for an IT system, you need to understand at a minimum articles 1 through 50 and especially articles 5 through 35. Start by reading these.

Let’s look at some of the ways you may be surprised.

Surprise 1: Consent and test data

In order to collect and use personal data, you need to have permission to do so (article 6). For most people, this either means that you are required by law to use the information (for example medical information in a public health context) or that you have obtained consent from the data subject. This has actually been the case for a while, but some things will be more explicitly required: First, you must make it clear what your user is consenting to and second, you must make it optional (and non-default!) to give consent in cases where it’s not needed to provide a given service.

One very common scenario is for organizations to use production data from their customers for testing purposes. You can forget about that in the future. You could ask your customers for consent to use their data for testing purposes, but you must make this consent optional and non-default. So that means that at best, you need to find good routines to extract only data for consenting customers. Good luck! You could anonymize the data, but you are liable if there is a risk of reidentification.

Instead, I recommend that you invest in other testing strategies. In particular, most testing organizations can improve a lot by creating synthetic data. By investing in synthetic data, you can also improve your ability to stress the system with large and unusual values. Another testing method is partial production, where you gradually let more users onto a new version of the system. Perfecting this will also improve your delivery cycle a lot.

Now you have an excuse to invest in better testing.

Surprise 2: Functions required to support the rights of the data subject – data portability

When you store personal data, you now have to plan for functionality where the subject of this data can exercise their rights regarding the data. Much of this has already been the case, but the existing rights have been strengthened. This is described in articles 12 through 23. Just add them to your product backlog as system functionality or manual processes that must implemented. Basically, your customers have the right to see what data you are storing about them, including who has accessed the data. They should be able to correct errors in the data and to ask for data to be deleted.

A new and very interesting right is the right to data portability (article 20). Your customers have the right to get their data from you and take it to your competitors in a portable format (“structured, commonly used and machine readable”). As I understand it – if your customers can get an exported JSON, XML or CSV-format with everything concerning them, you’re pretty much set. If you wonder: PDF is not good enough.

Data portability is one of the most exciting parts of the GDPR and seems to be motivated by a desire to promote innovation. Just imagine what you can do with it! Imagine an app that can import your purchase history from all major store chains and help you analyze your own buying habits. It’s long been a dream, but the data was locked up. Until now!

Or you can use it to keep your competitors honest! Tell your customers that if they give you consent and upload their data from your competitors, you give the discounts and prizes. You have to be prepared for the case that your customers withdraw their consent again, but you are still allowed to keep aggregated data. And if you’re competitors are laid-back about GDPR – well, you can really put their feet to the fire!

Surprise 3: Keep data safe during transfer and storage – everywhere!

The final important consideration I want to talk about that comes out of GDPR is keeping data safe and under control (article 32 and also article 25, which mandates Privacy by Design). You are required by law to protect personal data in transit and rest at all locations. You are also required to report to the local authorities any breach of data that you detect.

What you need to do now is to map out every place you transfer personal data. What about temporary storage areas? What about backups? What about third parties that receive data? And what about logs?

Most organizations have less access protection of application logs than any other data. And often you log without considering the contents of the logging. I used to practice an approach of logging the full payload of all incoming and outgoing communication messages. That may no longer be a good idea.

The easiest way to protect data is to make sure you never collect it, and failing that, making sure that you don’t store it unnecessarily. You need to trace every piece of personal data through your systems and find out who can access it. If you can collect less, that will make your job easier.

Get started

There are more aspects of the GDPR that I have not discussed. In particular, the role of the Data Protection Officer is critical and has some surprising nuances.

But the goal of this article was to give you a place to start where there’s a good chance that you have something you need to do and perhaps something that you can benefit from as well. Here are three suggestions: 1. Verify that you’re not using production data for testing and develop new testing techniques if you do, 2. Add development tasks to support the rights of the data subject – a good place to start is Data portability, 3. Analyze the flow of personal data through the system, especially looking for less secured storage locations like logs and temporary files.

It’s not long until the law comes into effect and you need to get ready! The three places I’ve pointed out where you need to start touch most of the IT system related aspects of GDPR and can be used as a basis for understanding, implementing and benefiting from our improved rights to our own data as citizens and individuals!

Posted in English, Non-technical, Software Development | 3 Comments

Deliver early without sleepless nights

Trailblazing the first delivery of a software system requires courage and conviction, especially on projects that replace existing business critical software. When I’ve been acting as system architect I’ve employed a number of tricks in order to structure functionality and technical solutions in such a way that we can complete these early deliveries without sleepless nights. The most important is to find a subset of functionality that can be used with the rest being completed and investing in bridges from the old to the new.

Next Thursday (April 27th), I share my experience in the keynote for the ARK architecture conference in Oslo. A few tickets are still available if you want to make it.

In this blog post I explore the topic of replacing business critical software step by step.

As an architect I haven’t always have the joy of seeing my projects all the way to completion, but in those projects where I can see the software in the hands of my users early on are those I enjoy the most.

This article is not about streamlining a working delivery team to get from releases every few months to continuous deliveries. Many others talk about that. This is about how to get to production early the first time on a greenfield project. I will illustrate with two of my favorite projects.

Getting to the production the first time on a greenfield project takes courage and conviction, especially when you’re replacing part of business operations with a new system. Project stakeholders have many reasons to wait and often see no compelling reason to delivery a partial solution early. But when you get to production the first time, it’s like the sun has finally risen. The priorities and discussions in the project totally change. Now it’s real!

The first project I want to talk about was with the Norwegian electricity transmission system operator Statnett. We spent 4 years replacing the system that handles all reserve capacity of electricity deliveries on the Norwegian grid, but less than a year after the contract was signed, the users started using our software to control the power grid. Without any sleepless nights. Actually, they considered it such a non-event that they forgot to inform the development team about it!

The second project I would like to talk about is an app for internal mobile workers in private sector. As it is a project which is part of the competitive strategy of my customer, I cannot mention the purpose of the software, but I can describe techniques and technologies.

mobile workforce architecture

The mobile workforce app is especially interesting. Just like with Statnett, we were replacing an existing system in business use and integrated with other business functions. But in this project, we were live with the first users after 3 months, most of the user base after half a year and the remain users after around 9 months.

Here is what we did:

  • Realize that the journey is going to feel long and temporary investments made to ease the journey are often worth it. In both cases, I made sure the new system integrated well with the new system. I cannot stress how much it lowers everyone’s stress level when you can say: Why don’t we try the new system and if you experience problems, you can continue the very same process in your old system.
  • Isolate functionality that is small enough to deliver with a small fraction of the total project effort yet interesting enough to care about. In the case of Statnett we were able to replace the most used feature in the old system (while preserving the path of retreat). Even though the feature was a small part of the system, it was a large part of the usage. In the mobile workforce app, we found a business process that was missing in the old system and implemented this first. The process was interesting enough to the customer that they dedicated a few users to only perform this task.
  • Don’t wait for the rest of the world to complete their work. All projects I’ve been on has been part of a larger landscape of change where several of your dependencies are still under development by other projects. In the case of the mobile workforce app, we were dependent on core data that was to be owned by a system still under development. Realizing that our integration model would in any case be based on synchronizing copies of data, we decided that our copy of the core data would have several sources: 1. A manual CSV file exported by a sysadmin, 2. An automated dump from the legacy source which we established after half a year, 3. A feed from the new system (which is still not completed).
  • Spend effort making your system faster to deploy and easier to monitor. When you make a greenfield project, there is actually nothing that stops you from creating the production environment immediately (as long as you don’t put production data there before it’s hardened!). I’ve never regretted spending a few extra hours making it easier to deploy new version or making my logging smoother. Currently, I receive error logs on Slack with a clickable stack trace and a link to the user who experienced the problem.

When you do a major rehaul of a building, you may end up spending a lot of your effort setting up the scaffold to make sure the work is safe and effective. When you build a new software system, you should do the same!

If you do, you may be able to get users on your new system sooner rather than later without losing your sleep.

And when you get actual users to use your system the primary question of the project changes from “when is it all going to be done” to “what can we do next to deliver value.” And that’s much more fun.

Posted in Agile Release Patterns, English, SOA, Software Development | Leave a comment