Replanning your project with a time machine

I have an amazing time machine that lets me think better about projects. This is part 1 in a series of blog posts exploring the use of a time machine.

Let’s say that you have a project that has been running for a couple of months. Looking back at your issue tracker and other artifacts, you notice that it’s hard to see what has been done and especially how much time remains. You really wish that you had a proper product backlog of what has been done, so you could forecast how the future will be.

Pulling out a mental time machine, you can answer this question. This is how I create a plan that I’d like to travel back in time to give to myself.

  1. Look over the actual features that you have build. If the application is a normal application with a set of web pages, list all the screens.
  2. Which of the screens have features that required extra work, such as integration, search, dialogues or complex business rules? Split the screens that required extra work into one item per work
  3. Which of the work items did you have to substantially change or even throw away all together? Add a work item for each of these events
  4. For each work item, list a rough date when it was completed. If you have know the sprint, that’s okay. Just list the end date of the sprint. If you know the week, that’s great!
  5. Look over the list that you have so far: Are there things that are especially big? Can you find a way to split them? Are there things that are so small that they don’t really count? Can you logically merge some of these together?
  6. Now you’re ready to count the number of work items completed per week.

If we’re lucky, you may come up with a list of 2-5 items done every week. Each of the items has a clear demonstrable effect that could be seen by a customer.

Now you can look at the work ahead of you. Can you find a similar-size items? If you have some sort of screen mockups of the rest of the work you’re interested in, this is a great source of information.

To complete a plan, list a planned completion date for each work item:

  • For past work items, the planned date is the same as the completed date.
  • For future work items, put a planned date that gives the same rate of items per week as in a reasonable interval of the past.

There are many sources of uncertainty still left in such a plan, but it is a quick way to get a reasonable idea of the work ahead.

Posted in English, Software Development | Leave a comment

Software 2015 i regi av DND

Jeg sitter i styret i Dataforeningen Østlandet hvor jeg er med å arrangerer Software 2015. Software er en kryssfaglig IT-konferanse hvor vi samler spesialister innenfor forskjellige fagfelt for å få dem til å snakke sammen på tvers av fagområdene.

På konferansen er det dagsaktuelle temaer og trender som står på programmet. Jeg er stolt av alle de spennende temaene faggruppene har tatt fram og samarbeidet vi har fått til på tvers av fagmiljøer.

Vi ser at både kravhåndtering, devops, prosjektledelse og arkitektur fokuserer mer og mer på å bevege seg mot kortere leveransesykluser. Da blir det viktig å forstå verktøyene og teknikkene som gjør dette mulig.

Et annet viktig tema er den stadig mer sentrale rollen IT spiller i samfunnet. Vi har samlet representanter fra store offentlige IT-miljøer som gjennomfører prosjekter som vil påvirke alle i Norge.

Software har også lykkes i å belyse den viktige debatten som skjer mellom sikkerhet, personvern, kriminalitetet og overvåkning i samfunnet.

Jeg synes også det var morsomt å snakke med Frode som er med å arrangere stormaskinsporet. Jeg har aldri satt meg nok inn i stormaskin, men har vært i flere organisasjoner der hjertet av virksomheten har ligget i stormaskinen. Det har som regel vært en målsetning å bli kvitt stormaskinen, men fordi man ikke har satt seg inn i hva den gjør blir dette veldig vanskelig. Frode sa: “De som vil bli kvitt stormaskinen setter seg ikke inn i hva den gjør – så de får det ikke til. Og de som setter seg inn i den vil ikke bli kvitt den”. Jeg er ikke 100% overbevist om at han har rett, men det er spennende tanker.

Meld deg på Software 2015 du også.

Jeg gleder meg, og håper å se deg også på Software 2015!

Posted in Non-technical, Norsk | Leave a comment

Interactive REST hacking with Sublime Text and RESTer

Do you ever miss a command line to a web application? If you’re lucky, you may use a REST API with appropriate tools for the same job.

This Christmas, I bought a Sonos system for our family. It’s a wireless speaker system that uses music streaming services. It supports both Spotify and Deezer, but works better with Deezer and it came with a one year free subscription, so Deezer it is.

First order of business: Importing Spotify playlists into Deezer. I found a free service that does this, but by accidentally refreshing the browser a couple of times, I managed to import five copies of each playlist. Disaster!

Thus began my quest: I had a number of accidentally imported playlists in Deezer, and it turns out that after deleting a playlist, I needed about five clicks, including hitting a small icon, in order to delete the next. This was not going to be fun.

As a programmer, you know there must be a better way. Perhaps that better way requires more time because you have to program something, but dammit, better is better. In the case of Deezer, I found that they had a well-documented REST API. But this time, I didn’t feel like writing a program just to solve this one problem. And so we get to the core of this post: How can you just execute a fairly large number of API calls without writing a program for it? All I needed to do was to find out which set of API calls needed to be made based on my playlist listing and execute these.

My text editor of choice these days is Sublime Text, which has a number of plugins. The RESTer plugin was just what I needed.

So, in order to delete the extra playlists:

  1. I installed SublimeText 3, Package Manager and RESTer
  2. Looking at the Deezer API doc, I created a test application and entered the URL: https://connect.deezer.com/oauth/auth.php?app_id=YOUR_APP_ID&redirect_uri=http://localhost& perms=basic_access,email,delete_library&response_type=token in a web browser. This gave me an authorization dialog with Deezer that ends up redirecting to an invalid URL on localhost. However, the URL will be something like http://localhost/#access_token=xxxxxxxxxxxxxxxxxx&expires=3600. Just copy the access_token part, as this will be needed for later.
  3. Now, let’s list the playlists. In Sublime Text, I write the URL http://api.deezer.com/user/2529/playlists?access_token=xxxxx (replace the id with your own user id and the access_token with the result from last step) and press ctrl-alt-r. This executes the REST call and returns the result in a new buffer as a formatted JSON text.
  4. In Sublime Text, I bring up the search dialog with ctrl-f, click the “.*” button (regular expression), enter "title"|"picture" and press “Find All”. This selects all the relevant lines from the JSON response.
  5. Now, some Sublime Text editing magic. All the lines from the output are selected. Press shift-end to extend the selections to the end of their lines, ctrl-c to copy the texts to the clipboard, ctrl-a to select everything, del to delete and ctrl-v to paste it.
  6. Whoa! That should replace the text with a set of alternating lines like "title": "something" and "picture": "something". We almost have what we want. Double click on the word "title" on any line and press alt-f3. This selects all the places where it says "title" in the buffer. Press end to move to the end of the lines and del to join the line with the next.
  7. Now each playlist should be on a separate line with the title and the picture link. Press F9 to sort the lines. In my example, I can now clearly see all the lines the are duplicates. So I delete the lines I want to keep and keep the lines I want to delete. For all playlists with duplicates, I remove one line in my buffer and keep the rest for deleting.
  8. Now I just have to craft the delete commands. I double click an example of the word picture. I press right-arrow until the cursors (several!) are between the ” and the https-part. Press shift-home to select everything before https://api... and del press to delete this. Press end to go to the end of the line and backspace until you have deleted the ‘/image”,’ part so each URL looks like “https://https://api.deezer.com/playlist/number“.
  9. We now have our targets to delete. At this point in time, I went to the beginning of the buffer and pressed ctrl-alt-down arrow until I have a cursor at the start of all lines. I now type “DELETE “, press end, paste in the access_token from step 2 and press enter. I now have all the requests that I need to execute separated by a blank line!
  10. Finally, I go to the first line, press ctrl-alt-r which executes the HTTP request. I press ctrl-w to close the response, go down two lines and press ctrl-alt-r to do it again.

This is the result of the first request:

This is what my buffer looks like after selecting titles and pictures and joining the lines:

This is what my buffer looked like in the end.

Sometimes, doing an everyday Christmas task like deleting a lot of playlists is hard to do with the programs available. Luckily, with a little bit of programming knowledge and some sharp tools, you can get the job done through an API.

Posted in Code, English | Leave a comment

Promises you can trust

JavaScript Promises provide a strong programming model for the future of JavaScript development.

So here I’m playing with promises.

First I need a bit of a package.json file:

Now I can write my first test (test/promises_test.js):

Notice that the “it” function takes a “done” function parameter to ensure that the test waits until the promise has been resolved. Remove the call to done() or to resolve() and the test will timeout.

This test fails, but because of a timeout. The reason is that done is never called. Let’s improve the test.

Using “done()” instead of “then()” indicates that the promise chain is complete. If we haven’t dealt with errors, done will throw an exception. The test no longer times out, but fails well:

And we can fix it:

Lesson: Always end a promise chain with done().

In order to separate this, you can split the then and the done:

There is another shorthand for this in Mocha as well:

But what is a promise chain?

This is extra cool when we need multiple promises:

Notice that done is only called when ALL of the strings have had their length calculated (asynchronously).

This may seem weird at first, but is extremely helpful when dealing with object graphs:

Here, the save methods on dao.orderDao and orderLineDao both return promises. Our “savePurchaseOrder” function also returns a promise which is resolved when everything is saved. And everything happens asynchronously.

Okay, back to basics about promises.

Here, the second function to “done()” is called. We can use “fail()” as a shortcut:

But this is not so good. If the comparison fail, this test will time out! This is better:

Of course, we need unexpected events to be handled as well:

And of course: Something may fail in the middle of a chain:

The failure is automatically propagated to the first failure handler:

It took me a while to become really comfortable with Promises, but when I did, it simplified my JavaScript code quite a bit.

You can find the whole source code here. Also, be sure to check out Scott Sauyet’s slides on Functional JavaScript for more on promises, curry and other tasty functional stuff.

Thanks to my ex-colleague and fellow Exilee Sanath for the inspiration to write this article.

Posted in Code, English, Java | 1 Comment

The reuse dilemma

The first commandment that any young programmer learns is “Thou Shall Not Duplicate”. Thus instructed, whenever we see something that looks like it may be repeated code, we refactor. We create libraries and frameworks. But removing duplication doesn’t come for free.

If I refactor some code so that instead of duplicating some logic in Class A and Class B, these classes share the logic in Class R (for reuse!). Now Classes A and B are indirectly coupled. This is not necessarily a bad thing, but it comes with some consequences, which are often overlooked.

If Class A require the shared functionality to change, we have a choice: We make the change or we stymie Class A. Making the change comes at a cost of breaking Class B. If these classes are in the same Java package (or .NET Namespace), chances are that we will be able to verify that the change didn’t break anything. If the reused functionality is in a library that is used by another library that is used by a Class B, verifying that the change was good is harder.

This is where Continuous Integration comes in. We check in our change to class R (for reuse, remember). A build is triggered on Jenkins, Bamboo, TFS or what have you. The build causes other projects to get built and eventually, we get a failing build where a unit test for Class B breaks. If we did our job right.

Even with this failing test, we’re not out of the woods. A build system for a large enterprise may take up to an hour or even more to run. This means that by trying to improve Class R, the developers of Class A have made a mess for the developers of Class B at least for a good while.

When organizations see repeated build breaks due to changes in dependencies, the reaction is usually the same: Lets limit changes to the shared code. Perhaps we’ll even version it, so that any change that you need will be released in the next version and dependencies can upgrade at their own pace.

Now we’ve introduced another problem: We are no longer able to change the code. Developers of Class A will have to take Class R as it is, or at the very least go through substantial work to change it. If the reused code is very mature, this is fine. After all, this is what we have to deal with when it comes to language standard libraries and open source projects.

Most reused code inside an organization, however, isn’t so mature. The developers of Class A will often be frustrated by the limitations of Class R. If the reused class is something very domain specific, the limitation is even worse.

So the developers do what any reasonable developer would do: They make a copy of Class R (or they fork the repository where it lives), creating a new duplication of the code.

And so it goes: Removing duplication leads to risk of adverse interactions between reusers. Organizations enforce change control on the reused code to avoid changes made in one context from breaking other parts of the system, resulting in paralysis for the reusers who need the reused code to change. The reusers eventually duplicate the reused code to their own branch where they can evolve it, leading to duplication. Until someone gets the urge to remove the duplication.

The dilemma happens in the small and in the large, but the trade-offs are different.

What we should do less is to reuse immature code with little novel value. The cost of the extra coupling often far outweighs the benefit of reuse.

Posted in English, Software Development | 1 Comment

Dead simple configuration

Whole frameworks have been written with the purpose of handling the configuration of your application. I prefer a simpler way.

If by configuration we mean “everything that is likely to vary between deploys“, it follows that we should try and keep configuration simple. In Java, the simplest option is the humble properties file. The downside of a properties file is that you have to restart your application when you want it to pick up changes. Or do you?

Here’s a simple method I’ve used on several projects:

The AppConfiguration class looks like this:

This reads the configuration file in an efficient way and updates the settings as needed. It supports environment variables and system properties as defaults. And it even gives a pretty good log of what’s going on.

For the full source code and a magic DataSource which updates automatically, see this gist: https://gist.github.com/jhannes/b8b143e0e5b287d73038

Enjoy!

Posted in Code, English, Java | Leave a comment

Turning a new corner

This summer has been the most significant in my life. While I don’t often write about personal matters in this blog, I will make an exception.

On July 12th my son Leon was born. Healthy, happy and obviously, from the perspective a father, perfect.

Going through this fundamental life change has made me decide to re-evaluate a lot of things. Most importantly, I wanted to evaluate my career.

In the last few years, I have spread my focus thin. I have enjoyed working with developers in Norway, Sweden, Sri Lanka and elsewhere tremendously. I have enjoyed working with many different customers in many sectors. I’ve enjoyed new knowledge in C#, Android, as well as continuing to deepen my knowledge in JavaScript and Java. I’m grateful to the whole team at Exilesoft for giving me the opportunities to learn so much.

As my personal life gets more focused, I also want my professional life to be more focused.

As of August, I start a position as Director of Software Development at BrandMaster; a small, but solid Norwegian product company which develops a Marketing Resource Management (MRM) portal in Java and JavaScript.

For years, I have fluttered around to various companies as a consultant, opining and hopefully helped them develop their products. Now I want to be part of building a product where I can feel ownership and long-term commitment.

BrandMaster is a small company that has the energetic feel of a startup, but the established product and customer base of a mature product. The field of MRM is moving from one of a specialized tool to something that more and more companies find themselves investing in, as witnessed by the entry of Microsoft, SAP, IBM and more large players into the field.

I find that the experiences that I have gathered in BBS, Steria and Exilesoft provide a good basis for contributing to BrandMaster. This means that I will focus on a more narrow set of problems in the future: Moving a product towards greater more frequent releases, more effective use of test automation and agile planning and tracking. It means that I will be able to focus more narrowly on my favorite technologies of Java and JavaScript. And it means that I will be able to work more closely together with a great team of developers and creative people.

This does mean that I will not stick my neck out quite as much when it comes to large projects and organizations, C# and mobile development (although there will be some mobility in the future) in this blog.

Stay tuned: Queued up on my plate are instead super-simple project tracking, demo-driven development, effective unit tests, continuous delivery, 12factor Java 8 and product development.

I’m looking forward to a set of challenging problems at the same time as I get the peace of mind to enjoy the company of my son and his lovely, beautiful and sparkling mother.

Posted in English, Personal | Leave a comment

The lepidopterist’s curse: Playing with java.time

Pop quiz: What will be the output of this little program?

The answer is, like with most interesting questions, “it depends”. How can it depend? Well, let try a few examples:

  • getHoursOfDay(LocalDate.of(2014, 7, 15), ZoneId.of("Asia/Colombo")) returns 24. As expected
  • getHoursOfDay(LocalDate.of(2014, 7, 15), ZoneId.of("Europe/Oslo")) also returns 24.
  • But here comes a funny version: getHoursOfDay(LocalDate.of(2014, 3, 30), ZoneId.of("Europe/Oslo")) returns 23! This is daylight saving time.
  • Similarly: getHoursOfDay(LocalDate.of(2014, 10, 26), ZoneId.of("Europe/Oslo")) also returns 25
  • And of course, down under, everything is upside down: getHoursOfDay(LocalDate.of(2014, 10, 5), ZoneId.of("Australia/Melbourne")) gives 23.
  • Except, of course, in Queensland: getHoursOfDay(LocalDate.of(2014, 10, 5), ZoneId.of("Australia/Queensland")) => 24.

Daylight saving hours: The bane of programmers!

Daylight saving hours were instituted with the stated purpose of improving worker productivity by providing more working hours with light. Numerous studies have failed to prove that it works as intended.

Instead, when I examined the history of daylight saving hours in Norway, it turns out that it was lobbied by a golfer and a butterfly collector (“lepidopterist”) so that they could better pursue their hobbies after working hours. Thus the name of this blog post.

Most of the time, you can ignore daylight saving hours. But when you can’t, it can really bite you in the behind. For example: What does the hour by hour production of a power plan look like on the day that changes from daylight saving hours to standard time? Another example given to me by a colleague: TV schedules. It turns out that some TV channels just can’t be bothered to show programming during the extra hour in the fall. Or they will show the same hour of programming twice.

The Joda-Time API and now, the Java 8 time API java.time can help. If you use it correctly. Here is the code to display a table of values per hour:

Given 2014/10/26 and Oslo, this prints:

And on 2014/3/30, it prints:

So, if you ever find yourself writing code like this: for (int hour=0; hour<24; hour++) doSomething(midnight.plusHours(hour)); you may want to reconsider! This code will (probably) break twice a year.

At the face of it, time is an easy concept. When you start looking into the details, there's a reason that the java.time library contains 20 classes (if you don't count the subpackages). When used correctly, time calculations are simple. When used incorrectly, time calculations look simple, but contain subtle bugs.

Next time, perhaps I should ruminate on the finer points of Week Numbers.

Posted in Code, English, Java | Leave a comment

C# tricks: Securing your controllers

This article is dedicated to the ExileOffice team – revolutionizing the way we run our business in Exilesoft.

As applications move more and more of their business logic to the client side, it falls upon us to simplify what’s left on the server to avoid distractions. My previous article shows how I get rid of lot of the boring and noisy code.

There is one thing that the server will always be responsible for: Security. Much functionality can be moved to the client side, but we have no way of stopping users from hacking our client-side code or faking requests.

In this article, I extend upon the generic controllers to handle the one of the most important aspects of security: Authorization.

This is not your data!

The first order of business is to ensure that only logged in users can access our controller. This is trivially done with AuthorizationAttribute:

Here, our custom AuthorizationAttribute sets a property by looking for the Company of the currently logged in user in the database. You probably want to cache this in a session.

The second problem is to ensure that the user can only read and write data.

According to my previous article, we want to move as much as possible of the boilerplate logic into a generic EntityController. Here we add authorization:

Here, we give the controllers a virtual AccessFilter property. Whenever we read data, we need to run it through the access filter, and whenever we write data, we need to check that the data matches the filter.

It’s up to each entity controller to determine how to match the data. Here is an example for PersonData:

The clue here is to ensure that PersonController can only get at Persons through the Entities collection. This collection is always filtered with the Company that the user has access to.

The code will resolve to simply: db.Persons.Where(p => p.Company == Request.Properties["UserCompany"]).Where(p => p.Type == PersonType.Admin && p.City == city).Select(p => new { p.FirstName, p.LastName }). By using the filtered Entities, we can ensure that we never accidentally forget to apply the users company.

Posted in C#, Code, English | Leave a comment

The madness of layered architecture

I once visited a team that had fifteen layers in their code. That is: If you wanted to display some data in the database in a web page, that data passed through 15 classes in the application. What did these layers do? Oh, nothing much. They just copied data from one object to the next. Or sometimes the “access object layer” would perform a check that objects were valid. Or perhaps the check would be done in the “boundary object layer”. It varied, depending on which part of the application you looked.

Puzzled (and somewhat annoyed), I asked the team why they had constructed their application this way. The answer was simple enough: They had been told so by the expensive consultant who had been hired to advice on the organization’s architecture.

I asked the team what rationale the consultant had given. They just shrugged. Who knows?

Today, I often visit teams who have three to five layers in their code. When asked why, the response is usually the same: This is the advice they have been given. From a book, a video or a conference talk. And the rationale remains elusive or muddled at best.

Why do we construct layered applications?

There’s an ancient saying in the field of computing: Any problem in computer science can be solved by adding a layer of indirection.

Famously, this is the guiding principle behind our modern network stack. In web services SOAP performs method calls on top of HTTP. HTTP sends requests and receives responses on top of TCP. TCP streams data in two directions on top of IP. IP routes packets of bits through a network on top of physical protocols like Ethernet. Ethernet broadcasts packets of bits with a destination address to all computers on a bus.

Each layer performs a function that lets the higher layer abstract away the complexities of for example resending lost packets or routing packets through a globally interconnected network.

The analogy is used to argue for layers in enterprise application architecture.

But enterprise applications are not like network protocols. Every layer in most enterprise application operates at the same level of abstraction.

To pick on a popular example: John Papa’s video on Single Page Applications uses the following layers on the server side (and a separate set on the client side): Controllers, UnitOfWork, Repository, Factories and EntityFramework. So for example the AttendanceRepository property in CodeCamperUnitOfWork returns a AttendanceRepository to the AttendanceController, which calls GetBySessionId() method in AttendanceRepository layer, which finally calls DbSet.Where(ps => ps.SessionId == sessionId) on EntityFramework. And then there’s the RepositoryFactories layers. Whee!

And what does it all do? It filters an entity based on a parameter. Wat?!

(A hint that this is going off the rails is that discussion in the video presentation starts with the bottom and builds up to the controllers instead of outside in)

In a similar Java application, I have seen – and feel free to skip these tedious details – the SpeakersController.findByConference calls SpeakersService.findByConference, which calls SpeakersManager.findByConference, which calls SpeakersRepository.findByConference, which constructs a horrific JPAQL query which nobody can understand. JPA returns an @Entity which is mapped to the database, and the Repository, or perhaps the Manager, Service or Controller, or perhaps two or three of these, will transform from Speaker-class to another.

Why is this a problem?

The cost of code: A reasonable conjecture would be that the cost of developing and maintaining an application grows with the size of the application. Adding code without value is waste.

Single responsibility principle: In the above example, the SpeakerService will often contain all functionality associated with speakers. So if adding a speaker requires you to select a conference from a drop-down list, the SpeakerService will often have a findAllConferences method, so that SpeakersController doesn’t need to also have a dependency on ConferenceService. However, this makes the classes into functionality magnets. The symptom is low coherence: the methods of one class can be divided into distinct sets that are never used at the same time.

Dumb services: “Service” is a horrible name for a class – a service is a more or less coherent collection of functions. A more meaningful name would be a “repository” for a service that stores and retrieves objects, a Query is a service that selects objects based on a criteria (actually it’s a command, not a service), a Gateway is a service that communicates with another system, a ReportGenerator is a service that creates a report. Of course, the fact that a controller may have references to a repository, a report generator and a gateway should be quite normal if the controller fetches data from the database to generate a report for another system.

Multiple points of extension: If you have a controller that calls a service that calls a manager that calls a repository and you want to add some validation that the object you are saving is consistent, where would you add it? How much would you be willing to bet that the other developers on the team would give the same answer? How much would you be willing to bet that you would give the same answer in a few months?

Catering to the least common denominator: In the conference application we have been playing with, DaysController creates and returns the days available for a conference. The functionality needed for DaysController is dead simple. On the other hand TalksController has a lot more functionality. Even though these controllers have vastly different needs, they both get the same (boring) set of classes: A Controller, a UnitOfWork, a Repository. There is no reason the DaysController couldn’t use EntityFramework directly, other than the desire for consistency.

Most applications have a few functional verticals that contain the meat of the application and a lot of small supporting verticals. Treating them the same only creates more work and more maintenance effort.

So how can you fix it?

The first thing you must do is to build your application from the outside in. If your job is to return a set of objects, with .NET EntityFramework you can access the DbSet directly – just inject IDbSet in your controller. With Java JPA, you probably want a Repository with a finder method to hide the JPAQL madness. No service, manager, worker, or whatever is needed.

The second thing you must do is to grow your architecture. When you realize that there’s more responsibilities in your controller than deciding what to do with a user request, you must extract new classes. You may for example need a PdfScheduleGenerator to create a printable schedule for your conference. If you’re using .NET entity framework, you many want to create some LINQ extension methods on e.g. IEnumerable (which is extended by IDbSet)

The third and most important thing you must do is to give your classes names that reflect their responsibilities. A service should not just be a place to dump a lot of methods.

Every problem in computer science can be solved by adding a layer of indirection, but most problems in software engineering can be solved by removing a misplaced layer.

Let’s build leaner applications!

Posted in C#, English, Java, Software Development | Tagged | 17 Comments