~Johannes

you could check code coverage before checking in, but I don't think that goes far enough. As we all know coverage != tested.

I'm thinking you might be able to write an issue tracking system which requires acceptance tests to be defined for each issue, and then do some validation when you check in code that ensures the code you checkin is covered by the relevant acceptance tests (linking with an issue id in the checkin comment). It would need to allow changes to existing acceptance tests too.

This doesn't work for refactorings though, as you are relying on the existing tests, not new ones. Perhaps we also need a tool to do some automated invariance testing – just auto generate tests that run on the old and new version of a class/module and check no behaviour has changed between revisions (remember artifactory?). Any API or behaviour changes would have to have a new or modified acceptance test anyway.

The more I look at this space the more I think developers need tools which force them to do the right thing, not just allow them to do it.

you could check code coverage before checking in, but I don't think that goes far enough. As we all know coverage != tested.

I'm thinking you might be able to write an issue tracking system which requires acceptance tests to be defined for each issue, and then do some validation when you check in code that ensures the code you checkin is covered by the relevant acceptance tests (linking with an issue id in the checkin comment). It would need to allow changes to existing acceptance tests too.

This doesn't work for refactorings though, as you are relying on the existing tests, not new ones. Perhaps we also need a tool to do some automated invariance testing – just auto generate tests that run on the old and new version of a class/module and check no behaviour has changed between revisions (remember artifactory?). Any API or behaviour changes would have to have a new or modified acceptance test anyway.

The more I look at this space the more I think developers need tools which force them to do the right thing, not just allow them to do it.

I agree that testing is the big barrier. It is what requires improvements in practice, not just tools, and that's always harder.

I'd also like to hear your thoughts on the auto-checkin and checkout ideas I outlined.

I don't think your vision of the future sounds far fetched at all. A few years ago I was on a project where we used David Saff's experimental continuous test runner and I thought that it was fantastic. We also had very small checkins form the policy of competitive commits: each pair tried to checkin more frequently to make merging the other pair's problem!

However I do think the idea of abundant automated tests will be the last element to go mainstream in practice, even later that distributed source control. Most of the teams I see actually do far less automated testing than you'd think, and often even less than they think (they assume someone else is writing more tests than they are).

One element of this kind of environment you didn't mention is the reporting aspect. When you have all these actions happening automatically it is much easier to know what builds are where, what the difference are between each build, etc.

Other than that I think your description fits very well with what Eric and I were describing as “Enterprise Continuous Integration”. Just today in IM I'd said that at the limit our Elements of ECI (http://www.anthillpro.com/html/resources/elemen…) would become Continuous Building, Continuous Deploying, Continuous Testing and Continuous Reporting, which is an idea I plan to develop further.