you could check code coverage before checking in, but I don't think that goes far enough. As we all know coverage != tested.

I'm thinking you might be able to write an issue tracking system which requires acceptance tests to be defined for each issue, and then do some validation when you check in code that ensures the code you checkin is covered by the relevant acceptance tests (linking with an issue id in the checkin comment). It would need to allow changes to existing acceptance tests too.

This doesn't work for refactorings though, as you are relying on the existing tests, not new ones. Perhaps we also need a tool to do some automated invariance testing – just auto generate tests that run on the old and new version of a class/module and check no behaviour has changed between revisions (remember artifactory?). Any API or behaviour changes would have to have a new or modified acceptance test anyway.

The more I look at this space the more I think developers need tools which force them to do the right thing, not just allow them to do it.

you could check code coverage before checking in, but I don't think that goes far enough. As we all know coverage != tested.

I'm thinking you might be able to write an issue tracking system which requires acceptance tests to be defined for each issue, and then do some validation when you check in code that ensures the code you checkin is covered by the relevant acceptance tests (linking with an issue id in the checkin comment). It would need to allow changes to existing acceptance tests too.

This doesn't work for refactorings though, as you are relying on the existing tests, not new ones. Perhaps we also need a tool to do some automated invariance testing – just auto generate tests that run on the old and new version of a class/module and check no behaviour has changed between revisions (remember artifactory?). Any API or behaviour changes would have to have a new or modified acceptance test anyway.

The more I look at this space the more I think developers need tools which force them to do the right thing, not just allow them to do it.

I agree that testing is the big barrier. It is what requires improvements in practice, not just tools, and that's always harder.

I'd also like to hear your thoughts on the auto-checkin and checkout ideas I outlined.

I don't think your vision of the future sounds far fetched at all. A few years ago I was on a project where we used David Saff's experimental continuous test runner and I thought that it was fantastic. We also had very small checkins form the policy of competitive commits: each pair tried to checkin more frequently to make merging the other pair's problem!

However I do think the idea of abundant automated tests will be the last element to go mainstream in practice, even later that distributed source control. Most of the teams I see actually do far less automated testing than you'd think, and often even less than they think (they assume someone else is writing more tests than they are).

One element of this kind of environment you didn't mention is the reporting aspect. When you have all these actions happening automatically it is much easier to know what builds are where, what the difference are between each build, etc.

Other than that I think your description fits very well with what Eric and I were describing as “Enterprise Continuous Integration”. Just today in IM I'd said that at the limit our Elements of ECI (http://www.anthillpro.com/html/resources/elemen...) would become Continuous Building, Continuous Deploying, Continuous Testing and Continuous Reporting, which is an idea I plan to develop further.

Funny that you are pointing out that we are really heading back to where we came from. In 1996 the deployment step to production was to do cvs up in the source-directory where the perl-scripts accessed by mod_perl or mod_cgi was placed. Immediate upgrade and deployment in seconds.

There were a lot of things that could go wrong, continuous staging and testing was a different story back then. What I think we could learn from these practices is that you can build and deploy systems avoiding all those tedious package-deploy-restart cycles that both complicates the process and increases lead-time to production. Add the staging and testing that you describe in your article to a system where immediate deployment is possible with minimum orchestration is needed; I hope that will be the direction of the future.

The stepwise integration that you are talking about has been provided in products like TeamCity for quite a few years, although I am not familiar with widespread deployment of this product. I fully support your viewpoint on distributed source control being the key here. The functionality need to be intrinsic in the SCM, not in a product put on top.

Often I find IT-departments standing in the way when opting for staging/test and acceptance infrastructure that are in the hands of the developers. Getting that problem out of the way, and in addition going for open-sourcing of code that you normally store in closed in-house repos, you don't even need to have a repository internally. Some of the answers to our problems with getting the equipment we need to facilitate the steps might be found in the cloud, both as a service and a platform; configuring and provisioning a new staging environment might be done in minutes.