Sat through an interesting seminar from Kevin Schofield, General Manager of Research at Microsoft Research yesterday, while he visited our Marsfield Lab. In what was a relatively short presentation, Kevin covered only a tiny part of the work at MSR in any detail. Despite time constraints, however, a couple of the points he made really made me stop and think, including his clearly heartfelt comments on the disappearance of Jim Gray.

On the technical side, one of the astounding take-home points for me was the magnitude of complexity in Microsoft’s various code bases. Code complexity is something I’ve been thinking about a bit lately, particularly in terms of concurrency. Kevin’s point was in rather a different dimension of complexity – that of software testing. In quantifying this complexity, he noted that running the full suite of regression tests over the Windows code base takes 8 weeks! On a large farm of servers!

Just think about what the impact of 8 weeks would be on your release schedule. Kinda makes it hard to have a reliable yet agile release cycle, no? If Microsoft wants to run their full suite of regression tests to ensure old bugs have not been reintroduced by new code changes, then there is a huge impact on the agility with which Microsoft can release new versions, respond to bugs and release critical security patches. While the magnitude of their problem may be somewhat larger than most due to the age, size and complexity of their code base, I’m quite sure Microsoft is not alone in having to face such a problem.

Understandably, MSR has been working to address this issue, in part by deriving mappings of the code exercised by each and every test in the regression suite. These mappings are stored and used to prioritise the regression tests, such that the tests that cover the modified code are exercised first.

Sounds like a very logical approach, and I’m surprised that I haven’t come across such techniques before. Perhaps I just haven’t looked in the right places. How do you manage your regression test suites? Do you partition or prioritise them in any novel ways?