cLabs: OptimizingVisualStudioForTDD

At my gig we've got a large Visual Studio solution - 37 projects at the time of this writing. Many of these are small library assemblies, and roughly half are unit test assemblies. It performs rather poorly. Even though Visual Studio is good about checking to see if it needs to build a project (except in a Rebuild, of course), it still appears to require some timestamp checking on source files each time, which is enough disk activity across all the projects to add-up. Plus any pre- and post- build steps are always run -- so having a minute or so build time prevents any pace required for some good TDD.

It'd be nice if it'd keep some dirty markers in memory or somesuch as I code, to just know ahead of time what should or shouldn't need to be built, and then skip the pre/post steps as well. Don't get me started on how it should really go the next step and do background compilation.

I've been tinkering with this recently, looking for a way to improve things, and I think I've come up with a way to have my cake (one fat .sln file with easy access to all the source code for refactorings and analysis) and eat it too (run my unit tests fast and get me in a good TDD groove).

Working with ReSharper, I'd noticed running unit tests for assemblies low in the dependency tree wouldn't take very long. If there's a way to eliminate building the unnecessary dependencies for all tests - well - that'd be yummy.

The first option I pursued was setting up separate solutions for each pair of assemblies - the production assembly and its unit test assembly - and changing all of the project references from the production assembly to file references. This works for increasing the performance, but there are some cons with this approach which cost too much for the performance increase:

- Lots of new .sln files added to source control.

- There's no source code available for the dependent assemblies (with .pdbs, the source can be stepped into during debugging, but it can't be browsed easily, and I don't think R# can work with any of it).

- In order to work with source across multiple assemblies, multiple instances of Visual Studio must be used.

- ReSharper's quick doc view isn't available for code in the file referenced assemblies.

My cohort, Tony Mocella, suggested staying in one big .sln, but adding new build configurations, a separate one for each assembly pair, where only the two assemblies are checked for building. I tried it. It performed well and it has fewer cons than the separate .sln file approach...:

- No crufty increase to files in source control.

- All the source code for all assemblies is available.

... but it still had some cons I didn't like. First, it adds a bunch of configuration cruft to each project file, separate entries for each pair's configuration. This makes the guts of the project file seem very un-DRY - but only when looking at it in a text editor, which ain't a common task. The DRY violations can come back to bite if something like optimization needs to be toggled in each and every one of those configs.

In addition, I found switching between configurations to be slow - 45 seconds to a minute to switch. Less time than firing up a second (or third) instance of Visual Studio like the previous option requires, but still enough to break a rhythm if working in more than one assembly is required.

Tony's idea is pretty good, but once I'd already tackled trying these two options, it turns out I'd already taken care of the core problem: project references. By replacing all project references with file references (except for a project reference from the unit test assembly to its assembly under test), now building each production assembly in isolation took only the time necessary to build that one assembly. Sweet. Without implementing either of our original ideas - additional .sln files or additional build configurations - I got the performance improvement I needed, without any downside.

Except for one thing.

Try doing a clean build of the whole solution. Whoops. Without the project references, Visual Studio doesn't have an accurate way to determine the build order.

Staring at my monitor, cake in hand, unable to take a bite, my mind got desperate.

Then an idea. What if I open the .sln file in a text editor and re-arrange the projects in the order they need to be built? Surely, if Visual Studio has no project references to determine order, it won't disturb the order it read them in from the file?

Sigh. Not so much.

But I found a bit of a silver lining. While Visual Studio won't obey the order of projects in the .sln file, the following macro will:

Dim aProject As Project

For Each aProject In DTE.Solution.Projects
    DTE.Solution.SolutionBuild.BuildProject("Debug", aProject.UniqueName, True)
Next

Put a keyboard shortcut on this macro and I'm good to go. So, all said and done, my cake doesn't have any frosting on it, but it'll do.

There's some details I've glossed over so far. In a large solution, changing all the project references to file references is a pain. But I found opening the project files in a text editor with good macro support was much faster than doing it all through Visual Studio.

I changed the output folders for all projects to be the same folder in both the Debug and Release configurations, and pointed the file references to each project's output folder. I suppose a single common folder could be used across all projects, but I didn't play with it. A single shared output folder could be more convenient. For example, if I'm doing TDD in a dependent assembly, then want to run the entire application, in my current setup, I have to build assemblies further up the dependency chain for the copy local actions to take place. Seems a single shared output folder wouldn't have that problem.

After first discovering the build order in the solution was toast, I experimented with establishing manual dependencies in the Project Dependencies dialog, but these dependencies are the source of the performance problem. Might as well go back to project references.

The macro as listed above has a problem - if a particular build fails, the macro continues on building the rest of the projects. This is just like Visual Studio, except unlike a regular VS build which accumulates error reports for all project failures, each project build in the macro is a separate action, and the error output is reset in between each build. If a project halfway through the solution fails, many of the rest may fail too, but the original error output will be lost. To fix this, there's a macro post build event you can hook into, to set a variable that can be checked in the middle of the original loop. See the “Canceling a failed build” section in this article at the Visual Studio Hacks website for details.

This is still a fairly kludgey approach - if you've any ideas for improving upon, I'd love to hear about it.

tags: ComputersAndTechnology