Help my software team go from good to great.

March 22, 2008 12:04 PM

I'm specifically looking at automated builds, unit testing, better source control, and generally moving away from seat-of-the-pants coding and deployment, to specific processes. This is a Microsoft shop, if that helps.

I'm a regular ol' software developer on a team that is growing in size, complexity, and responsibility. We work on internal applications like the intranet and the CRM, and have recently begun integrating these things with our extranet as well. Historically there has been "the guy" that works on any particular system, so imagine "the CRM guy" and "the intranet guy" who knows everything there is to know about that system, from its hardware setup all the way to specific code release processes. We are mainly a C# shop.

The thing is that we are growing and multiple developers have started working on the same projects, the same code bases. This means that setting the working directory to a dev folder and holding exclusive locks from source control forever to work on things in solitude is ending, fast. Developers are starting to trip over each other, and the learning curve when you're assigned to an existing project is steep, as you have to be mentored by that system's "guy" for days to fully extract his knowledge about all the intricacies of the system.

For systems I build from scratch, I try to set up a true development environment; local virtual machines, documentation on the dependencies to release that code, a staging environment, and whatnot. This still isn't cutting it in some ways. Our knowledge of source control (we use SourceSafe) is tribal, limited to the aforementioned check-in/check-out. No labels, no branches, and sometimes not even an easy way to develop locally, instead having several developers hacking at a system on a "development" server. Which, as you could imagine, gets nasty as someone starts doing multiple IISresets while troubleshooting something and the other people on the system grind to a halt. Also deployments get tricky; for instance recently a release had issues because nobody could remember all the sproc and table updates necessary for the code to work right, and we basically spent an hour after the deploy running through the live code to see what SQL call would break next, triggering multiple "oh yeah, that!" moments.

I try to track down ways to help us out by religiously reading and researching ideas I see from places like Coding Horror and Eric Sink's awesome source control writeup, but it seems a huge challenge. Things that might be taken for granted at other places, like a build server or required build scripts or unit tests, don't exist here. I think this could be chalked up to the pressure-cooker environment and the fact that we aren't pissing off true customers by screwing up a release, only internal employees that typically know us well (being "the guy" brings a certain amount of fame/infamy in the company). We can move very fast when an issue pops up, but I want to keep that from popping up in the first place.

We are "developers" or "software engineers" by job description, but there's a lot more to it. QA is done by the engineer mostly, then by the internal end users requesting the system/change, so no dedicated guy there. No DBA; let's just say you learn SQL and database administration really fast when you start here. So talk of things like using Team Foundation Server and Team System might be way too much of a change; I'm sure it's awesome, and looks it, but injecting a massive process change like that might backfire.

I have done things like DDL triggers on sprocs and tables to timestamp database modifications and store the SQL that did it, and this weekend I hope to get one of the projects I'm working on now some build scripts, and maybe even unit tests if I get really brave (TDD is not something we do, either, which is starting to scare me as some of the systems grow in complexity). I will be looking at MSBuild in particular to help us out, though I have done some relatively simple build scripts for other apps using PowerShell.

FWIW, our direct technical manager is all for this, as he came from a much more structured shop, and recognizes that we are growing and have to change. He encourages better process, but the inertia of "the way it's always been done," the down-to-the-wire deadlines, and the normal problem of herding the developer cats seems tough to overcome quickly. I want to spend some extra time breaking new ground in what we do, and then present the process to others as a model for how to do things in the future, and hopefully get some formality on it after that. My team is great, I love the job, and want to better our processes so we're not as stressed out.

So to make a long story longer, my question to the hive mind is this: what small incremental steps do you see that could get us more streamlined and less prone to "the one guy" development problems?

posted by quixxotic to computers & internet (15 comments total)

Start using Subversion for source control.
posted by mpls2 at 12:36 PM on March 22, 2008

setting up a wiki and actually populating it with the knowledge "the guy" has locked away in his head seems like a good start.

vss uses exclusive locks by default buy can be configured to use what they call shared checkouts. i have not used vss in a long time but its merge capabilities were pretty bad last time i did. you may want to look at other merge tools.

in my experience many small development teams migrate to vault from vss as they grow. vault has its issue but it is certainly better than vss. vault is designed with this in mind and as a result the transition is pretty painless.

as for database changes i would look into one of the many sql diff tools out there. i have used apex sql diff before and have no complaints.
posted by phil at 12:40 PM on March 22, 2008

I'll recommend a few things to read:

* The Joel Test: 12 Steps to Better Code
* Daily Builds Are Your Friend
* The Pragmatic Programmer: From Journeyman to Master (Book)
* Pragmatic Version Control: Using Subversion (Book)

If I were you, after reading all of the above, I would probably do the following, in this order:

1. Set up a subversion repository and start using it on all projects. Even projects with a single developer should be in subversion. Schedule a task to back up your repo to another server nightly. Now you'll never lose any code on any project again.

2. Set up local development environments for all of your developers. Every developer ought to be able to run his application on his own computer. It sounds like you're building web applications, so this means every developer runs IIS, SQL Server, etc., locally. You should be able to disconnect from the Internet and still work on your app. Now folks can restart their local web servers, make sweeping changes to code, and so on, without disturbing anyone else.

3. Set up an automated build process for every project. You should be able to build an application from source and deploy it into production in a single step. Everyone on the team should be able and know how to run your build task. (I haven't done any .NET development in a while, but I think Nant is the tool of choice in the .NET world for scripting builds.)

Your build process should handle database schema changes. I don't know the best way to handle this in .NET projects, but look for something similar to migrations in Rails.

4. Set up a build server, and schedule a daily build of every application. So, every day at noon, your build process runs: it checks out the code from the SVN repo, compiles source code, swaps/modifies configuration files as needed (for example, so that the built app points to the database on the build server), deploys it to the server, and restarts IIS. The build process should email the team if it fails.

After you've done all of the above (which, don't get me wrong, will take substantial effort) you will notice a massive improvement in your team's efficiency and well-being. You will wonder how you ever got along without these tools and processes.

Next, I would begin to think about using bug tracking software and starting to create automated test suites for your applications.

It sounds like you have the ability and will to make these changes happen. If you can pull this off, you will be a hero at your company. Good luck!
posted by medpt at 1:14 PM on March 22, 2008 [2 favorites]

I'd like to take medpt's recommendation of an automated build, and suggest taking it to the extreme of a continuous build server. It does a checkout, build, runs unit tests, then deletes, and does it again. Then you can do the once a day integration build automatically that puts it up on the local server for everybody to view.

As far as VSS is concerned, I have heard too many bad things about it. Re: subversion, there are various ways to get subversion directly into the environment, google on that. Be sure to use labels for anything that gets shipped/moved to production. Don't worry too much about branches. Look into what they can do, but they lose lots of their advantages when you are doing internal development (since you never support old versions, as there is only one install of your software).

As far as unit tests are concerned, developers who aren't used to it take some time to get used to them. But once they start understanding the benefits, they really take to them. I think a big key is a culture that doesn't punish somebody for writing tests, don't rush somebody when they say "I have to finish testing this module", and code reviews can then enforce coverage. Just a quick 20 minutes one-on-one review can catch when a developer slacks off. Just have all pieces of code looked over by another developer informally before checking in. Don't worry about jumping right into TDD. The developers can be pushed into that later, after they realize the huge value in having good tests.

You mentioned virtual machines. Having a standard SQL Server install sitting in VMWare could be a very powerful tool. Maybe keep it up to date with a copy of the live database? That way they have representative data to test against. Also, one restore to snapshot away from fixing any problem induced by config testing. Having IIS installed is important too, so the developer can mess with anything they need on their own system.
posted by cschneid at 1:41 PM on March 22, 2008

If you're worried about using something like Subversion or CVS for source control because you're a Windows shop, you should look into TortoiseCVS or TortoiseSVN (depending on the source control you end up using) because they actually provides hooks to Windows Explorer, so that any IDE that offers a Windows Explorer view to your files will let you easily do file checkout, commit, update, etc.
posted by Deathalicious at 1:44 PM on March 22, 2008

mpls2: I'm genuinely curious - why would you suggest SVN for a Windows shop? Especially as they already have source control in place?

To be honest quixxotic, it sounds like you're doing everything right. The trick, IMO, to improving processes is to just go ahead and do it. Start small with incremental improvements; Rome wasn't built in a day. For example you don't need buy-in from the entire team to knock up a unit test every time you fix a bug. Once the unit tests are in source control, and seen to be useful, the rest of the team will be on-board soon enough.

Need to keep notes? Throw up a wiki on the test server and start typing. Allow anonymous edits. Respond to emails about processes with links, rather than text. Format comments in whatever C#'s equivalent of JavaDoc is.

Keep everything lightweight and implicit. The more you talk about formal processes and documentation, the less buy-in you'll get.
posted by Leon at 1:53 PM on March 22, 2008

step 1 is get the hell off source safe. It's a ticking time bomb.

step 2 is move to a modern version control system; exclusive locks on files simply don't work with today's development styles. I use TortoiseSVN; CVS is a bit long in the tooth. Many love Mercurial, YMMV.


step 3 is to set up a build script using nmake or nant, so you aren't relying on the IDE to run your compiles.

step 4 is to start using NUnit to execute automated unit tests as you develop. I had extreme difficulty using the test code in visual studio 2008- it seemed crippled / obtuse, particularly around file paths that changed per developer instance.

step 5 is to start using cruisecontrol.net to run your builds every few checkins.

In a java shop, step 3.5 is to migrate to a project management system like Maven that will manage your documentation, compilation, and dependencies- I'm not sure if there's a .net rendition out there that is very strong yet.

Good luck! None of these steps will take that much time per project (a day or two, tops)- the harder part will be retraining your co-workers and modernizing your processes.
posted by jenkinsEar at 1:55 PM on March 22, 2008

@jenkinsEar: Glad that you mentioned the Visual Studio 2008 testing capabilities, as I was about to jump into them a bit. I'll do some research, but it does seem that the majority of people out there with a Microsoft shop still prefer NUnit. I'll do some research, but would love to hear of other experience with VS2008 Unit Testing versus NUnit (or another package).
posted by quixxotic at 2:11 PM on March 22, 2008

I work as a Software Engineer and over the past several years, we've implemented most of the tools you desire. We are primarily a Java shop, but we have a large legacy code base of Visual Basic, C, ASP, and every other language known to man. We have a staff of five people and use Agile methodologies to plan and manage our projects.

We use CruiseControl to provide a continuous build process. Every time code changes are checked in, it executes our test suites, generates reports, and packages our software for distribution.

We use CVS as a source repository for all of our code.

We have a home grown release management application. It allows us to associate each code change with a specific product feature or defect and manages CVS tags for each release. Essentially, each file version gets a tag for the feature or defect and one for the release of the software where it will be applied. Our parent company dictates that we do two releases a year, but we actually generate "shippable" software once a month and our continuous build process allows us to ship on a weekly basis as needed. The software we use here is not ideal, but works for us for now. If anyone has any recommendations on software that does this more gracefully, I'm willing to listen.

We use a Wiki for all of our documentation. Our customer service, implementations, and sales staff use it as a reference point for all things associated with our products. It is quick, easy, very flexible, and a great way to capture information. We still generate traditional documentation and manuals for our major releases, but it all comes from the wiki.

We practice Test Driven Development and use code coverage reports like EMMA as metrics for continuous improvement. We do well in the Java world with this but have not found good ways of doing this for our legacy code base as of yet.

Over the last two years we have used a Scrum-like approach to project management. We use weekly sprints and work directly with a product manager, customer service manager, project implementation manager, and quality control manager to establish our workload and project priorities. From your description, it sounds like Agile methodologies would work well for you. They allow you to focus on the most important things to get done and give you the flexibility to change priorities on the fly. When we create our user stories and development tasks, we establish acceptance criteria up front. When our developers implement anything, they are not allowed to pass the work to our QA person without it meeting all the criteria. This makes QA more of a "add a done checkmark" process and allows them to focus on issues like regression testing (also automated) and managing newly found, customer centric defects.

For a long time, we tracked our weekly sprints on index cards and had a central project board where the latest information was stored. We recently switched to using XPlanner to create better visibility outside our development group and to facilitate our occasional telecommuting habits.

Our corporate overseers have dictated the we are a Microsoft shop, but as we were an acquisition, we haven't been tossed into the pool yet. They have recently started taking a closer look at how we are doing what we do because we have lower defect rates, faster time-to-fix rates, and faster release schedules than most of the rest of the company and we are doing it with a smaller staff. This works very well for us. The company is looking to deploy MS Team Foundation Server, which might be a good fit for you if you want to through a bunch of money at this. In my mind, it is overkill for a small team. One thing you'll notice is that all the software we have deployed is OSS.

Feel free to MeMail me if you have specific questions about what we use or how we do things.
posted by boba at 2:46 PM on March 22, 2008 [3 favorites]

2nding the suggestion of setting up a wiki. I've been on two projects in exactly this posture (small + single points of failure => medium-size + fewer irreplaceable experts) and in both cases we set up a wiki and began to track that kind of domain-specific and project-specific knowledge. The very general rule of thumb was that if it had to be explained by a "guy" rather than figured out from poking around, it ought to be documented. This served twin purposes: as a spin-up guide, and also as an "omg the foo is crashing and Fred is in Cancun all week" safety net.

I also highly recommend Subversion. It's very simple to set up, figure out, and use. It also has a great free online guidebook. My current project is using Trac, which is an opensource wiki and ticket tracking system which integrates very nicely with Subversion. You don't mention bug tracking/ticket tracking — if you're not doing that either, setting up Trac + Subversion is three easy birds with one stone (okay two). MediaWiki is a much more powerful/spiffy wiki but doesn't have the easy integration w/ source control and ticket tracking that Trac does. If you're going to stick with SourceSafe then I'd totally recommend MediaWiki.

Along with Joel and the others, I also highly recommend automating your builds. Visual Studio has commandline switches (or at least I know msvc++ does) which will non-interactively build a solution using the IDE. Sorry I can't remember the details. Nightly builds and the stigma associated with breaking them are a great gentle but constant pressure to trend quality upwards. So you can cron that or however Windows does task scheduling (can't remember).

Finally, unit-test driven development is a very paradigmatic pill to swallow but it is life-alteringly awesome. Look into it.

Cheers and good luck!
posted by mindsound at 2:59 PM on March 22, 2008

Oh yeah, forgot: some people recommend using nmake.exe or GNU Make to automate Visual Studio builds. FWIW, we struggled off and on for months to get a Make setup working with our small Visual Studio project (it was a Windows port of a unix lib that we didn't have many man-hours to devote to maintaining) before giving up and just building with the IDE. The problem for us was less that it actually wasn't possible and more that it seemed as though we were the only ones on the intertubes trying to transition a fancy Visual Studio build to makefiles, and couldn't figure it out from scratch due to that lack of googleability.

Visual Studio is really designed to build from the IDE and going against that cost us nontrivial time.
posted by mindsound at 3:05 PM on March 22, 2008

If you're using Visual Studio 2005 or 2008, automated builds are built in. Don't worry about nmake or make or even nant unless you really want to. Your csproj files are already in msbuild format.

At my shop, our products are a series of pretty popular newsmagazines and other informational sites. We're currently running 3 sites on the same platform and preparing to bring others online. As you might imagine, we have a lot of build issues and a lot of collaboration issues.

We've managed to get to a smooth running ship by doing the following:

We use SVN. By default, everyone gets TortoiseSVN, but those who have less sophisticated needs (generally UI or front end people) or anyone who is interested is encouraged to use VisualSVN (a fantastic plugin that gives you tortoise's features in VS's solution explorer). We follow a fairly traditional trunk/branch model for development. We tag every revision that makes it to production. This lets us branch off the tags for emergency hot fixes and then easily merge those changes back to the appropriate location without disrupting ongoing development.

We have well-delineated environments. In addition to production and our local machines, we have a development environment, an integration/QA environment, and a staging environment. Each runs in isolated VMs in their own windows domain. You have some leeway here, but if you go the VM route, you should be able to do pretty much anything.

We use CCNET. Every project we work on has 3 major types of builds: Unit, nightly, and package. Unit builds kick off every time someone commits some code. Nightly builds kick off at midnight and they integrate stuff that might take too long to run in the unit build, such as databases. This build also generally releases the results into our dev environment. When you want to release a build into the integration/qa environment, you run a package build. This creates a zip file that contains a couple of directories and bat files like "install-qaint.bat" or "install-stage.bat." Copy the zip, extract it, and run these in the correct environment and you upgrade that environment. We also do an archive step here so you can roll back via revert.bat. We have templates for these builds and are evaluating something like CI Factory to make them even more standardized.

This is just high level process though. It's only part of the equation. My guess is that your applications don't really have an overall architectural vision that will carry them through these complexity growth spurts and allow them to interoperate with each other. Once you tighten the process, start focusing on this next.
posted by feloniousmonk at 7:02 PM on March 22, 2008

Unit tests are a whole other matter. The vast majority of momentum is away from mstest, but some people do use it. I would recommend not sinking a lot of investment into this at this moment in time. Frameworks like xUnit and nUnit are where you should start. MbUnit can be thought of as a specialized version of nUnit (this isn't entirely true but it's close enough) and you might want to take a look at some of its more advanced features. A lot of them are incorporated into xUnit.

I use ReSharper's Unit Test Runner. They have a free version of it too. This is a fantastic way to integrate unit testing into your daily development routine.

You're going to be really way of TDD at first. This is a good thing. Don't fall for it hook, line, and sinker, but keep an open mind. The fundamental message is absolutely right, and to my mind this message is "do not program by coincidence." Sometimes for whatever reason you can't write tests first. This is okay. In the pursuit of code coverage, you may find yourself testing things that are either part of an already tested 3d party library, such as NHibernate, or even parts of the BCL itself. Avoid this. Only write tests when they test something useful.

If you're having a hard time imagining how you will structure your tests, particularly if you're trying to retrofit unit tests against untested legacy code, check out Working Effectively With Legacy Code for ideas on how to introduce the tests, and xUnit Test Patterns for overall structure.

The term xUnit is going to confuse you at first. For a long time it referred to the whole family of unit testing tools inspired by junit, including things like sqlunit, jsunit, nunit, cppunit, etc. However there is a more recently introduced framework called xUnit for .net. It can be hard to keep track of which one is which.
posted by feloniousmonk at 7:14 PM on March 22, 2008

Awesome advice, to all. I highly appreciate the feedback; looks like I have my work cut out for me. :)
posted by quixxotic at 11:11 AM on March 23, 2008

No problem. For some odd reason I love talking about this kind of stuff, so feel free to contact me via memail if you have further questions.
posted by feloniousmonk at 12:18 PM on March 23, 2008