Software Development Methodology
December 22, 2004 3:30 AM Subscribe
Software Development Methodology/Process. I am seeking advice/resources/experiences which could improve a small software company's ability to deliver quality software on time and on budget and the lives of the people who work there. [+]
I work for a company which has no defined development process in place. Requirements are scant and often oral, timelines are unrealistic, quality is low, testing sporadic, documentation rare, hours long and morale low. Some feel that it is a lost cause that we will simply continue firefighting until we are engulfed. I am more optomistic and believe that there is an opportunity to improve things which I would like to take. While not strictly within my remit I have been encouraged to look at 'quality'/'process' within the company. I have read about ISO 9000-3, CMM and DSDM and they all sound great but I feel I would benefit from some independent advice from experience on the matter. I don't exactly what question to ask. So I will just ask a few and trust that you can tell me some answers to those and other questions I should have asked.
What have other's experiences of similar situations?
Do you believe change is possible? How can it be achieved?
What methodologies have you see in practice in small (or any size) companies?
How does one translate the ideal of the theory to the reality of a small company finding it's way?
I would really appreciate the help/advice.
I work for a company which has no defined development process in place. Requirements are scant and often oral, timelines are unrealistic, quality is low, testing sporadic, documentation rare, hours long and morale low. Some feel that it is a lost cause that we will simply continue firefighting until we are engulfed. I am more optomistic and believe that there is an opportunity to improve things which I would like to take. While not strictly within my remit I have been encouraged to look at 'quality'/'process' within the company. I have read about ISO 9000-3, CMM and DSDM and they all sound great but I feel I would benefit from some independent advice from experience on the matter. I don't exactly what question to ask. So I will just ask a few and trust that you can tell me some answers to those and other questions I should have asked.
What have other's experiences of similar situations?
Do you believe change is possible? How can it be achieved?
What methodologies have you see in practice in small (or any size) companies?
How does one translate the ideal of the theory to the reality of a small company finding it's way?
I would really appreciate the help/advice.
Echoing LairBob a little, I wouldn't try and adopt one of the big methodologies wholesale. In my (admittedly cynical) view no number of acronym-driven manuals for doing development stop stupid people from making mistakes.
My suggestion would be instead of overhauling the whole thing from the top down, just start picking off problem areas one-by-one. Dodgy config management? Spend a week writing some build scripts and start using source control. Badly documented requirements? Get people to write them down in an e-mail when they ask for them or agree to them. Low quality? Put some simple regression test suites in place. Etc. etc. No step on its own will solve the entire problem, but each one will help a little and start to build morale.
For general improvement of morale, get the team to deliver something successfully. Start with something small and let them control scope and timescales, and then do your damndest to make sure they succeed. They'll feel much more positive when they do.
I've worked on smallish teams that have been turned around on more than one occasion, and the simple quick wins are worth far more than a hundred-weight of books about DSDM or ISO 9001. (Not to say there's nothing useful in there).
Joel on software makes a lot of similar points more eloquently than I and might be worth mining for ideas.
posted by grahamspankee at 5:38 AM on December 22, 2004
My suggestion would be instead of overhauling the whole thing from the top down, just start picking off problem areas one-by-one. Dodgy config management? Spend a week writing some build scripts and start using source control. Badly documented requirements? Get people to write them down in an e-mail when they ask for them or agree to them. Low quality? Put some simple regression test suites in place. Etc. etc. No step on its own will solve the entire problem, but each one will help a little and start to build morale.
For general improvement of morale, get the team to deliver something successfully. Start with something small and let them control scope and timescales, and then do your damndest to make sure they succeed. They'll feel much more positive when they do.
I've worked on smallish teams that have been turned around on more than one occasion, and the simple quick wins are worth far more than a hundred-weight of books about DSDM or ISO 9001. (Not to say there's nothing useful in there).
Joel on software makes a lot of similar points more eloquently than I and might be worth mining for ideas.
posted by grahamspankee at 5:38 AM on December 22, 2004
Martin... Is that you?
We have the same sort of problems, and the honest answer is that there is no tried and tested way of changing things. Read the Joel on Software articles and then read both "The pragmatic Programmer" and everything by Steve McConnell.
My experience is that the people least likely to change are the ones who are spending the least time on the development process. For small companies, this is usually the boss, and it's this person who scrawls the requirements out onto a piece of paper and passes them onto you. You're not going to change this guy (or gal), but there are a few tricks.
1) Ensure that you know what small / cheap / useful tools are available and use them yourself. Secretly. I used Mantis for about 6 months before my boss figured out I was using it, and it was a great help.
2) Don't knock a paper trail. We've found that paper is a really useful tool. All changes / new development ends up on a sheet of paper that simply states that the work is to be done / is done. This formalises the lines of communication without either a big whoo-ha about "what the process should be" or the implied accusation that a system is needed because of poor communication on the part of others.
3) Trick people into discovering methods for themselves. Spoof emails from mailing lists containing links to JoelOnSoftware. When your boss asks you what you think, then "Resist Change". Finally, concede that it may be a good idea, and then "give it a go".
4) You're a small company, and in truth, you probably don't need anything major in the way of methodology. In your case, the management of the project should be like the code. Keep things simple, ensure that there are as few communication lines as possible, and don't try to fix bugs until (a) you really have to and (b) you know exactly what the bugs are.
posted by seanyboy at 5:55 AM on December 22, 2004
We have the same sort of problems, and the honest answer is that there is no tried and tested way of changing things. Read the Joel on Software articles and then read both "The pragmatic Programmer" and everything by Steve McConnell.
My experience is that the people least likely to change are the ones who are spending the least time on the development process. For small companies, this is usually the boss, and it's this person who scrawls the requirements out onto a piece of paper and passes them onto you. You're not going to change this guy (or gal), but there are a few tricks.
1) Ensure that you know what small / cheap / useful tools are available and use them yourself. Secretly. I used Mantis for about 6 months before my boss figured out I was using it, and it was a great help.
2) Don't knock a paper trail. We've found that paper is a really useful tool. All changes / new development ends up on a sheet of paper that simply states that the work is to be done / is done. This formalises the lines of communication without either a big whoo-ha about "what the process should be" or the implied accusation that a system is needed because of poor communication on the part of others.
3) Trick people into discovering methods for themselves. Spoof emails from mailing lists containing links to JoelOnSoftware. When your boss asks you what you think, then "Resist Change". Finally, concede that it may be a good idea, and then "give it a go".
4) You're a small company, and in truth, you probably don't need anything major in the way of methodology. In your case, the management of the project should be like the code. Keep things simple, ensure that there are as few communication lines as possible, and don't try to fix bugs until (a) you really have to and (b) you know exactly what the bugs are.
posted by seanyboy at 5:55 AM on December 22, 2004
Response by poster: Seany I am not martin if it's me you are asking.
Funny that you should show up in this thread though. I was just looking at your profile due to your last FPP..
Guys all sounds like good advice. I think I will spend some of the Xmas thinking of small things we can do. We have already listed 27 items which need to be adressed, but that was 2 months ago and we have only really made inroads into three items from the list. Maybe with the right attitude, some simple ideas and New Year Resolution we could knock a few more off.
posted by kenaman at 6:04 AM on December 22, 2004
Funny that you should show up in this thread though. I was just looking at your profile due to your last FPP..
Guys all sounds like good advice. I think I will spend some of the Xmas thinking of small things we can do. We have already listed 27 items which need to be adressed, but that was 2 months ago and we have only really made inroads into three items from the list. Maybe with the right attitude, some simple ideas and New Year Resolution we could knock a few more off.
posted by kenaman at 6:04 AM on December 22, 2004
One other broad set of resources you might look into is "organizational change"--some of the Joel on Software stuff is good on this front, but a lot of what's written in that kind of forum is written by programmers. (Not always the best judges of human character, and how to influence it..."This is better, so do it." "No." "Do it." "No." "Idiot. I'm outa here.").
Googling "organizational change" or "change management" will bring up a ton of corporate-speak dreck, but there are almost certainly some good insights to be found online. I've found that there's a decent chunk of intelligent, well-minded people who have focused on this broader issue...how to get corporate cultures to change successfully, whether it's around development methodology, or any other business process. Wouldn't help to gather some pointers from someone who seems credible.
posted by LairBob at 6:47 AM on December 22, 2004
Googling "organizational change" or "change management" will bring up a ton of corporate-speak dreck, but there are almost certainly some good insights to be found online. I've found that there's a decent chunk of intelligent, well-minded people who have focused on this broader issue...how to get corporate cultures to change successfully, whether it's around development methodology, or any other business process. Wouldn't help to gather some pointers from someone who seems credible.
posted by LairBob at 6:47 AM on December 22, 2004
Response by poster: Cheers Bob.
Changing my google threw up a short pdf book describing and comparing agile software development methods, which should make for interesting reading. I take your point that it is not for want of a methodology that things are the way they are. The culture has to change and I have to identify and sell the benefits of these changes - oh yeah and then figure what the changes should be.
posted by kenaman at 7:10 AM on December 22, 2004
Changing my google threw up a short pdf book describing and comparing agile software development methods, which should make for interesting reading. I take your point that it is not for want of a methodology that things are the way they are. The culture has to change and I have to identify and sell the benefits of these changes - oh yeah and then figure what the changes should be.
posted by kenaman at 7:10 AM on December 22, 2004
This is a very big topic. Aside from echoing some of the other points (skip ISO, read McConnell), here are some pointers:
- If you can, find a champion on the business side. A technology evangelist from there is worth 10 from IT.
- Use cases. Yeah, they suck, but they help so much. Work with your clients (either internal or external) to "script out" scenarios using the software they want developed. It forces people to think concretely about software requrements, rather than in vague "I wish I had something to fix this problem" terms. It also, invariably, leads to clients identifying process issues on their end that need to be addressed- if you drop a piece of software into a broken process, you're almost always going to just make it break faster instead of fix it. Which leads to...
- Managing expectations. Learn to say "No." But say it in a way that leads the client to think they made the decision themselves. For example: "I can add this whiz-bang feature to this page, but it will add X days and $Y to the project, which makes me less comfortable about delivering this to you on time." Bring your client back to the Use Cases- does what they're asking for address something specific in any Use Case?
posted by mkultra at 7:32 AM on December 22, 2004
- If you can, find a champion on the business side. A technology evangelist from there is worth 10 from IT.
- Use cases. Yeah, they suck, but they help so much. Work with your clients (either internal or external) to "script out" scenarios using the software they want developed. It forces people to think concretely about software requrements, rather than in vague "I wish I had something to fix this problem" terms. It also, invariably, leads to clients identifying process issues on their end that need to be addressed- if you drop a piece of software into a broken process, you're almost always going to just make it break faster instead of fix it. Which leads to...
- Managing expectations. Learn to say "No." But say it in a way that leads the client to think they made the decision themselves. For example: "I can add this whiz-bang feature to this page, but it will add X days and $Y to the project, which makes me less comfortable about delivering this to you on time." Bring your client back to the Use Cases- does what they're asking for address something specific in any Use Case?
posted by mkultra at 7:32 AM on December 22, 2004
there were a few other book suggestions here - i am going through the same thing although i am actually changing roles in order to hopefully fix some of the problems.
posted by GrumpyMonkey at 8:15 AM on December 22, 2004
posted by GrumpyMonkey at 8:15 AM on December 22, 2004
Kenaman, I'm guessing by the tone of your post that your company is small to medium sized and not a huge megalith that has lots of organizational inertia. With that in mind...
I'd recommend a light-weight, agile method called Scrum. I've had success using many of the Scrum techniques in scenarios where there was absolutely no methodology to scenarios where there is an established, heavy-weight methodology (eg. Macroscope).
Agile methods article at Wikipedia.
Now, regardless which toolkit you select or pieces you assemble, the thing to keep in mind is that methodologies don't solve problems. People solve problems. Without establishing a corporate culture that is willing to adapt to the new methodology, then nothing you attempt to implement will actually work.
You need to convince the business that operations using well understood internal communication standards (etc...) will provide tangible business benefits (ie. increased worker productivity, better quality product, etc...).
LairBob gives excellent advice with the whole Change Management thing. Don't let this slip your mind.
posted by C.Batt at 8:34 AM on December 22, 2004
I'd recommend a light-weight, agile method called Scrum. I've had success using many of the Scrum techniques in scenarios where there was absolutely no methodology to scenarios where there is an established, heavy-weight methodology (eg. Macroscope).
Agile methods article at Wikipedia.
Now, regardless which toolkit you select or pieces you assemble, the thing to keep in mind is that methodologies don't solve problems. People solve problems. Without establishing a corporate culture that is willing to adapt to the new methodology, then nothing you attempt to implement will actually work.
You need to convince the business that operations using well understood internal communication standards (etc...) will provide tangible business benefits (ie. increased worker productivity, better quality product, etc...).
LairBob gives excellent advice with the whole Change Management thing. Don't let this slip your mind.
posted by C.Batt at 8:34 AM on December 22, 2004
mkultra hits the nail on the head about dealing with changing requirements. If you are always sure to talk about changes with your customers in terms of financial and schedule costs you will find yourself being jerked around a lot less.
As with changes to your team, be sure to take small steps. A automated build system, source code control, bug tracking, unit testing need to be introduced slowly to the group. Introduce too much change too fast and you will get the cold shoulder treatment. My personal preference is to introduce bu g tracking and automated build early. Everyone already understands bug tracking so it shouldn't be a hard sell. Just use something simple like Mantis. Automated builds are another easy one because generally speaking, it only needs to be built once and the other people just need to use it. If it is written right, you aren't asking a whole lot of your team mates.
Good luck with source code control and unit testing. While most people understand the value of SCC, in practice it can be hard to get people to be good SCC citizens. They will forget to add new files to the repository. They will leave files checked out for weeks or months at a time. They will not make frequent checkins. They will check in broken code. They will not make meaniful comments when they check code in and so on. I don't think I need to go into the struggles that unit testing will bring.
Good luck and please find a way to report back to the group or at least me (e-mail in profile) about how much progress you have made.
posted by mmascolino at 8:47 AM on December 22, 2004
As with changes to your team, be sure to take small steps. A automated build system, source code control, bug tracking, unit testing need to be introduced slowly to the group. Introduce too much change too fast and you will get the cold shoulder treatment. My personal preference is to introduce bu g tracking and automated build early. Everyone already understands bug tracking so it shouldn't be a hard sell. Just use something simple like Mantis. Automated builds are another easy one because generally speaking, it only needs to be built once and the other people just need to use it. If it is written right, you aren't asking a whole lot of your team mates.
Good luck with source code control and unit testing. While most people understand the value of SCC, in practice it can be hard to get people to be good SCC citizens. They will forget to add new files to the repository. They will leave files checked out for weeks or months at a time. They will not make frequent checkins. They will check in broken code. They will not make meaniful comments when they check code in and so on. I don't think I need to go into the struggles that unit testing will bring.
Good luck and please find a way to report back to the group or at least me (e-mail in profile) about how much progress you have made.
posted by mmascolino at 8:47 AM on December 22, 2004
The company I work for uses elements of XP, and although our release cycle is longer than XP wants, it works well for us. You can add parts piecemeal if you like. A lot of people won't go for some of the more radical things right away -- such as pair programming -- but it's easier once you've had success with a few techniques.
posted by kindall at 8:54 AM on December 22, 2004
posted by kindall at 8:54 AM on December 22, 2004
something i don't think others have said - starting writing job applications. this kind of change is impossible to drive effectively from the middle. it has to come from the top down. if you're working long hours in bad conditions with no processes it's not going to get better, whatever you try. get out as soon as you can.
posted by andrew cooke at 9:09 AM on December 22, 2004
posted by andrew cooke at 9:09 AM on December 22, 2004
Response by poster: mmascolino - I will indeed let you know how it goes..and yes C.Batt our company is small - six developers, a trainer, support, sales and the Boss. There is not really a senior developer and the boss is so busy with other aspects of the company that he does not have time to get involved in improving the development process........so the problem which I described in the initial post has emerged.
Amyway I must say to all of you that I really appreciate the help advice and support.
It is fantastic to be able to get no strings attached unbiased expertise.
on preview - andrew I have done an interview recently but missed out. This is my second I.T. job and the last was with an even more dysfunctional small company. I am determine to try make a go of it here before moving on and do feel that the boss recognises that change is required in order to survive. But I have had moments when I just wanted to walk......
posted by kenaman at 9:16 AM on December 22, 2004
Amyway I must say to all of you that I really appreciate the help advice and support.
It is fantastic to be able to get no strings attached unbiased expertise.
on preview - andrew I have done an interview recently but missed out. This is my second I.T. job and the last was with an even more dysfunctional small company. I am determine to try make a go of it here before moving on and do feel that the boss recognises that change is required in order to survive. But I have had moments when I just wanted to walk......
posted by kenaman at 9:16 AM on December 22, 2004
LairBob is totally right about how difficult it is to get things to change overnight. One thing that can give you some insight is this book: The Career Programmer: Guerrilla Tactics for an Imperfect World. It describes how you can start to make process change happen in a company culture that is resistant to change.
Also anything by Joel on Software, as mentioned above, including his new book, will be very informative (you can also read the original columns that make up the bulk of the book here.
Second the recommendations above for anything by McConnell as well as The Pragmatic Programmer.
It is especially tough when the company is so small that there is not a director of development and the programmers have to report directly to a non-tech boss - that person's interests are often antithetical to your own. I wish you the best of luck - I've sure been there (and the way I dealt with it was to leave).
posted by matildaben at 9:55 AM on December 22, 2004
Also anything by Joel on Software, as mentioned above, including his new book, will be very informative (you can also read the original columns that make up the bulk of the book here.
Second the recommendations above for anything by McConnell as well as The Pragmatic Programmer.
It is especially tough when the company is so small that there is not a director of development and the programmers have to report directly to a non-tech boss - that person's interests are often antithetical to your own. I wish you the best of luck - I've sure been there (and the way I dealt with it was to leave).
posted by matildaben at 9:55 AM on December 22, 2004
good luck - hope that wasn't too negative; i think the other comments are good too and have just forwarded matildaben's book link to a bunch of people.
posted by andrew cooke at 10:32 AM on December 22, 2004
posted by andrew cooke at 10:32 AM on December 22, 2004
A recent question on requirements writing has some answers that I think you'll find useful. I note that it included my plug for reading Joel on Software, so I'm seconding others in this discussion who have made the same recommendation.
posted by WestCoaster at 12:24 PM on December 22, 2004
posted by WestCoaster at 12:24 PM on December 22, 2004
Hire a good project manager.
It's REALLY hard to feel your way through these kinds of organizational problems without having someone experienced do it for you the first few times.
posted by Caviar at 12:38 PM on December 22, 2004
It's REALLY hard to feel your way through these kinds of organizational problems without having someone experienced do it for you the first few times.
posted by Caviar at 12:38 PM on December 22, 2004
One technique that will contribute a great deal to software quality is covering as much code with unit tests as possible.
Given a functional unit of code, write as many unit tests as there are possible ways you think you'll use the code in the system you're building. Every time you change the code, run the tests as a sanity check. Every time you find a bug, write a test that exploits the bug before fixing the bug. When the bug is fixed, the test should pass. Every time you add a feature, write the test first, then add the feature.
This is something that individual coders can agree to do, and management cannot interfere, and will end up making a major difference in your confidence in the system as a whole.
If you're coding in C++, Java, or C#, I can recommend XUnit/JUnit/NUnit (all, essentially, the same unit testing framework). It is trivial to use, and in addition to testing, provides a great entry point for debugging during deelopment. Perl has its own testing tramework. Use it.
Thorough unit testing makes integration testing easier, because you have some built-in confidence that the functional units are behaving as they ought to.
posted by jimfl at 6:52 PM on December 22, 2004
Given a functional unit of code, write as many unit tests as there are possible ways you think you'll use the code in the system you're building. Every time you change the code, run the tests as a sanity check. Every time you find a bug, write a test that exploits the bug before fixing the bug. When the bug is fixed, the test should pass. Every time you add a feature, write the test first, then add the feature.
This is something that individual coders can agree to do, and management cannot interfere, and will end up making a major difference in your confidence in the system as a whole.
If you're coding in C++, Java, or C#, I can recommend XUnit/JUnit/NUnit (all, essentially, the same unit testing framework). It is trivial to use, and in addition to testing, provides a great entry point for debugging during deelopment. Perl has its own testing tramework. Use it.
Thorough unit testing makes integration testing easier, because you have some built-in confidence that the functional units are behaving as they ought to.
posted by jimfl at 6:52 PM on December 22, 2004
I might be too late to respond, but I wanted to pick up on the mention of agile project management.
While taking an agile approach to running your projects means you can be more flexible in your deliverables (i.e. changes to client requirements can be reflected in the software more easily by using iterative development cycles), it does require the complete buy-in of all those involved.
Using an iterative/agile method is no good if you can't get your client to agree that they need to be actively involved in specifying/clarifying their requirements at each stage. Neither would it work if your developers base their work on their original understanding of the client's requirements rather than the additional clarity that is reached during the project's lifecycle.
Hope this makes sense.
posted by Lleyam at 4:15 AM on December 24, 2004
While taking an agile approach to running your projects means you can be more flexible in your deliverables (i.e. changes to client requirements can be reflected in the software more easily by using iterative development cycles), it does require the complete buy-in of all those involved.
Using an iterative/agile method is no good if you can't get your client to agree that they need to be actively involved in specifying/clarifying their requirements at each stage. Neither would it work if your developers base their work on their original understanding of the client's requirements rather than the additional clarity that is reached during the project's lifecycle.
Hope this makes sense.
posted by Lleyam at 4:15 AM on December 24, 2004
This thread is closed to new comments.
The odds of a company culture changing successfully in one big step from the one you describe, to one based on DSDM or any other rigorous and complete methodology is just about nil, in my experience. Organizations have to be taught how to change--how to appreciate the value, how sort the good options from the bad, and how to commit to real change. Organizations also have to be taught to value discipline--not in the messed-up, corporate-fascist kind of way, but they have to learn how better specs make things better. They have to learn how QA delivers better, more reliable results.
Put simply, people change their behavior because they see some real benefit in changing it. I was working with a big company once on an e-business process engagement, helping a company similarly behind the curve plan out this totally cool future, and a guy looked at me and said, "This is great, but you're trying to sell me on running shoes while I'm sitting here with two broken legs. Even if this stuff was in place, I'm so far from taking advantage of it that it doesn't really matter. Help me get a reliable connection from my field office. Help me reduce the amount of paperwork I have to do for each sale. Then I'll care about this other neat-o stuff."
Start small, by helping people identify the major obstacles, and knock off the big problems one by one. Ground your approach in real, practical issues. ("Four out of the past five projects were delivered late. Let's identify the top three reasons why that keeps happening, and what we can do to at least lessen them.") For each problem, try to make sure you're helping people understand why they care about that change, and how it's going to help them on some level. Also, make sure you've got some platform, after the fact, to promote each success--whether it's some kind of simple metric ("projects completed on time this quarter"), or even just a chance to formally present/discuss results with management or your peers.
Once you've got a few moderate successes under your belt, then you can start thinking of a more systemic approach. (And for a small shop, I'd really have a hard time imagining how one of the all-encompassing methodologies you listed would really work well. They've all got good pointers that you can pick, and try to apply in a custom manner, so it wouldn't hurt you to get more familiar with them, but I'd be careful about getting your whole organization to adopt one. The best solutions I've seen have all clearly been an expression of the company's specific needs, priorities and ways of working.)
posted by LairBob at 5:05 AM on December 22, 2004