Good citation re engineering performance metrics.

November 6, 2015 7:35 PM

Please help me find good, preferably publicly available, citations regarding eg # of commits as lousy performance metrics.

A related tangent to previous questions.

I'm looking for something with more umph than stack overflow answers to show to HR that the accepted wisdom is that eg # of commits or pull requests is a useless metric of engineering performance.

There's at least one good Martin Flowler blog post, but it's not particularly concise, and doesn't speak specifically to # of commits.

I really want to help HR understand why these kinds of metrics are counterproductive, and I'd like some authoritative citations.

Thanks!

posted by colin_l to Work & Money (8 answers total)

The book Making Software: What Really Works, and Why We Believe It has some good research about whether certain metrics are meaningful. Following bibliographies from those should lead you to several others. Offhand, I don't recall anything specifically about commits, but if there is it will likely be in that web of research.

I also have this specific paper, An Empirical Evaluation of Object Oriented Metrics in Industrial Setting, bookmarked -- it makes the case that most metrics related to OOP aren't really more significant indicators than simple metrics like lines of code.
posted by silentbicycle at 7:59 PM on November 6, 2015

A well-constructed metaphor should suffice, i'd think.

You have two mechanics that you go to when your car breaks down. Mechanic A always gets the job done quickly, but 50% of the time, something breaks later and you end up having to bring the car back for further repairs, and half of the time THOSE fixes need fixes, so it ends up taking a long time to actually get a 100% working car. Mechanic B takes twice as long to do the initial fix as Mechanic A, but 95% of the time your car is in perfect working order when you get it back from him.

Something like that.
posted by deathpanels at 9:11 PM on November 6, 2015

There are some fabulously pithy quotes about this: Djikstra (as in the most famous computer scientist on earth) once said "If we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent"". Bill Gates (the richest) once said "measuring progress on software by amount of code produced is like measuring progress on an airplane by weight".

But maybe an object lesson is in order: spend a week submitting every single line of code you write as a separate pull request and see if they change their tune.

Though honestly, this doesn't sound like a problem you can fix with any number of good articles. Sometimes a place or a process is just broken. The software engineer approach is always gonna be to try to fix them, but sometimes you just gotta vote with your feet and go work someplace that actually knows how to develop software effectively. Let them collapse under the weight of their own bad incentives.
posted by Itaxpica at 9:21 PM on November 6, 2015 [1 favorite]

I actually went hunting for this a few months ago when I got dinged on my performance review for not submitting enough code. It is surprisingly difficult to find hard data, probably in part because there really isn't any better way of measuring the performance of a software engineer. The real way to know how well a software engineer is doing is to be a lead who's very involved with the codebase, design processes, and planning, but once a manager has about ten or more direct reports this usually becomes too much to keep in one's head so they have to come up with proxies. Lines of code and number of commits are proxies that feel right to a lot of people, probably mostly because they're numbers, and numbers are easy to compare with other numbers.

After grumbling to myself about how underappreciated I am for awhile, I eventually just did a mindset shift and said, well, if I commit something I'm not sure about I get even MORE lines of code when I realize it's broken and go in and fix it. I doubt it's what my manager actually wants, and if he gave it more than fifteen seconds of thought maybe he'd reach the same conclusion, but as it stands he is happier with me and I'm happy to have him off my back. I'm sure there's some pithy quote someone can post about getting what you incentivize.

I'm not in a position to look for new opportunities right now for personal reasons, and other than the goofy-ass performance metrics I'm fine with where I am. But you can bet I've got one passive eye open for better options, because once I realized people were just optimizing for LOC sooooo many things about our codebase fell into place for me. When everyone is just angling to check shit in as fast as possible there's no atmosphere of stewardship. Our code reviews are superficial and not educational, and lots of classes lurch along with bits and pieces of cruft hanging off them. I do my best to patch things up as I go but if I have the choice between a cleanup refactor that might go awry if I don't spend a little time and effort on it and two nasty hack bugfixes that I know I'll be able to get past a reviewer today or tomorrow, that refactor gets back burnered so fast it'd make your head spin.
posted by town of cats at 9:32 PM on November 6, 2015

But how many lines of code would that refactor touch? INSTAWIN!
posted by russm at 11:06 PM on November 6, 2015

there really isn't any better way of measuring the performance of a software engineer

I completely disagree. Measure completion of the actual goals. Presumably your company's goal is not "have more lines of code than anyone else" (well, maybe it is if you're an outsourced dev shop...). You have features you're building, bugs you're fixing, and products you're shipping. If you must boil down software engineer productivity to a set of numbers (a practice I generally disagree with, but it's not a battle that can be won at all companies), use things like user stories, story points, cycle time, etc. There are many, many, MANY problems with measuring on those metrics (and I'm too lazy and tired to type out the whole diatribe right now), but they're infinitely better than measuring on lines of code. At least they are somewhat related toward the actual goal of the business.
posted by primethyme at 11:23 PM on November 6, 2015

A comic strip may be enough if they understand; minivan
posted by stuartmm at 2:50 AM on November 7, 2015

A coworker once said to me, "I'd give you less code if I had more time."
posted by ph00dz at 7:25 PM on November 7, 2015 [1 favorite]