Fighting the corporate stat monster

July 23, 2010 10:01 AM

I work in a technical company where our ability to deliver fixes for technical problems is measured by a bunch of statistics. Our team seems to lost track of our real goals (solving problems) and instead cares more about improving the stats even if it means we don't solve problems as well as we could. How do I get everyone to understand the real needs of the job and to use stats as a supporting tool rather than the end goal?

So we have a metric, TFT (Total Fix Time), which measures the time till a fix is delivered. We have a goal of keeping this above a certain level.

Last week in a meeting one of my colleagues announced that we could improve this metric by closing the case, and thus generating the TFT, when the case was passed on the development team. The only problem being we would have to devise a seperate method of then tracking the actual fix being created and delivered outside of our case tracking system. This would improve the TFT.

My response was this was a very bad idea. It would complicate the system making it harder to track and get fixes to the customer. The only advantage would be an improved TFT but customer service would be worse.

I explained that if an alternative system was better providing quicker fixes then fair enough but it was wrong, and indicative of the mindset we shouldn't have, if the only reason to change the system was to improve the TFT regardless of how fast the actual fix was delivered to the customer.

I find it kind of depressing. The important thing is we deliver a good service and we keep trying to improve. To help us do that we measure some metrics. However, many people in the company lose focus and see improving the metric as the goal rather than improving the service.

One argument of my colleague is that we often have to provide reports to management, internally and at our customer, and we can win or lose contracts based on these statistics. My response was while that may be a minor consideration it pales in to insignificance compared to the actual quality of service we deliver and we should never accept worsening service to improve a statistic.

It's worrying because the low level management I deal with seem to favour the improve TFT in any way possible because that is what they are judged on. I'm sure higher up they care about the service delivered but I'm stuck down here waging a one man battle.

So am I right? Or have I lost perspective here? It's classic 'the map is not the territory' stuff and a basic business problem, right?

Can anyone tell whether you think I am right or wrong ? If I am right what is the best way of attempting to correct this mindset in the company? Are there any business books that discuss this issue well? Have you ever come across this in your own business?

posted by aTrumpetandaDream to work & money (17 answers total)

The metric is supposed to be an indicator of your success. If it isn't, the metric shouldn't be relevant. I've run into a facsimile of this problem before, and the answer turned out to be having "customer" satisfaction surveys be a part of the final metric. That way there was a numbers reason to keep the customer on-board and actually solve the problem, while still doing it in a reasonable amount of time.

If you aren't in control of the metrics, then you can't expect to resolve the situation yourself. If might be a good idea to tell lower management that you'd like to share some ideas with the management as a whole, and that you are deeply concerned with the current state of the metric and its results. If this gets rebuffed, your leaders are against you and the situation isn't likely to get better, but at least you can put in the effort and get it off your chest to everyone, make your perspective known, and maybe even make a few positive impressions to anyone who's listening up the chain.

I'd say sit down, write a very specific email that thoroughly discusses the problem from your perspective, and send that to the management you routinely interact with. See what happens from there. Worst case, you'll discover you're in an environment that doesn't value the customer.
posted by Phyltre at 10:11 AM on July 23, 2010 [2 favorites]

If you the time, put together a presentation with real numbers. If we implement "Z", we will save "Y" dollars.
I had an idea for my company that kept getting neglected and pushed down the priority list. I ended up making a stat sheet showing the VP the potential savings. He approved the changes that day.
When in doubt, PowerPoint.
posted by WhiteWhale at 10:15 AM on July 23, 2010

My response was this was a very bad idea. It would complicate the system making it harder to track and get fixes to the customer. The only advantage would be an improved TFT but customer service would be worse.
[...]
It's worrying because the low level management I deal with seem to favour the improve TFT in any way possible because that is what they are judged on. I'm sure higher up they care about the service delivered but I'm stuck down here waging a one man battle.
[...]
If I am right what is the best way of attempting to correct this mindset in the company?

A buddy of mine once worked at a company supplying ISP telephone technical support. When a customer called up with a connection problem he would conscientiously work with them to diagnose and fix the problem - always aiming for a permanent fix, not just to get the customer off the line. Sometimes this took a long time. He would send service techs to check the lines if the problem seemed to lie there - or send replacement hardware if it seemed to be a problem with the ISP-supplied hardware.

He had a co-worker who would tell the customer to turn their modem off and back on again, regardless of what the problem was, and ask them to call back when it came back on. Guess which employee got bonuses for handling lots of calls, and being frugal with service visits and replacement hardware?

I think the best thing would be to adopt a better metric, so that people are judged on what management want them to do.
posted by Mike1024 at 10:25 AM on July 23, 2010

Thanks for asking this. I'm currently in a department where the process is much, much more important than the results, so I'm interested in reading some other responses.
posted by emelenjr at 10:26 AM on July 23, 2010

Yup, it has a name in my company and the name is Wrong Metric Disease. The root cause of WMD is wishful thinking. You can't fix it without finding and changing the edict that converted wishful thinking into policy. To do that you need to show how the metric fails to support a higher level of policy *and* what metric would be more supportive of it. It is difficult to do.
posted by jet_silver at 10:31 AM on July 23, 2010 [2 favorites]

This needs to be a conversation with management above you.

It sounds like your metrics aren't properly aligned with the company's goals. If that's the case, new metrics need to be developed.

If performance is measured and bonuses are assigned based on a metric, people will do their jobs according to that metric. It's only human nature, and you can't enforce/train around that. It's that simple.
posted by swngnmonk at 10:32 AM on July 23, 2010 [1 favorite]

Yup, it has a name in my company and the name is Wrong Metric Disease.

Weapons of Metric Destruction? Are TFT's tied to job performance, reviews, and/or salary increases? If so, they should be untethered so that the employees aren't focused on better TFT = better review. What swngnmonk said.
posted by schleppo at 10:36 AM on July 23, 2010

Change the metrics.

People optimize what they can measure. Perhaps the better metric would be Average Support Time Per Customer Per Year. Quick fixes to get someone off of the line would only so far in those cases. Permanent fixes, though, solve the problem in question. Callbacks Per Customer might be another metric.

Additionally, if you have more than one metric, styles can develop. Bob gets people off the line fast. Alice is the one who fixes problems permanently, though, so she's got a better Average Support Time Per Customer Per Year. Suddenly, the idea develops, "Throw people at Bob first. If Bob can't fix the problem, send to Alice." Develop a metric for that one guy in the back who doesn't seem to do much and then he suddenly says, "Oh, yeah, I figured out the source of all of these 'random' problems. They're not so random. Check this out ..."

Change the yardstick, people begin behaving differently.
posted by adipocere at 10:37 AM on July 23, 2010 [1 favorite]

This is a very common problem. It's the same problem that schools face when they try to measure teacher performance by testing students. The teachers just end up teaching the test rather than making sure their students actually learn.

It sounds like your direct managers benefit from this arrangement, at least in the short term. So maybe they won't cooperate with you. You could put together a presentation, like WhiteWhale suggests, but what if your direct managers interfere or end up resenting you?

Maybe send a hard copy of that presentation to upper management anonymously? Or, if you're bold and want recognition, send it with a note that identifies you but requests that your association with the presentation is kept anonymous (initially, anyway). But I think staying truly anonymous gives your ideas more credibility during the initial stages. And if your company has a decent HR department, also send them a copy.
posted by mullacc at 10:43 AM on July 23, 2010 [1 favorite]

IANAStatistician. But as someone who has been frustrated with customer service, it seems like these metrics should be used to educate and help customers manage their expectations, rather than as a demonstration of the fallacious notion that the best solution to a problem is the fastest one. Could you propose that certain TFT's indicate that a problem will take a certain amount of time to fix? In cases of repair, I know that I am much more patient for solutions that take a while if I am kept informed about what the fix entails, and whether or not my wait time is out of the ordinary.
posted by pickypicky at 10:48 AM on July 23, 2010

Point out to them that the metric is called TOTAL Fix Time. If you're not measuring the time before the problem's fixed then you might as well call it PFT (Partial Fix Time).

Sounds to me like you might want more fine grained metrics. Maybe Helpdesk Fix Time and Developer Fix Time such that TFT = HFT + DFT. Then the helpdesk doesn't get penalized by management when the developers are the ones dragging their feet.
posted by sbutler at 10:51 AM on July 23, 2010 [1 favorite]

Yes, it is tied to bonus.

I'm not too bothered about that. It is important to measure and it does keep us focussed on getting a fix out as quickly as possible.

But to be honest it's also really easy to manipulate. I'm not helpdesk I'm a TPM (Technical Project Manager) for several large accounts. So I tend to manage the technical side of how customers use our technology. So if my TFT is below what it needs to get the top bonus for that I just phone them up have a chat about how things are going and lead them to talk about some simple technical issues. Then I log distinct calls for each of those and solve them instantly. That increase smy average TFT massively. Nobody in management would be technical enough to understand what is going on there.

I know that's kind of sleazy and is kind of against what I am saying here but it doesn't actually detract from customer service... and to be honest I'm not a rich man and I need the bonus which can be thousands... I still wouldn't do anything to improve the TFT at the expense of delivering good service though. That's a whole different issue.
posted by aTrumpetandaDream at 11:05 AM on July 23, 2010

You asked about books. "The Black Swan..." by Nassim Nicholas Taleb discusses the limitations of mathematics and models. Probably the best one for your purposes. Topical (the market meltdown) and a very easy read. Some notable quotes on the inappropriate use of math and the false comfort it gives.

You could also check out "Apollo's Arrow" by David Orrell but that has more of a "limitations of math and models in prediction" flavor.

Personally, I think the business world got severely damaged by the utter nonsense that U.S. business schools preach. "If you can't measure it, you can't manage it" is one of their canards. A corollary is that if you can manage a shoe company, you can manage a software company. Why? Because all you need is the numbers and the right spreadsheet. It is very hard to quantify vision, so let's not bother to measure that.

There is the old joke about the drunk looking for his car keys under a lamplight. Some passersby stops to help him and asks "Where did you lose them?". The drunk points off into the darkness. "Then why are you looking here?" the passerby asks. "Because the light is better". Sound like your management?
posted by PickeringPete at 11:21 AM on July 23, 2010

Metrics are everything, people want their bonuses, helping the customer is secondary in this case. I think that you definitely should add a customer satisfaction survey as a metric, but of course this costs money so you will need to make a business case for this to your bosses.

The key pitch here is corporate image, an example would be Dell - at one point they were #1 in sales and customer service (and had a fantastic image), then they outsourced their customer service to Absurdistan and look at them now (that isn't the whole story, but you don't need to tell the whole story).

I believe that your goal to improve customer service is a good one, but you need to prove to upper management that it is a key metric with economic ramifications.

Good luck.
posted by Vindaloo at 11:21 AM on July 23, 2010

Yeah, we already have customer satisfaction surveys as a metric. They are important but again I get the feeling that they only looked on as numbers when the reality is more complex than that... but it's a hard thing to measure...

I definitely think we should keep doing them but at the moment they give bonus based on the customer satisfaction number alone without taking in to account the background.

So for instance I got flawless satisfaction surveys last year and got the highest bonus for that but all it really tells you is that my customers like me. I work hard to make that happen but my customers are just nice guys really.

I have colleagues who got lower survey scores but they have more challenging customers and management don't make any intelligent decisions to level things out. You can bend over backwards for some customers and they are never happy.

It's just another example of corporate stupidity. I'd prefer to use the surveys as a guide and to actively encourage customers to give us bad feedback as well as good but as it stands you kind of develop unspoken pacts with your customer where they always give you top marks for everything and then you will be friends and do that bit extra for them.

So yeah surveys are good but they shouldn't be just a number but in big companies they often are.
posted by aTrumpetandaDream at 11:32 AM on July 23, 2010

You're dead right. Check out Goals Gone Wild. I like to present it to management types with the preface, "It's from some hippy rag called the Harvard Business Review."

Your problem is going to be that you see rules and policies and such as mutable tools that someone uses to get something done. If a rule or policy is having a negative effect, then's it's broken and needs to be fixed.

The problem is, you're going to run into people to whom the notion of changing the rules is almost an alien notion, as if someone in management started with the rules of number theory and derived them from your corporate mission statement. (Or, to cast them as less than insane, they realize that your corporate overlords are made of an alloy of titanium and hubris and aren't going to be made to see flaws in their policies with any light short of a class V laser and aren't going to be moved by anything other than a nuclear blast.)

Then there's the less obstinate. I'll let a pro describe them:

There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. For the reformer has enemies in all those who profit by the old order, and only lukewarm defenders in all those who would profit by the new order, this lukewarmness arising partly from fear of their adversaries … and partly from the incredulity of mankind, who do not truly believe in anything new until they have had actual experience of it.

– Niccolo Machiavelli

If it's any consolation, as soon as I found the Highly Technical Corporation of America I'll be sending you a MeFi mail.
posted by Kid Charlemagne at 12:43 PM on July 23, 2010 [1 favorite]

Management gets what it measures. And it really gets what it rewards. Sometimes this leads to closing tickets any way possible "to fix the metrics". I've seen it happen in far less bonus driven departments, and I've seen managers tell their employees to do it.

In the best of worlds, management would take a comprehensive view of things. Total Fix Time for example, shouldn't be completely attributed to technicians. If you have to foist some things to developers, then that shouldn't be tacked on to your department, but to theirs. And by splitting the tracking up, it may indicate that more developers are needed to serve this part of customer need.

Statwise, what you're doing is easily detectable. Standard deviation, call volume, outliers etc. are much harder to fake, just check out Freakonomics, which gives some examples of statistical evidence of cheating, or this blog post from a high school math teacher. I wish I could find a reference right now to an MBA program that focused on experimental design in the business, because surely one of their textbooks would give you sufficient guidance.

But really, most of the data for this should be outside your control. I.e., you shouldn't be able to log multiple calls based on a single call you made. Or at least, there should be a distinction between things a customer cares enough about to call you out of the blue for and things you lead them into raising.
posted by pwnguin at 1:06 PM on July 23, 2010