A question about who is liable for potential data loss when backups don't work as promised

June 8, 2009 10:44 AM

I'm involved in a dispute about whose responsible for data. The basics: A RAID array in a small company's production server died. Under extreme pressure, the technician reconfigured the array, losing data in the process. The customer assured the tech the data was backed up. It wasn't. Now the customer says the tech should have checked the backup tapes before doing anything. Who is responsible for the data repair?

I work with a few independent IT techs - we share projects or send work each others way when one of us has a specific expertise that a customer might need. I got called by one, in this scenario:

- Customer production server RAID array failed. It was a RAID 5 array.
- the IT tech was instructed by the customer to get the server working "by noon" - there's a lot of pressure to bring the system up. The tech asked, several times, if backups were available and where were they - the customer showed him a stack of backup tapes, said they'd been working every day, and the office manager er, backed up her boss by saying that she changed it every day.
She went on to say there were problems before but their software vendor had fixed backup problems on their last visit. The tape backup software is a common, good quality system that obviously hadn't been configured well.

- The tech broke the array (I know...), took out the bad drive, and reinstalled the OS on a new array made from the two remaining disks. Which is when I arrived on the scene. After reinstalling the backup software, we found that the tapes hadn't even been formatted. And then I found that the onboard RAID controller was hosed, because it kept killing drives. Hence the RAID array wasn't bad, but the server's mainboard was definitely bad.
So, after a fair bit of panic, a new server is ordered (with proper RAID hardware), the broken RAID set is sent off to a data recovery center for recovery (they got everything back in spite of the reformat) and the process of restoring the production server was completed successfully, and that's where I and the tech spent most of the time - getting the new server running and everyone working again.

The obvious result - the customer doesn't want to pay any of the bill now begins. His reasoning is that "The fact that he (as an IT professional), when he asked us if we had backup tapes, which we did, should have made sure there was information on the backup tapes before anything else was done".

Now, I have issue with the way this independent tech did the repair, and he knows it, but he's the customer's tech, I'm a flunky in this situation. At the time I got called in, though, the time to do anything about it was long past.

My question: is the customer justified in not paying the bill? Or did the IT tech have reason enough to believe the customer and his office manager?
I'm asking the question anonymously for liability reasons, even though I'm not liable, if that makes sense.

posted by anonymous to computers & internet (9 comments total)

You presumably have a contract between the two companies, whom does it say is responsible for the backups? Part of good backup practices is testing the backups, not just the backup process, so whomever was in charge of doing the backups should be feeling most of the heat, but the tech who didn't verify first should share some of that blame

In QA, we have a phrase we live by, "trust but verify".
posted by nomisxid at 10:51 AM on June 8

" His reasoning is that "The fact that he (as an IT professional), when he asked us if we had backup tapes, which we did, should have made sure there was information on the backup tapes before anything else was done".

The customer is weaseling in an attempt to find someone to blame. Customers are like that, at least bad customers are.

If your technician had taken the time to do that, this same customer would most likely have been screaming about extending the downtime and insisting your tech take him at his word.

" is the customer justified in not paying the bill? "

The customer should insist that the party responsible for the backups cover the cost. Is the customer responsible for backups? You mention some third party as being involved in the backups; is that third party on the spot? The fact that the backups weren't present is the problem, and it seems as thought it was represented to the customer that backups existed.

The customer is absolutely, completely on the hook to pay for the service they received -- especially the expensive and critical data recovery! If they want to recover that cost, they should do so by looking toward whoever was responsible for preventing production data loss in the first place. That doesn't sound like it was your technician.
posted by majick at 10:52 AM on June 8 [7 favorites has favorites]

Well, the tech not only broke the array, but failed to make even the most basic check on the back-up tapes, and completely missed the fact that the problem was on the RAID controller not the array itself. I'd say you need to cut the client some slack here, not least because they seem to have made good faith attempts to both back up their data and help you by providing the backups.
posted by unSane at 11:07 AM on June 8

"If your technician had taken the time to do that, this same customer would most likely have been screaming about extending the downtime and insisting your tech take him at his word."

This. If they only have the single production server, it would have been quite time-consuming to verify the tapes - I'm just picturing the scenario with Arcserve/Veritas, for example. If your company didn't do the backups, you're not responsible for maintaining/verifying them, either.

I understand your issues with the troubleshooting the regular tech did - but it was a triage situation. The tech's job was to get the server (hardware and OS) functional again. This was accomplished. The problem of the data for additional software not being backed up correctly is not his fault. The data recovery is the customer's responsibility, and they can take it up with whoever set up the backups.
posted by HopperFan at 11:27 AM on June 8 [4 favorites has favorites]

100% customer's fault. There is no way for a technician to fully verify data that isn't his, other than getting verification from the customer (which was given).
posted by wongcorgi at 12:15 PM on June 8 [1 favorite has favorites]

hard way to learn a lesson. Hope the fallout isn't too bad.

But I agree with Magick - it's the backup providers fault fault they didn't have KNOWN GOOD backups. Once the company verbally confirmed that the had the data, it's their responsibility to deal with the fallout when they didn't.

This should be an expensive lesson for them, and they should pay the bills 100%, although it wouldn't be crazy to consider letting them slide on paying the tech that screwed up so badly. Not checking the tapes is damn near unforgivable.
posted by anti social order at 1:09 PM on June 8

100% customer's fault. There is no way for a technician to fully verify data that isn't his, other than getting verification from the customer (which was given).

This is so true. While I'm not big on the blame-shifting game, there is no way for anyone, IT technician or not, to properly "verify" backups of data that is not their own. How would the technician know if all the proper data was there, if it contained current versions, etc.? That's not to say that the tech shouldn't share in some responsibility for this, however. Just keep that fact in mind and bring it up to your customer.
posted by joshrholloway at 2:43 PM on June 8 [1 favorite has favorites]

The client accepted the risk when the tech told him that he will need their backups. The client needs to pay for all and the tech needs to learn the basics of raid.
posted by damn dirty ape at 2:59 PM on June 8

The tech went to service a hardware problem. Would he even know how to verify that the data on the backups was valid? I agree, 100% customer problem and responsibilty of the entity charged with taking backups. It sounds like someone needs a list of who's responsible for what as this type of finger pointing is quite common in multi vendor installations.
posted by x46 at 3:02 PM on June 8