The website got hacked. What now?

May 30, 2008 8:57 AM

A website I programmed got hacked. Credit card numbers were compromised. What do I do now?

I found the security hole and plugged it up, but at least one or two credit cards have already been stolen. There are a few hundred orders (less than 300) in the system. According to the error logs there were hits to the backend from the UK and Africa.

Obviously, I will recommend that all of the customers be notified. I don't suppose there's any way to do this in a classy manner that won't make our client look bad.

The worst of all is that the client will obviously suffer -- who knows how many of their customers will stop buying from them -- when it was not their fault. I feel like there is nothing I can do to resolve this problem...it's like a nightmare I can't wake up from. Please, please help.

posted by anonymous to computers & internet (22 comments total)

1. Notify customers as you said.
2. Offer to modify the site to use standard practices (encryption or last 4 digits only) for a very reduced rate.
posted by hitopshelf at 9:05 AM on May 30, 2008

If you don't have any assets worth going after, of course, it's possible that they won't bother, but if you have a successful business, I think a lawsuit is fairly likely. So your first order of business should probably be to consult an attorney. Your second order of business ought to be to try to stave off the lawsuit, perhaps by offering to pay for an independent security audit of the site and to fix any problems found.

Hopefully you have liability insurance that covers this. If not, hopefully you are at least incorporated so they can't go after you personally. If not, then I hope your contract included a disclaimer of liability (though this may not stand up in court, so they may sue you anyway).
posted by kindall at 9:10 AM on May 30, 2008

1. You should have business liability insurance if you don't already. This will protect you if you are sued. A million dollars should be the minimum coverage you get, and it should cost you about $50/month. If you can't afford this, either up your rates until you can or stop consulting.
2. Notify customers. There is nothing to be gained by hiding it.
3. Your customer needs to notify their merchant account company, and their merchant account company may choose to terminate their agreement or up their fees in response to your poor coding practices. Be prepared for this, and prepare your client for it.
4. You should fix it for free.
5. Do not, ever, even encrypted, store credit card numbers in an online system. Do. Not. Do. This. Ever. There are accepted ways ('CISP' or membership systems) to store a customer's account information up at the level of the merchant processor so that your client no longer faces this liability.

And if you ever run into a situation again where you need to store customer's credit card numbers for a client, do not do the job. It's really a bad idea.
posted by SpecialK at 9:10 AM on May 30, 2008 [1 favorite]

You may consider reporting the gaff to the credit card companies for each individual compromised card if the numbers are so small. It's in fact necessary to reduce liability to have them cancel the cards based on the information you provide, if it's able to be corroborated, as contacting the victims may be difficult to do reliably or in a manner in which there is no lag to them canceling the cards themselves.

Prepare a statement explaining why this won't happen again. You shouldn't be storing credit card numbers in any way that is easily retrieved from a database anyway, without at least some basic two-way encryption applied to them. If you were storing CVV/CVV2 numbers, you need to recognize that you may have violated merchant accounts rules and that your client may have theirs revoked. As it is, the CC company will be likely wanting to speak to you.

You need to contact your client first and foremost. You need to identify all numbers that were compromised (and yes, that's every single number stored on your system, regardless of what your error logs show) and have a firm figure for when it happened and how many were compromised. Work with them to issue a statement and build a strategy. Do so for free.

You also need to absolutely ensure, 100%, that this won't happen again. Again, simply "plugging the leak" may not be enough. If you're storing CC numbers in plain text and with context in a database, they can far too easily be stolen. There are ways around this—determine if storing the numbers to begin with is at all necessary.

No matter what, this will make you look bad. This is because you failed them. Mistakes happen and they may understand that, but this will make *them* look bad and that's because the company they hired to do a job failed at it and now bigger risks exist. You need to step up and take action to ensure that no one is actually hurt by this, and you need to do so as soon as possible.

Good luck and godspeed.
posted by disillusioned at 9:10 AM on May 30, 2008 [1 favorite]

The noble thing to do would be to urge your company to publicly claim responsibility, and urge the client to name and shame you. It might be a drop in the ocean from the point of view of bad publicity, but it will help a little - and you'll get (possibly only karmic) points for jumping on the grenade.

Then I'd take the experience and turn it public. You can be the guys who went through the fire and learned from your mistakes. Write a blog about it, put it in your pitch presentations*, don't try to hide it - because you can't.

The positive side? You're about to go on a learning curve that most people only ever dream of - and you've got front row tickets to the perfect shit storm.

Take a deep breath, and unleash the shit. Take your kicking, get your scars, earn your "the time I fucked up so badly story" and in a few months it'll only make you wake up screaming once a month at most. I promise.

Good luck!

*actually, maybe not so much your pitch presentations. :)
posted by Jofus at 9:11 AM on May 30, 2008 [1 favorite]

Oh, and don't tell any of the customers yourself but don't go home today until the client agrees that they're going to be told before the end of the day.
posted by Jofus at 9:13 AM on May 30, 2008

You need legal advice. Some states (such as California) now have strict laws requiring particular disclosures. And you have significant financial liability exposure now. Good luck to you.

I found the security hole and plugged it up

You found that one security hole. How many others might you have?
posted by Nelson at 9:21 AM on May 30, 2008

If I was one of your customers, I would appreciate immediate notification. You can figure out the rest in the near, near future.
posted by jabberjaw at 9:28 AM on May 30, 2008 [1 favorite]

2. Offer to modify the site to use standard practices (encryption or last 4 digits only) for a very reduced rate.

I think you mean for free ;) There is absolutely no way you the client should pay to fix this.

All customers need to be notified so they can cancel their cards, and you/your company should be prepared to get sued for loss of business.

In all honesty, you should also recommend your client finds a developer who knows what they're doing. Not to sound overly harsh here but this is a potentially business destroying mistake that was your fault - if you can't develop secure back-end facilities there's no way you should be anywhere near anything that handles customer's credit card information.
posted by missmagenta at 9:57 AM on May 30, 2008 [1 favorite]

Consult a lawyer. Your client may not be happy to learn that you did not follow standard practice, you stored unencrypted credit card numbers, and you exposed them to risks which have now been realized.

Lawyer up. In the future, do not program applications which deal with critical data unless you learn to handle that data properly. In this case there should have been no reason for you to ever store the credit card information.
posted by splice at 9:59 AM on May 30, 2008

I did not notice your proposal to modify the site to correspond to standards for a "reduced rate".

Seriously, lawyer up right now. Charging a client to fix a mistake you should never have made in the first place is a terrible idea.
posted by splice at 10:00 AM on May 30, 2008

If the 'hacker' had got into the database then they would have got all the card numbers not just one or two - I'm guessing the numbers were stored encrypted/encoded in the db and the back-end decodes them and the back-end login was the vulnerability - sql injection?
posted by missmagenta at 10:07 AM on May 30, 2008

seconding that you need legal advice.

these customers should already have been notified, as should have been the bank/cc processor you are dealing with. I am fairly certain your contract with the latter specifies this explicitly, causing you to be in breach of contract right now.

did I say legal advice is necessary here? yeah. get on it.

your client doesn't need to obviously suffer, at least not to the extend you may think now. it's how you handle your relationship with your customers now will make all the difference. what you do next will send a message to the affected customers about their value to you.

the consumers need to be notified and eventual damages need to be covered. this can easily become a PR nightmare if handled the wrong way. consider having the CEO write a personal letter apologizing to the individual harmed customers, paying for credit watch programs for a year or two for them (especially if it's just two of them) and promising to personally (!) help with any problems. don't try avoiding this because it's difficult, it will only infuriate them further. high-level, personal attention. don't let some assistant be the person in charge of this. go further than other companies usually do.

if this story becomes public, talk to the media about it. make this a story about you realizing you (the company) had made a mistake and how you went to extraordinary lengths to make it up because you value your customers. you found out about the problem, immediately corrected it and then did A,B,C,D,E... make this a case for how your company values are different from those of anyone else. you can shape this story into a juicy don't you wish other companies were like this bait that's far too tasty for many journos not to swallow.

consider sending the client emails so you can later prove what you said to them.

don't waste time.
posted by krautland at 10:08 AM on May 30, 2008

The specific reason you and/or the client you did the work for need legal advice is this: By storing card numbers unencrypted, you weren't compliant with thePayment Card Industry Data Security Standard, and PCI DSS compliance has been required of merchants by card brands since 2004-2005. Being non-compliant shifts pretty much 100% of the liability for the breach (both civil suits and fines) from the card brands and acquirer to the merchant. The PR problem and loss of business to your client might be nothing compared to the liability the breach incurs.

As a data point, the PCI DSS requires an incident response procedure which (among other things) reports breaches to acquirers and card brands, and typically police will be involved. You're sitting on evidence of a crime, after all. This also means you probably have an obligation to preserve evidence.

I imagine you will personally luck out because PCI DSS compliance will have been your client's responsibility. They may luck out because their acquirer did not require compliance for whatever reason, but they may also have attested compliance without obtaining it.

(And if it makes you feel any better, communicating the specific requirements in the DSS, or at least verifying that your custom code complied, was your client's responsibility.)
posted by mendel at 12:19 PM on May 30, 2008 [3 favorites]

PCI DSS certification is ridiculously easy to get, by the way. I've dealt with "professional compliance testers" who clearly had no idea what they were doing.

So you should get that next time. :)

And what everyone else said: God, why are you STORING credit card numbers?!
posted by rokusan at 12:51 PM on May 30, 2008

If you are in New York State, you or the client might be subject to the "Information Security Breach and Notification Act". Other states have similar laws, I am just providing New York's as an example. A lawyer may not even be aware that laws like this exist.

From the site: "The "Information Security Breach and Notification Act," effective December 7, 2005, provides New York State residents with the right to know when a security breach has resulted in the exposure of their private information. "

This means that businesses who experience computer intrusions that expose (the word exfiltrate is not used in the law) personal information are required to inform New York State residents of the intrusion. I know that I would like to know if somebody is not keeping my personal information secure and that I might need to check all my credit reports for fraudulent activity.
posted by LightMayo at 1:34 PM on May 30, 2008

[This is a followup from the anonymous asker.]

1) In all fairness our site was not "hacked". The vulnerability in question was in the page that was supposed to check for credentials and then redirect to a login page if the credentials weren't found. Due to a recently introduced bug in the code by yours truly, it was not doing the redirect, so the entire page was being displayed. So really the problem was that restricted pages were not being restricted from view. Basically anyone who guessed the url for the orders could get in. Encrypting credit cards would not have made a difference, since the site would have decrypted them anyway. I fixed the bug by restoring the redirect.
2) As it happens, this is an OLD site that had been written years ago, back before it was all that common to encrypt all information on the database end (it was definitely created before 2005). The newer web apps that we write have encrypted credit card numbers, hashed passwords, etc.
3) Why are we storing credit card numbers? Most of our clients are small mom and pop shops transitioning from mail order business. They enter their credit card information in terminals. They tend to prefer this method rather than having the payments be processed automatically online, but I willl take everyone's recommendations under consideration in this and look into doing direct processing of orders.
4) I have no intention of charging even a "small fee" for any fixes or improvements we make to their site. We will probaby just have them transition to one of our newer apps for free (i.e. free upgrade and web redesign).
5) We are a very, very small shop. So I or the "CEO" will certainly be directly involved in contacting the customers involved. The budget for this website was pretty small -- I think we made $2500 -- maybe $4k tops -- on the sale of this website. While it would be great to offer things like credit watch or start a blog about our experiences with having our site hacked, we just do not have the resources to do this.

I'm just about to call the client now and will recommend he immediately notify both the customers and the merchant account of the security breach.
posted by cortex at 1:42 PM on May 30, 2008

In all fairness our site was not "hacked". The vulnerability in question was in the page that was supposed to check for credentials and then redirect to a login page if the credentials weren't found. Due to a recently introduced bug in the code by yours truly, it was not doing the redirect, so the entire page was being displayed. So really the problem was that restricted pages were not being restricted from view. Basically anyone who guessed the url for the orders could get in. Encrypting credit cards would not have made a difference, since the site would have decrypted them anyway. I fixed the bug by restoring the redirect.

Isn't that worse than being hacked?
posted by xmutex at 2:00 PM on May 30, 2008

While it would be great to offer things like credit watch or start a blog about our experiences with having our site hacked, we just do not have the resources to do this.

uhm... that is not an argument. you need to take care of your customers in relation to the potential harm they might have suffered, not the potential harm it may have cost you based on the size of your business. the difference being that if they feel the response is inadequate to their problem they will be hostile towards you.

see it this way ... your customer isn't going to say "oh, they are a small company, of course I have to front the $75 to watch my credit report now myself" ... they are going to say "hey, they caused this problem, they should take care of it now." it's the colin powell "you broke it, you own it" theory.

this unfortunately does have the potential to wipe your company and potentially you out. treat it as such an important managerial issue. seek help from legal professionals NOW, not monday. I am also relatively certain there are people who have dealt with such issues before. this might be a good time to make some calls to friends running mid-size companies involving end customers. ask people you know without being too specific if they know anyone... crisis management specialists, etc...

(this might make a great case study for a magazine like harvard business review. they have a kicking series in their pub, which you can get in decent bookstores for around $16.)
posted by krautland at 2:08 PM on May 30, 2008

Do NOT talk to the press. That is your attorney's job.
posted by 4ster at 12:19 AM on May 31, 2008

Replying to the anonymous response above. I don't know what the contracts with the card companies say so this is just from a technical point of view.

I understand about the same shop manually entering the CC info into a terminal etc but from your comments I think you are still missing a few important points especially when you say:

"Encrypting credit cards would not have made a difference, since the site would have decrypted them anyway."

and

"The newer web apps that we write have encrypted credit card numbers, hashed passwords, etc"

Encrypting the data in the database sounds good but it's effectiveness and safety is seriously reduced if the key required for decryption is stored on the server. I know you want the store owner to be able to log in to the admin side of the site and see the CC info. That probably means that there is code on the server to read the encrypted data from the db, decrypt it and send it to the browser. I guess that sort of encryption is certainly better than nothing because it stops a hacker who gets access to the db but nothing else but it completely fails if someone gets access to the server file system. Then it's like locking your door but leaving the key under the mat.

I'm no security expert so others can correct me if this is wrong but if the CC info is stored on the server at all then it should be using public / private key encryption (e.g, PGP or GPG) where the server knows the public key but not the private key. Therefore the server can only encrypt. It cannot decrypt. There then needs to be some sort of system (even email would do) where the store owner receives the encrypted data and uses his private key to decrypt. The private key does not leave his hands, or at least his computer, protected with a passphrase.
posted by tetranz at 5:50 AM on May 31, 2008

I'm no security expert so others can correct me if this is wrong but if the CC info is stored on the server at all then it should be using public / private key encryption (e.g, PGP or GPG) where the server knows the public key but not the private key.

Seriously, the requirements are in the specification I linked above. They're spelled out there. Following them pretty much absolves you of liability.
posted by mendel at 10:11 AM on May 31, 2008