SOS: BOFH Gone Postal!

November 17, 2009 2:13 PM

Our small company is about to lay-off our sysadmin who refuses to share passwords and could go postal. Advice on how best to proceed?

Right now, our sysadmin is working from home, logging into our Windows server from there. Due to a number of reasons, we have decided to let him go. He has not shared passwords with anyone, and has the type of personality that leads me to believe he could cause a lot of problems once we let him go.

Before we even hint of a layoff, we're trying to get all of our ducks in a row. We will disconnect from the internet and hopefully get an IT consultant in here to lock us down and change all passwords.

We're around Portland, OR and need advice on who we can trust with this task. We're not comfortable with just putting an ad up on Craigslist; rather, we'd like to find a reputable company that we can trust throughout this situation (and probably consult with over the years, because I'll probably become the default "IT guy").

I understand how illegal it would be for him to refuse to share passwords and cause damage to our network, but honestly, I don't think legal threats will scare him. We just need to know how best to proceed at this point. Personal recommendations for local IT consultant companies would help a lot (especially consultants who deal with emergency isolation situations).

Throwaway email: [e-mail]

Thanks!

posted by anonymous to computers & internet (21 comments total)

It depends how he accesses the system and how paranoid he is. Dropping your internet connection locks him out, unless he has a back door. Could there be an alternate link, a DSL line, modem, etc.? You need to check for those.

If your system is not complex, one server, one router/firewall then you should be able to get eyes on and see what other interfaces there are. I have no recommendation for an IT consultant in your area, but when you get one, if your admin is not around, bring him in before terminating the admin to make sure your system is secure.

Potentially useful link, assuming you have a domain:
Change Domain Admin Password in Windows Server 2003 AD

Bear in mind he probably has a few other accounts with domain/enterprise admin privileges.

If you have physical access to a machine he has logged into as an admin, run Ophcrack to see if it can get any of his passwords.
posted by IanMorr at 2:36 PM on November 17, 2009

Basically, you need a company that provides the following things:
- Knowledge of IT and IT security (how to get access in the event it's not given)
- Knowledge of IT forensics (how to recover lost data and prove what happened)
- Knowledge of applicable law (how to pursue legal avenues if required)
- Knowledge of how you can improve your processes (how to keep this from happening again)

Warning: this is a plug: I'm inclined to recommend KPMG, but that's because I have some measure of experience with them. Not only do they have the boots on the ground with the tech knowledge to Get This Job Done, they can help you in preventing this from ever happening again. They also have a big building full of lawyers and can act as your legal council (thus preventing communications from being subject to discovery). MeMail me if you'd like more info. End plug

Now is a very good time to double-check your backups. Run backup jobs right now. Grab the backup drives and tapes and disconnect them from any plugged-in system. Time things such that disruption can be contained and remediated before customers notice or revenue is affected. Let the CEO know the situation immediately so that rough patches can be smoothed over and all the guards can be on call. Take an inventory of all the systems and data that he has access to.

Most of all, watch this guy very carefully. He may have confidential data that could damage the company's financial reporting, and that would be a Very Bad Thing. You need to know what he knows and you need to be aware that if there are any leaks, he may be behind them.
posted by TheNewWazoo at 2:39 PM on November 17, 2009 [1 favorite]

Goes as a given for all important stuff but...

Double check that you have usable backups and ongoing backups are working. Going with the worst case scenario you've got admin Kurtz gone up the river. No telling what kind of damage a twisted person with full access for a long time could do. Be sure that even if efforts to lock him out fail you can recover as quickly and cleanly as possible.
posted by Babblesort at 2:42 PM on November 17, 2009

Non preview - I see that TheNewWazoo had the same thought. Backups!
posted by Babblesort at 2:44 PM on November 17, 2009

1. Backup everything you care about on media that is disconnected from any network (e.g. tapes in a box). Do test restores on anything important. Do not leave media anywhere he has access to, and change any offsite storage accounts to prevent spurious tape recalls/net backup erasures.
2. Change all physical access methods to servers (e.g. code-based locks)
3. Inform all other users of the change to prevent phone calls and social engineering for passwords. Change all user passwords that can give remote access as soon as possible since it's likely he may have them stored.
4. See if physical access reset methods are locked down on servers, e.g. is the BIOS locked on any systems? Can you boot to a cdrom and run a Linux password change utility? What is the password reset procedure for your networking equipment?
5. You may wish to sniff his connection for a while before letting him go to see where he has stuff stashed. If you do so, use a physical network tap for the connection so it's not detectable via interrogating switches or looking for promiscuous NICs.
6. Check the documentation for the network and servers and do some reality checks to see how trustworthy it is. If the guy is a hack it will probably be non-existent or garbage.
7. Pull the plug once you have a procedure in place that you are comfortable with, assuming he has a time-delay script that will wipe your servers or turn off your network. Do not restore internet connectivity until all network equipment has had passwords changed and rules set to prevent any external access. Also remember that awarding a decent severance package contingent upon password transfer and verification does wonders for reforming bad apples. If you have a good worst-case plan, this will allow you to walk away if the severance negotiation turns into hostage extortion.
8. Monitor all outgoing and ingoing traffic for a while to verify no backdoors are being accessed and no time-delay scripts are phoning home.

But yeah, you'll need a full time consultant at the ready, and prepare for some downtime.
posted by benzenedream at 2:58 PM on November 17, 2009

I'd go to an established IT temp agency in your area for the initial lockdown/password change, and worry about establishing a relationship with a long-term IT consultant later. It'll cost more, but you won't have to worry that you picked someone off Craigslist who's not up to the job. It's actually not super complicated, but you definitely don't want to mess up.

Assume that the IT guy has a copy at home of everything on the network. How much of a problem is that?

If it's a big problem, consider having the IT temp back up the network first. Now e-mail the sysadmin and tell him he's forbidden to access the network effective immediately, and fired if he doesn't send you the administrator password by noon. If he logs in, and especially if he does any damage, you have grounds to go after him with a search warrant, restraining order, etc. Obviously, you also want a lawyer involved if you go this route.

And you don't lay off a sysadmin who won't share passwords with management, you fire him.
posted by gum at 3:09 PM on November 17, 2009

You can almost certainly retrieve the passwords using this tool. I'd do what benzenedream suggests with the addition of:

1. Grab the existing passwords using that tool, you'll need to reboot things in order to do it, so pick a time that the sysadmin won't be online.
2. Check that the passwords work, and then change them, making sure, again, to do this while he's offline.

posted by togdon at 3:10 PM on November 17, 2009 [2 favorites]

I'll piggyback onto gum's comment - any infosec consultant of any size worth their salt will jump on this project. Not only is this the sort of "omg intrigue!" thing that gets staffers all jumpy and excited to get to work, there's an obvious need here for work beyond just the immediate. Some sort of temp agency, IMO, isn't necessarily the best course of action unless you want a layer of indirection between the "thinkers" and the "doers" as a cost-savings measure.
posted by TheNewWazoo at 3:13 PM on November 17, 2009

On the carrot side as opposed to the stick side, you can make whatever severance you're planning to offer him contingent on providing the passwords and providing for a smooth transition. He may be willing to act against his own self-interest to get some payback, so be prepared with the other stuff, but it's worth at least giving him a chance to be professional and not burn bridges.
posted by willnot at 3:33 PM on November 17, 2009 [3 favorites]

Destroying property in the way you are suggesting the sysadmin is capable of doing is in fact a crime. In the worst case scenario, be sure you contact your local police department.
posted by Freen at 4:33 PM on November 17, 2009

On the carrot side as opposed to the stick side, you can make whatever severance you're planning to offer him contingent on providing the passwords and providing for a smooth transition.

Assuming you've got any room to move, I'd get a quote from someone like KPMG above and use this as ballpark for negotiating severance. Otherwise, what you're talking about doing here would be interpreted (in military terms) as "hostile action" (justified or not). As such, you're running the risk of open combat which could be chaotic and very difficult to contain.

Peace is more important (and often far less costly) than all-out victory.

Good luck.
posted by philip-random at 4:41 PM on November 17, 2009 [1 favorite]

1- benzenedream gives good advice. This goes back a few years, but there was a way on NDS to create a container (and users in that container) that were completely invisible. Even to full admins, even to that user. But that user had whatever access it was created with.

No clue if this is even possible anymore, but you never know.

Frankly, as he suggests, I'd plan to be able to rebuild everything from scratch. Plan for the worst, hope for the best. Maybe now would be a good time to consider an upgrade? Call in a consultant to assess the network, current setup, etc., and have them build a new machine offsite. Probably best would be to pull the plug on him and on the network on a friday afternoon, and spend the weekend rebuilding. If you need a cover story, call it a disaster recovery test.

2- Don't forget to have a plan for the workstations. If this guy is as sneaky as you think he is, he probably would put something on someone's desktop that gives him access again. Have the consultant build images for the workstations and redeploy them at the same time.

3- And don't forget to get with your internet vendor and change any security codes that might be on the account. Like the business equivalent of mother's maiden name. Maybe even change vendors. Or at least have them change IP addresses.

4- Physically scan the network for machines. I'd unplug everything, and then only plug back things that are wiped and necessary. You never know if someone put a zombie machine inside the ceiling...

But yeah, offering him a severance package that makes it worthwhile for him to not tamper with your stuff is probably the easiest path.

5- The business analogy is "carrot ON a stick". It is about always increasing expectations so that they are just out of reach of your employees. Like a farmer who gets his mule to plow the fields by holding a carrot on a stick just out of reach of the donkey, who is too stupid to know any better. It is NOT an analogy you want to hear your boss use. Doubly so if the boss doesn't even know what it means.
posted by gjc at 5:44 PM on November 17, 2009

Forgot to add. If you are really feeling sneaky/evil/etc., set up a honeypot on your network to catch and log if he does try to gain access.
posted by gjc at 5:47 PM on November 17, 2009

Carrot and/or/on stick.

Right with you on the "It is NOT an analogy you want to hear your boss use", though, regardless of word choice - in the end it means you're either getting beaten, or screwed out of something you were chasing…
posted by Pinback at 6:21 PM on November 17, 2009 [1 favorite]

You're not discussing this via e-mail I hope?
posted by geoff. at 7:11 PM on November 17, 2009

Seconding Geoff's question there...if he has even a whiff of what is going on, you could be doomed.

You should not discuss this via email or IM and you certainly shouldn't be doing anything regarding this AskMeFi post on your work connection. He may be snooping in on all the office communications. Be super careful about word getting out as you never know if an employee might overhear something and word could spread to him.

TheNewWazoo's suggested company sounds like it could be a good fit given their partnership with legal resources. Also, I'm assuming you know how to handle the actual termination but in case you don't, I'd recommend taking him to an "offsite" strategy meeting with senior management for an entire day. Make him feel important and disguise it as a real business meeting. You can even make a show of having everybody turn off their blackberries and leaving them on the table so he isn't able to check/access anything from it.

Then, while he's out of the office for the day, you have the "cleanup crew" come in and remove his access to everything, lock things down, etc. Terminate him at the offsite, confiscate his mobile device on the spot and inform him his personal belongings will be shipped to his home address. Offer him the severance in exchange for a smooth transition and make sure the IT consultancy is involved in that process to make sure it is done satisfactorily.

Also, when handing him the severance offer, you should also provide him with a fresh copy of the technology agreement/NDA I'm hoping to god you had him sign when he started and remind him that all of this applies regardless of whether he signs or not.

Lastly...and this is the "cover your ass no matter what" side of me talking...consider hiring a private investigator to monitor him for a period of time after the termination for suspicious activity. But consult with a lawyer prior to doing this as I have no clue as to the legality of it.
posted by Elminster24 at 7:46 PM on November 17, 2009

I can't believe anyone wants to hand this guy a carrot. He's the sysadmin and he's refusing to disclose passwords? Do you let your accountant refuse to disclose your bank balance? Your mail clerk refuse to disclose whether you got any letters? The babysitter refuse to disclose where your kid is?

If the boss demands passwords and he refuses, then he has disobeyed a clear and direct order and completely disqualified himself from being trusted to do his job. I'm all for mercy layoffs for the marginally incompetent or the accidental fuckup, but this guy does not deserve an unemployment check, and this company doesn't deserve to see its unemployment insurance rates go up to provide him one.
posted by gum at 7:56 PM on November 17, 2009

Gum, that is a good point. Although to be honest, it may be less risk/cheaper for the company to offer a couple grand of severance than it would be to confront him about the passwords and put him on guard and possibly provoke damaging actions from him.

I'd be curious to learn more from the OP about the situations in which he has refused to provide passwords and who he refused to provide them to.

Gum is dead on though about the possibility of his refusal to provide passwords being a reason to fire him with cause (and thus save the company unemployment insurance hikes).

Either way, the actual notice of termination needs to be done in a way that catches him COMPLETELY off guard without any ability to activate a "kill switch", access the systems, or steal company data (such as if he keeps a company laptop at home).

Odds are this guy is smart enough to backup a "personal copy" of his company files (lets face it, just about everybody does that) so like it or not he probably has sensitive info on some form of storage media.
posted by Elminster24 at 8:54 PM on November 17, 2009

Before you fire him, install a keylogger that does not show up in the system tray or task manager and runs as a hidden service so it's running regardless of what account may be logged on, and log his activities for a few days. The employer has total rights to install anything on their own machines. This may get you passwords and/or an indication of whether he is clued it, deleting or altering data, sending proprietary information offsite, etc.

Then run through the backups and security processes outlined above and be sure to 1) completely physically disconnect the internal network from all external connections at the time he is fired, 2) have the trustworthy outside network and security contractor available before, during, and after the firing, and 3) alert the contractor to the presence and use of the keylogger and request that it be completely removed prior to restoration of external network connections.

Years ago we had a smartass with a certification or two transition from a position as a technician to network admin, due to issues with upper management. I made a number of attempts to alert management to the fact that we were having system problems as a result of this guy's actions, but those were taken as sour grapes due to my not getting the network admin position and disregarded. One day this guy bragged to another co-worker that he "could bring the entire system down anytime" and that he had backdoor access. Co-worker took the issue to the CEO, who alerted me so I could protect the system, fired the guy immediately, then apologized to me for not taking my concerns seriously until then. Vengeful IT types can wreak havoc with your data, and it's very good your comany is taking the potential for damage seriously.
posted by notashroom at 11:11 AM on November 18, 2009

Also seconding gjc's #3 item regarding your ISP - I hadn't thought about if you are hosting anything significant outside your internal network.

If you have any public-facing web sites or stores, make sure to secure all methods by which they could be repointed or erased. This would include the following:

A) Hosting company point-of-contact and admin console
B) ISP technical contacts and web consoles
C) ARIN records
D) DNS providers, if outsourced - e.g. UltraDNS, DNSmadeeasy
E) FTP sites
F) Admin consoles on any external SaaS sites (Salesforce, etc.)
G) MX records, or hosted spam filtering service e.g. Postini, etc.
H) Transaction processing sites, e.g. Paypal, credit card companies, payroll, etc.

At my current company, our (not so thorough) HR person was surprised when I hired an IT person and asked for a full security check including credit report. They didn't really understand that the person who can backup and restore your accounting system can also replace anything in that same accounting system.

Also, before installing keyloggers and/or any sniffers, GET PERMISSION IN WRITING FROM YOUR CEO and keep a copy at home. It's pretty common for nontechnical people to freak out when they finally understand that very few things they do on a company network are actually private, so make sure this is discussed up front.

I'd also advise against a software keylogger unless you get an expert involved, since many of these are hidden from view but not hidden from somebody who knows how to check process tables and could be running tripwire-like apps. Hardware keyloggers are really hard to detect (especially the ones build into the keyboard) but won't work in your case since the guy is working from home.

As gjc also mentioned, re-imaging the desktops with a freshly created image is the safest way to make sure there are no backdoors installed (LogMeIn, VNC, dameware, custom malware, etc.). If somebody had admin access and you suspect them of malicious intent, wiping and installing a new OS is the only way to really trust that machine again.
posted by benzenedream at 1:29 PM on November 18, 2009

After all is said and done, build a system of 'fences' that prohibits any one person from having too much power. If structured correctly, a network admin doesn't need access to email and confidential network files. It will add some overhead when resolving issues, but will eliminate risk. This is difficult to do in a small shop and will require discipline. Once you have your fences in place, electrify them.
posted by jasondigitized at 3:04 PM on November 18, 2009