Why did zotob get to take hold?

August 17, 2005 3:29 PM

Wormfilter. Why did so many corporations wait so long to implement the recent updates to Windows 2000? Microsoft released them last Tuesday. As of yesterday many corporations were just implementing them, and by then it was too late and they got hit with the zotob worm.

I know some of the so called fixes have in the past caused more problems than they have solved. However, when there was so much discussion about how to exploit this vulnerability last week why did so many companies just plod on with the regular several to more week schedule of updates? I spoke with an IT person at a major corporation which is down two days and counting and their response, "we have to test any new patch against hundreds of applications to make sure it is compatible." This seems a bit bureaucratic to me. The risk of screwing something up with the patch is lowered, but then you leave yourself wide open for weeks while you test. Are many companies this bureaucratic and if so will this experience empower the IT departments to move faster on looming threats?

posted by caddis to computers & internet (15 comments total)

You don't roll out and you get burnt by a vulnerability: you can point to corporate policy.

You do roll out and it breaks stuff: you, personally, are to blame.

No-one gets fired for playing CYA.

(I'm in a cynical mood today).
posted by Leon at 3:38 PM on August 17, 2005

Leon's spot-on as far as my experience goes. Sometimes developers take months to approve new patches after MS releases them, if they approve them at all.
posted by mr_crash_davis at 3:40 PM on August 17, 2005

You mean Windows 2000 right?

Speaking generally without knowledge of this virus.. Perhaps it's because Win2k didn't (until recently) have an "automatically download and install critical updates" capability—at least I don't think they did. Most small IT departments have not prioritized the chore of going around manually installing updates on screaming-downtime servers and far flung luser machines, so updates are not prompt.
posted by fleacircus at 3:52 PM on August 17, 2005

There are a number of reasons really, Leon pointed out probably one of the most crucial. Lack of manpower if centrally deployment is not an option (IT department is not located on site). Lack of expertise or experience in dealing with threats like this. Microsoft also labelled the specific patch as a low-risk so that definitely did not help.

The number one reason that I have personally run into a number of times is the lack of client support. Regardless of the number of stories that are all over the place when a problem like this happens, people are still pissed-off if you enforce some draconian policy on their machine(s). I've explained it all about 1000 times but it just does not resonate with a lot of people. It really depends on the amount of influence the IT department has within an organization. Sadly, (and far too often) they're still regarded as a necessary evil and people treat them as such.

Thankfully, my current place of employment provides me the necessary backing to do this sort of thing (and it wasn't always this way). For the most part, we were completely patched-up and this particular problem did not affect at all.
posted by purephase at 3:56 PM on August 17, 2005

Oops. Yes, Windows 2000. (that is sort of embarrassing.)
posted by caddis at 3:57 PM on August 17, 2005

My favorite forgotten factor: many people are away in August for vacation. It doesn't matter how good your patching system is, it's not going to patch a laptop that's not hooked into the network. Suddenly on Monday or Tuesday an exposed system shows up ripe for the plucking. Oops.
posted by smackfu at 4:18 PM on August 17, 2005

fleacircus: a bare W2K install has Automatic Updates turned on by default.

However, I've never met a SysAdmin who doesn't turn it off immediately. Nobody wants MS silently breaking 500 machines overnight.

Another reason: with a halfway decent firewall and some simple filtering at the mail server, most corporate desktop (not laptop or server) PCs shouldn't need to be updated every 5 minutes - security updates, in that situation, are more likely to break something than to fix a hole in your systems. In that case, it's good policy to wait for at least the first round of fixes to the fixes before you install.
posted by Leon at 4:19 PM on August 17, 2005

I'm very glad we use Macs at my paper, since our IT department puts all of yours to shame. We're still using OS 9.1, and we only upgraded to that long after OS X.2 was out. If we were a Windows operation, we'd be well screwed.

That said, there are very good reasons for not installing patches promptly. There's no good reason to trust them not to break something, for a start -- most of Apple's recent patches have broken something or other, in my experience -- and while those breaks are acceptable at home, they're not when those machines *are* your company.

In the case of the New York Times, as at my paper, downtime longer than about an hour can be utterly catastrophic. So system patches have to be applied in the small window between the night staff leaving at about 4am, and the day crew shuffling in at 10am. Six hours to patch an editorial floor? You'd better be damn sure your patch is going to work first time on all the software configurations you have out there.

The way you make sure it does work is by testing on all those configurations, by consulting with the relevant people in those departments and by having a roll-back plan. Doing all that in a large business is by necessity bureaucratic, because flying by the seat of your pants is a good way to get fired. 10 days is not enough for all of that, and neither is net chatter enough reason to skip it.

"Screwing something up" as you put it can cost hundreds of thousands of pounds, and reducing the risk of that is an IT man's job. (Course, using Windows in a publishing environment really should get someone fired anyway.)

I don't see this empowering them to "move faster", because moving faster exposes companies to that sort of catastrophic risk, while worm infections can so far be cleared up relatively quickly -- take the network off the internet, restore from backups or disinfect machines, done.
posted by bonaldi at 4:51 PM on August 17, 2005

We're still using OS 9.1

Really? What applications does your paper rely on?

I've never had an OS X update break anything except very minor problems with a few Apple applications (mostly Safari and Mail). Server or client side. I've heard of a few (mostly major version number updates) that have caused widespread problems, mostly application specific or affecting the power management configuration (failure to wake from sleep) in the early days of OSX. I'm just trying to imagine an environment where a completely unsupported and dwindling OS makes more sense. I don't disbelieve you, just curious.
posted by realcountrymusic at 6:17 PM on August 17, 2005

"I know some of the so called fixes have in the past caused more problems than they have solved."

Also, in my experience, this is not really the case anymore. Service Packs can be quite detrimental but they are effectively a new OS in and of itself, not just a simple patch. They usually include a lot of the past security updates, but they also include a lot of the latest (or later) development on the operating system itself.

Patches (or critical updates) are largely aimed at fixing bugs in existing system components that should (theoretically) not impact system stability in any serious way. The ready excuse of delaying updates due to software incompatibility is, IMO, neglecting responsibility. Since Windows 98/ME (XP Home to some degree as well) most professional Windows-based operating systems can be safely patched without really worrying about application compatibility.

Some may argue (and rightly so) but in my years of working with Windows 2000, XP, 2003 and the 500+ applications that I've tested in each iteration, I've probably found one or two that would not work despite all resuscitation attempts. In both situations, the applications were 10+ years old, so it did not really come as a surprise.
posted by purephase at 7:05 PM on August 17, 2005

[changed fpp to say Windows 2000 instead of Microsoft 2000]
posted by jessamyn at 7:37 PM on August 17, 2005

realcountrymusic, we're using QPS 1 (QuarkXPress with its own article and layout server plus a dedicated text editor called Quark CopyDesk), integrated with a wires service from QuickWire and photos by FotoWare.

OS X versions of all those applications are now available, it's true ... but the company's tight-fisted, nothing's broken (though IE 5 becomes daily less usable) and the upheaval of an upgrade will be *mighty*. Moreover, we're stuck in new-system-approval limbo as various factions all pull in separate directions for their chosen system, on top of a bubbling staff resentment to Windows.
posted by bonaldi at 8:06 PM on August 17, 2005

Many corporate IT departments, already overburdened and understaffed, probably have not finished qualifying the patch on all supported system images. I've seen turnaround time on this stuff stretch into the months, let alone a few days.

"Are many companies this bureaucratic and if so will this experience empower the IT departments to move faster on looming threats?"

Are you new to the corporate world? Most of them are far worse than you describe. And no, this one experience is not enough to change policies at more than a handful of companies. Similar things have happened in the past, but very, very rarely has the corporate response been increased agility.
posted by majick at 3:25 AM on August 18, 2005

This is one of the reasons most IT departments are not too keen on shipping patches to their users too quick.
posted by sebas at 5:34 AM on August 18, 2005

Also, in my experience, this is not really the case anymore.

Except, as a sysadmin, I can't trust that. If I patch, and break the software that runs the company's business, we're hosed. Period. The files must run.

So, I do what I can. I watch the lists, when I see a vulnerability, I look at the problem, and decide what to do.
The other thing. There are tons of things running on a default Win2K/Win2K3 server that you very probably do not need to have running, and shouldn't have running on a production server. Find them, turn them off. Then, if there is a hole in one of them, that machine is safe.

The biggest problem with Windows is they convinced everyone that Windows was so easy that they didn't need those fancy-pants sysadmins to keep their networks running. The problem is, of course, that it isn't. It's easy to get two machines talking to each other. It's much harder to control what they say. That's the weakness that worms exploit.

I actually doubt that "increased agility" is a good thing. Remember: There's always an answer that's quick, easy and wrong. Your defense against worms shouldn't be "quick, patch!" Your defense against worms should be an ongoing, constant evaluation of what traffic you allow on your networks. Patching is a part of this -- but if you install a patch that keeps your software from running, one has to ask: "Wouldn't the worm have been better? Or, at least, not any worse?"

In, oh, err, many years in the field, no network that I've managed has been wormed. Not one. And panic patching had nothing to do with it. Worse, the time between patch and exploit continues to drop (never mind the times when the exploit hits before the patch.) Thus, panic patching isn't helpful.

Thus, I don't use patching as my primary defense. Patching is the final piece of the puzzle, not the first.
posted by eriko at 5:41 AM on August 18, 2005