Is it wise to install Windows server updates without restarting?
August 10, 2011 3:18 PM Subscribe
Is it wise to install Windows server updates without restarting?
We have an systems administrator who routinely pushes out critical Microsoft updates to servers without restarting them. This causes various problems like "Hey, I keep getting prompted to restart my machine!" from Citrix/Remote Desktop Users, and it often stops services like SQL Server without restarting them (unless you manually restart the service or server). I would like this behavior to stop; besides the obvious explanation "the updates aren't helping until the machine is restarted, assuming the updates require a restart" and "disrupting stasis by installing updates and not rebooting is not a good idea," do you have any other information I can use to put a stop to this behavior?
The justification offered by the sysadmin is that some of the updates don't require a restart and are providing protection right away, and he can't get around to restarting the servers immediately, so he'd rather take the chance of just installing them and restarting at his convenience.
We have an systems administrator who routinely pushes out critical Microsoft updates to servers without restarting them. This causes various problems like "Hey, I keep getting prompted to restart my machine!" from Citrix/Remote Desktop Users, and it often stops services like SQL Server without restarting them (unless you manually restart the service or server). I would like this behavior to stop; besides the obvious explanation "the updates aren't helping until the machine is restarted, assuming the updates require a restart" and "disrupting stasis by installing updates and not rebooting is not a good idea," do you have any other information I can use to put a stop to this behavior?
The justification offered by the sysadmin is that some of the updates don't require a restart and are providing protection right away, and he can't get around to restarting the servers immediately, so he'd rather take the chance of just installing them and restarting at his convenience.
I sympathize (only a very tiny bit) with your admin, but you know what I do? I don't install the updates until I CAN reboot. Which for me is at night or over weekends when users aren't in the system. It's just...untidy and uncool to do updates during primary production hours.
posted by Lyn Never at 3:50 PM on August 10, 2011
posted by Lyn Never at 3:50 PM on August 10, 2011
You also perhaps underestimate what 'his convenience' entails. Dealing with rebooting a server that might be supporting many different users or processes can be time consuming. As can running updates. What might be a simple 5 minute process on a desktop PC could take several hours on a server. Getting everyone off it, stopping all processes that depend on it, downloading all the drivers, installing them, rebooting and then discovering there are more updates now suitable for it. Or, worse yet, some oddball part of the hardware has a conflict with one of the updates. Meanwhile some errant users are trying to get back into the box think it's back online.
So 'his convenience' might mean a whole damned day juggling a nightmare. Cut him some slack.
posted by wkearney99 at 4:58 PM on August 10, 2011
So 'his convenience' might mean a whole damned day juggling a nightmare. Cut him some slack.
posted by wkearney99 at 4:58 PM on August 10, 2011
I am doing this as we speak... is it wise? meh you can do it, but the certain updates do require a reboot (some do not). Best thing to do is get on a schedule so everyone knows when the servers will be rebooted. Do updates as needed and when it comes time to reboot everyone will know why the system is down.
posted by alfanut at 5:50 PM on August 10, 2011
posted by alfanut at 5:50 PM on August 10, 2011
Sounds like your admin is a afraid of downtime and instead risks the functionality of some services on the server instead of restarting the server after updates. I've found that most users are pretty tolerant of downtime as long as you let them know in advance. All he needs to do is plan a date in advance when he will perform all the necessary updates, and email the users saying something like, "The server will be down for X hours on X day for maintenance issues. Please contact me if you have any questions."
As an admin, it sounds like he's causing more annoyances by not restarting the server. Whatever "inconveniences" he thinks there are, he is far out numbered by multiple users being inconvenienced by his actions. Some admins become really attached to their servers and forget that a server is meant to provide services to multiple users. To the admin, the server becomes a beloved pet-project where the admin can run it any way they want, and no one can be more inconvenienced than the admin himself. Assuming the server belongs to your business, he needs to be reminded that it's not his server, and he must do what everyone else in the business industry does, and that's make the customer happy (where the customers are the users). If he didn't have any users, he probably wouldn't have a server.
When it comes down to it, you are a user and the only justification you need is to say that not restarting the server is annoying. I highly recommend that you suggest an alternative to him, like sending out an email as I suggested above, so you don't come of as a complainer. If you can petition several users to approach him at the same time with the same complaints, he's more likely to budge. You can also send a professional email to him, cc'd to other users, and gently suggest to him that maybe it would it would be better if he restarted the server. One on one, he can assert his dominance as admin, but if multiple people join in, he's more likely to realize his duty as an admin is to serve his users.
posted by nikkorizz at 5:54 PM on August 10, 2011
As an admin, it sounds like he's causing more annoyances by not restarting the server. Whatever "inconveniences" he thinks there are, he is far out numbered by multiple users being inconvenienced by his actions. Some admins become really attached to their servers and forget that a server is meant to provide services to multiple users. To the admin, the server becomes a beloved pet-project where the admin can run it any way they want, and no one can be more inconvenienced than the admin himself. Assuming the server belongs to your business, he needs to be reminded that it's not his server, and he must do what everyone else in the business industry does, and that's make the customer happy (where the customers are the users). If he didn't have any users, he probably wouldn't have a server.
When it comes down to it, you are a user and the only justification you need is to say that not restarting the server is annoying. I highly recommend that you suggest an alternative to him, like sending out an email as I suggested above, so you don't come of as a complainer. If you can petition several users to approach him at the same time with the same complaints, he's more likely to budge. You can also send a professional email to him, cc'd to other users, and gently suggest to him that maybe it would it would be better if he restarted the server. One on one, he can assert his dominance as admin, but if multiple people join in, he's more likely to realize his duty as an admin is to serve his users.
posted by nikkorizz at 5:54 PM on August 10, 2011
I want to add that almost all users are willing to except that servers need to go down for maintenance. It's a fact of life that applications need to be updated and vulnerabilities need to be removed.
If you logged on to Meta-Filter, and saw a message like "The server will be down on Friday between 5PM and 7PM eastern time for scheduled maintenance." You wouldn't think much of it. You may think, "Oh okay, I guess I'll just do what I need to get done on Meta-Filter before/after the outage," and schedule around it. At this point, it becomes your responsibility to remember, so if you forget about the outage, it's not the server or the server admin's fault, it's your fault. It's much better to have that kind of control instead of the unpredictability of random outages.
What users DON'T like is unscheduled outages. It sucks when a service you'd normally expect to have, like a SQL server, to be unavailable. You don't want to be demonstrating some new database feature to business partners at a meeting, and then suddenly find out that the the server is unavailable. Now THAT sucks.
posted by nikkorizz at 6:10 PM on August 10, 2011
If you logged on to Meta-Filter, and saw a message like "The server will be down on Friday between 5PM and 7PM eastern time for scheduled maintenance." You wouldn't think much of it. You may think, "Oh okay, I guess I'll just do what I need to get done on Meta-Filter before/after the outage," and schedule around it. At this point, it becomes your responsibility to remember, so if you forget about the outage, it's not the server or the server admin's fault, it's your fault. It's much better to have that kind of control instead of the unpredictability of random outages.
What users DON'T like is unscheduled outages. It sucks when a service you'd normally expect to have, like a SQL server, to be unavailable. You don't want to be demonstrating some new database feature to business partners at a meeting, and then suddenly find out that the the server is unavailable. Now THAT sucks.
posted by nikkorizz at 6:10 PM on August 10, 2011
Mod note: From the OP:
Sometimes it's a week or more before the servers are restarted. To me, I agree that it's best to wait for a weekend rather than disrupting the stasis of a production environment.posted by jessamyn (staff) at 6:22 PM on August 10, 2011
I wasn't asking specifically for an answer on how inconvenient this is or advice on whether to cut slack; I'm looking for best practices for system administration, vs. inconveniencing an entire organization and specifically one's own department to serve the interests of a single person.
I've served as an SA backup for many years and just recently I worked from 9AM to 4AM the following day because a critical server took so long to restart due to drive errors that necessitated a several-hour-long-CHKDSK response. Guess who scheduled the CHKDSK but didn't want to be around for the restart...
To me, if the security updates that affect system files are not in effect, and cause clear disruption, which is not solely limited to annoying Citrix end users and stopping mission-critical services like SQL Server...one should wait until they have the opportunity to restart.
I would cut the guy more slack but this causes real-world disruptions and I'm looking for any sort of industry-standard-esque approach. How do serious enterprises handle this?
One more disruption: you can simply type the wrong keystroke at the wrong time during an RDP session and restart a mission critical server when the restart pop-up appears...I usually stop the automatic update service to prevent this from happening.
Theres a wide variety of answers here... I think implementing some ITIL standards like change control and agreed maintenance windows would help control this.
With this, to install patches a change request would be raised that would detail whats happening, roll back plan, impact to end users etc. The approvers related to the servers (IT guys, database owners, someone who represents the interests of the end users etc) would review the details and approve, and then the work would go ahead in the agreed maintenance windows.
No need for the admin to stress, they can just reboot the system at this time without worrying about end user impact as this is already agreed upon, planned and scheduled.
I would also recommend for citrix servers to have two servers with the apps load balanced across them. That way before you patch one you can stop logons, send a message to users on the box saying there will be maintenance but they can log off the app and log back on to connect to an available server, and that they will be force logged off in ten minutes. Force log them off, do the patch, reboot, test, make the system available.
posted by Admira at 7:31 PM on August 10, 2011
With this, to install patches a change request would be raised that would detail whats happening, roll back plan, impact to end users etc. The approvers related to the servers (IT guys, database owners, someone who represents the interests of the end users etc) would review the details and approve, and then the work would go ahead in the agreed maintenance windows.
No need for the admin to stress, they can just reboot the system at this time without worrying about end user impact as this is already agreed upon, planned and scheduled.
I would also recommend for citrix servers to have two servers with the apps load balanced across them. That way before you patch one you can stop logons, send a message to users on the box saying there will be maintenance but they can log off the app and log back on to connect to an available server, and that they will be force logged off in ten minutes. Force log them off, do the patch, reboot, test, make the system available.
posted by Admira at 7:31 PM on August 10, 2011
Very few Windows updates that affect runnining system functions/services do anything before being rebooted. The installer just sets itself up to replace the file(s) in question at boot up time because they are in use.
you can simply type the wrong keystroke at the wrong time during an RDP session and restart a mission critical server when the restart pop-up appears.
Whoa, whoa. The rdp/citrix users shouldn't be local admins or have the power to reboot. This is your larger problem. Remove these privs and they won't be tempted to hit the reboot button. Not to mention it will be a lot more secure. Heck, you won't see the reboot or update message again.
posted by damn dirty ape at 7:44 AM on August 11, 2011
you can simply type the wrong keystroke at the wrong time during an RDP session and restart a mission critical server when the restart pop-up appears.
Whoa, whoa. The rdp/citrix users shouldn't be local admins or have the power to reboot. This is your larger problem. Remove these privs and they won't be tempted to hit the reboot button. Not to mention it will be a lot more secure. Heck, you won't see the reboot or update message again.
posted by damn dirty ape at 7:44 AM on August 11, 2011
At my organization we handle this by having regularly monthly maintenance on the schedule. For about two hours each month it is known to staff on the calendar that updates will be applied after hours and the terminal server will be rebooted. Occasionally there is security update that necessitates mid-month updates, but I always do them after hours. I come from a Unix administration background, so I find the frequent reboot cycle on Windows Server machines to be a tad silly, but it is the standard operating procedure. Given the choice to delay updates until the reboot is scheduled or apply them ad-hoc, I would definitely stick to the schedule.
posted by dgran at 7:57 AM on August 11, 2011
posted by dgran at 7:57 AM on August 11, 2011
The justification offered by the sysadmin is that some of the updates don't require a restart and are providing protection right away, and he can't get around to restarting the servers immediately, so he'd rather take the chance of just installing them and restarting at his convenience.
As others have said, this is not the case with Windows. Most of those updates do not apply until the system is entering the boot phase prior to the GUI loading. He's going to have to pick a low peak time to enforce a mandatory reboot, especially if these machines are connected to the internet.
Your main concern here is critical updates. So having a good firewall and keeping the network traffic internal is a good start. Your out facing servers however (WWW servers) should be rebooted immediately after updating. If this causes a downtime concern, then they really should look into load balancing these services so one can reboot while the other takes over for short while.
posted by samsara at 11:41 AM on August 11, 2011
As others have said, this is not the case with Windows. Most of those updates do not apply until the system is entering the boot phase prior to the GUI loading. He's going to have to pick a low peak time to enforce a mandatory reboot, especially if these machines are connected to the internet.
Your main concern here is critical updates. So having a good firewall and keeping the network traffic internal is a good start. Your out facing servers however (WWW servers) should be rebooted immediately after updating. If this causes a downtime concern, then they really should look into load balancing these services so one can reboot while the other takes over for short while.
posted by samsara at 11:41 AM on August 11, 2011
This thread is closed to new comments.
posted by msbutah at 3:29 PM on August 10, 2011