Every couple of months or so, we'll get a call from an Administrator reporting that his system hung when he tried to reboot it after installing patches. More often than not, the patches were installed by an Administrator logging on to the server via Remote Desktop (without using the /console or /admin switch) and using either the Windows Update web site or the "Automatic Updates" tray icon. Both methods do the same thing and use the same processes. After the updates are installed, the Administrator clicks on the "Restart Now" button to complete the installation. The Remote Desktop Session goes away, and the Administrator thinks that the server is in the process of rebooting.
However, the problem is that the server may not really be rebooting. When the Administrator tries to connect back into the server via RDP after several minutes, he discovers that he cannot. When he logs on at the console of the machine to investigate, he discovers that the RDP Listener is listening on port 3389 but no-one can connect via RDP. To resolve the issue, he has to reboot the server from the console. So what happened?
The first place to start is with the installation log files for the updates. In the %SYSTEMROOT% folder, there are several .log files created when patches are installed:
When you open up one of the files you can walk through the installation of the patch, including the following information:
What time the patch install started:
0.656: 2008/02/05 10:20:34.046 (local)
0.656: c:\53f60e03a81769236c7d3218\update\update.exe (version 220.127.116.11)
0.656: Service Pack started with following command line: /q /z
0.859: ---- Old Information In The Registry ------
Location of the files being updated
Whether any errors occurred. For our scenario this is crucial. One of the most common error message seen in this scenario is:
1.703: Failed To Enable SE_SHUTDOWN_PRIVILEGE
Getting back to our problem, the most likely problem is that there is a global variable for Terminal Services which has been set. This is set at the beginning of the shutdown process. Since we are not in the Console Session, we "ask" other processes if we can restart the system as opposed to forcing them to close and restarting the system. If one of those processes does not "allow" us to restart, then the system shutdown is aborted. Since the RDP session has already exited, the Administrator thinks that the system is restarting, but we are really in limbo. The Terminal Services process restarts the listener on port 3389, but since the global variable has been set, any attempt to connect to RDP is unsuccessful. Obviously the reboot clears this, and everything appears normal once more.
So what's the solution? There are a couple of different ways to address this (beyond installing the patches at the physical console):
Use the SHUTDOWN.EXE utility with the /r and /f switches. This will force applications to close and restart the machine. Now, the caveat here is that forcing an application closed may cause it to lose data. It is a risk to be aware of.
Log on to your RDP session using the /console or /admin switch (we discussed these
in an earlier post