Resolving WinRM errors and Exchange 2010 Management tools startup failures
As was discussed in the previous (related) blog post "Troubleshooting Exchange 2010 Management Tools startup issues", in Exchange 2010 the Management tools are dependent on IIS. As was discussed in that blog, we have seen situations where the management tool connection to the target Exchange server can fail, and the error that is returned can be difficult to troubleshoot. This generally (but not always) happens when Exchange 2010 is installed on an IIS server that is already in service, or when changes are made to the IIS server settings post Exchange Install. We have seen that these changes are usually done when the IIS administrator is attempting to "tighten up" IIS security by editing the Default Web Site or PowerShell vdir settings. The situation is further complicated by the fact that some of the errors presented have similar wording; most seem to originate with WinRM (Windows Remote Management), and in some cases different root problems can produce the exact same error message. In other words, depending on how knowledgeable you are with these errors, troubleshooting them is all around... not much fun. I was approached by a good friend of mine and he asked what I thought we could do to make these errors a little easier to troubleshoot. I was studying PowerShell and PowerShell programming at the time (I just happened to be reading Windows PowerShell for Exchange Server 2007 SP1), and I thought that this would be a perfect opportunity to try and apply what I was learning. This is the result. Introducing the Exchange Management Troubleshooter (or EMTshooter for short). What it does: The EMTshooter runs on the local (target) Exchange server and attempts to identify potential problems with management tools connection to it. The troubleshooter runs in 2 stages. First, it will look at the IIS Default Web Site, the PowerShell vdir, and other critical areas, to identify known causes of connection problems. If it identifies a problem with one of the pre-checks it will make a recommendation for resolving the problem. If the pre-checks pass, the troubleshooter will go ahead and try to connect to the server in the exact same way that the management tools would. If that connection attempt still results in a WinRM-style error, the troubleshooter will attempt to compare that error to a list of stored strings that we have taken from the related support cases that we have seen. If a match is found, the troubleshooter will display the known causes of that error in the CMD window. Here is an example of how this might look like: When I was designing the troubleshooter, I could have just written a little error lookup tool that handed over the appropriate content for the error you were getting, but I felt that was not as robust of a solution as I was aiming for (and not much of a learning experience for me). So the tool runs active pre-checks before moving on to the error look-up. The amount of pre-checks it can run depends on the flavor of OS you are running on and the options you have installed on it, such as WMI Compatibility. Basically, I have taken all of the documentation that has been created on these errors to date, and created a tool that will make the information available to you based on the error or problem it detects. Hopefully this will cut down on the amount of time it takes to resolve those problems. Event reporting: When you run the EMTshooter it will log events in the event log. All results that are displayed in the CMD window are also logged in the event log for record keeping. Events are logged to the Microsoft-Exchange-Troubleshooters/Operational event log and are pretty self-explanatory. Things to remember: Depending on your current settings, you may need to adjust the execution policy on your computer to run the troubleshooter, using: Set-ExecutionPolicy RemoteSigned Or Set-ExecutionPolicy Unrestricted Remember to set it back to your normal settings after running the troubleshooter. This version of the troubleshooter needs to run on the Exchange Server that the management tools are failing to connect to. While our final goal is that the troubleshooter will be able to run anywhere the Exchange Management tools are installed, the tool isn't quite there yet. We have seen instances where corruption in the PowerShell vdir or in IIS itself has resulted in errors that seemed to be caused by something else. For instance, we worked on a server that had an error that indicated a problem with the PowerShell vdir network path. But the path was correct. Then we noticed that the PowerShell vdir was missing all its modules, and quite a few other things. Somehow the PowerShell vdir on that Exchange Server had gotten severely... um... modified beyond repair. WinRM was returning the best error it could, and the troubleshooter took that error and listed the causes. None of which solved the problem. So be aware that there are scenarios that even this troubleshooter cannot help at this time. The troubleshooter is still a bit rough around the edges, and we plan to improve and expand its capabilities in the future. We also hope to be able to dig a little deeper into the PowerShell vdir settings as time goes on. Also note that the troubleshooter will NOT make any modification to your IIS configuration without explicitly asking you first. Permissions required: In order to run the troubleshooter, the user will have to have the rights to log on locally to the Exchange server (due to local nature of the troubleshooter at this time) and will need permissions to run Windows PowerShell. Installing the troubleshooter: First, you will need to download the troubleshooter ZIP file, which you can find attached to this post. Installing the EMTshooter is pretty easy. Just drop the 4 files from the ZIP file into 1 folder, rename them to .ps1 and run EMTshooter.ps1 from a normal (and local) PowerShell window. I personally just created a shortcut for it on my desktop with the following properties: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noexit -command ". 'C:\EMTshooter\EMTshooter.ps1'" However, as most users probably won't run this more than a few times you might not need or want an icon. Just remember that EMTshooter.ps1 is the main script to run. Providing feedback: As I mentioned before, the troubleshooter is still a work in progress. If you wish to provide feedback on it, please post a comment to this blog post. I will be monitoring it and replying as time allows, and also making updates to the troubleshooter if needed. If you run into errors that are not covered by the troubleshooter, please run the troubleshooter, reproduce the error through it and send me the transcript.txt file (you will find it in the folder where the 4 scripts have been placed), along with what you did to resolve the error (if the problem has been resolved). My email is sbryant AT Microsoft DOT com. Errors currently covered: Connecting to remote server failed with the following error message : The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL. Connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command 'Discover-ExchangeServer -UseWIA $true -SuppressError $true'. Connecting to remote server failed with the following error message : The WinRM client received an HTTP status code of 403 from the remote WS-Management service. Connecting to the remote server failed with the following error message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. Connecting to remote server failed with the following error message : The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: Connecting to remote server failed with the following error message : The WS-Management service does not support the request. Connecting to remote server failed with the following error message : The WinRM client cannot process the request. The WinRM client tried to use Kerberos authentication mechanism, but the destination computer - Steve BryantTroubleshooting Exchange 2010 Management Tools startup issues
EDIT 12/7/2010: For additional help resolving those issues, please see our newer blog post Resolving WinRM errors and Exchange 2010 Management tools startup failures. In this blog post, we will be highlighting some of the most common errors that may be seen when attempting to open the Exchange Management tools (Exchange Management Console and Exchange Management Shell). To start off, you first need to be aware that in Exchange 2010, all management is done via Remote PowerShell, even when opening the Management Tools on an Exchange server. Where this differs from Exchange 2007 is that there is now a much larger dependency on IIS, as Remote PowerShell requests are sent via the HTTP protocol and use IIS as the mechanism for connections. IIS works with the WinRM (Windows Remote Management) service, and the WSMan (Web Services for Management) protocol to initiate the connection. When you click on the Exchange Management Shell shortcut, a Remote PowerShell session is opened. Instead of simply loading the Exchange snap-in (as we did with Exchange 2007), PowerShell connects using IIS to the closest Exchange 2010 server via WinRM. WinRM then performs authentication checks, creates the remote session and presents to you the cmdlets that you have access to via RBAC (Role Based Access Control). Since all Remote PowerShell connections go through IIS, we have identified some of the most common errors that may be exhibited when attempting to open the Exchange Management tools along with the most common causes of those errors and how to address these issues. We have attempted to list these in order of frequency. Issue: Connecting to remote server failed with the following error message: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic. Possible causes: 1. Remote PowerShell uses Kerberos to authenticate the user connecting. IIS implements this Kerberos authentication method via a native module. In IIS Manager, if you go to the PowerShell Virtual Directory and then look at the Modules, you should see Kerbauth listed as a Native Module, with the dll location pointing to C:\Program Files\Microsoft\Exchange Server\v14\Bin\kerbauth.dll. If the Kerbauth module shows up as a Managed module instead of Native, or if the Kerbauth module has been loaded on the Default Web Site level (instead of, or in addition to, the PowerShell virtual directory), you can experience this issue. To correct this, make sure that the Kerbauth module is not enabled on the Default Web Site, but is only enabled on the PowerShell virtual directory. The entry type of "Local" indicates that the Kerbauth module was enabled directly on this level, and not inherited from a parent. 2. If the WSMan module entry is missing from the global modules section of the C:\Windows\System32\Inetsrv\config\ApplicationHost.config file, as follows: <globalModules> <add name="WSMan" image="C:\Windows\system32\wsmsvc.dll" /> This will result in the WSMan module displaying as a Managed module on the PowerShell virtual directory. To correct this, make sure that the WSMan module has been registered (but not enabled) at the Server level, and has been enabled on the PowerShell virtual directory. 3. If the user that is attempting to connect is not Remote PowerShell enabled. To check if a user is enabled for Remote PowerShell, you need to open the Exchange Management Shell with an account that has been enabled, and run the following query. (Get-User <username>).RemotePowershellEnabled This will return a True or False. If the output shows False, the user is not enabled for Remote PowerShell. To enable the user, run the following command. Set-User <username> -RemotePowerShellEnabled $True Issue: Connecting to the remote server failed with the following error message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more information, see the about_Remote_Troubleshooting Help topic. Possible Causes: 1. The http binding has been removed from the Default Web Site. A common scenario for this is if you are running multiple web sites, and attempting to set up a redirect to https://mail.company.com/owa by requiring SSL on the Default Web Site, and creating another web site to do the redirect back to the SSL-enabled website. Remote PowerShell requires port 80 to be available on the Default Web Site. If you want to set up an automatic redirect to /owa and redirect http requests to https, you should follow the instructions located at http://technet.microsoft.com/en-us/library/aa998359(EXCHG.80).aspx and follow the directions under the section "For a Configuration in Which SSL is Required on the Default Web Site or on the OWA Virtual Directory in IIS 7.0". 2. The http binding on the Default Web Site has been modified, and the Hostname field configured. To correct this issue, you need to clear out the Hostname field under the port 80 bindings on the Default Web Site. Issue: Connecting to remote server failed with the following error message: The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic. It was running the command 'Discover-ExchangeServer -UseWIA $true -SuppressError $true'. In addition, you may see the following warning event in the System log: Source: Microsoft-Windows-WinRM EventID: 10113 Level: Warning Description: Request processing failed because the WinRM service cannot load data or event source: DLL="%ExchangeInstallPath%Bin\Microsoft.Exchange.AuthorizationPlugin.dll" Possible Causes 1. The ExchangeInstallPath variable may be missing. To check this, go to the System Properties, Environment variables, and look under the System variables. You should see a variable of ExchangeInstallPath with a value pointing to C:\Program Files\Microsoft\Exchange Server\V14\. 2. The Path of the Powershell virtual directory has been modified. The PowerShell virtual directory must point to th e \Program Files\Microsoft\Exchange Server\v14\ClientAccess\PowerShell directory or you will encounter problems. Issue: Connecting to remote server failed with the following error message: The connection to the specified remote host was refused. Verify that the WS-Management service is running on the remote host and configured to listen for requests on the correct port and HTTP URL. For more information, see the about_Remote_Troubleshooting Help topic. Possible Causes: 1. Make sure the MSExchangePowerShellAppPool is running. If it is, try recycling the Application Pool and check for errors or warnings in the Event logs. 2. Make sure that the user that is trying to connect is Remote PowerShell Enabled (see the first error for details on how to check this). 3. Make sure WinRM is properly configured on the server. a. Run WinRM Quick Config on the server and ensure that both tests pass and no actions are required. If any actions are required, answer Yes to the prompt to allow the WinRM configuration changes to be made. b. Run WinRM enumerate winrm/config/listener and ensure that a listener is present for the HTTP protocol on port 5985 listening on all addresses. Issue: Connecting to remote server failed with the following error message: The WinRM client received an HTTP status code of 403 from the remote WS-Management service. Possible Causes: 1. The "Require SSL" option has been enabled on the PowerShell Virtual Directory. To resolve this, remove the "Require SSL" option from this Virtual Directory. The Exchange Management Tools connect over port 80, not 443, so if Require SSL is set, when a connection is attempted on port 80, IIS will return a 403 error indicating SSL is required. - Ben Winzenz, Solange TrombiniOutlook clients receive error 0x8004010f when downloading the Offline Address Book
EDIT 10/01/2008: To read the updated (and more comprehensive) guidance for troubleshooting errors 0x8004010f, please go here. There are a multiple reasons for why an Outlook client can receive the 0x8004010f sync error. Unfortunately, 0x8004010f is just a generic MAPI error and will show up for a variety of problems. Here is what the error looks like under err.exe (Microsoft Exchange Server Error Code Look-up Tool): C:\WXP\system32>err 0x8004010f # for hex 0x8004010f / decimal -2147221233 ecNotFound ec.h ecAttachNotFound ec.h ecUnknownRecip ec.h ecPropNotExistent ec.h MAPI_E_NOT_FOUND mapicode.h # 5 matches found for "0x8004010f" This is what the error looks like from the sync log from within Outlook: 12:45:53 Synchronizing Mailbox <dgoldman> 12:45:53 Done 12:45:54 Microsoft Exchange offline address book 12:45:54 0x8004010f Some of the most common reasons for Outlook clients to receive the 0x8004010f error with regards to downloading the OAB are listed below. Most of these are documented and I have linked articles to each of these to help everybody out. Please note that these solutions can change a small bit depending on unknown factors in a company's environment. An administrator decommissioned the last Exchange server in a site and never pushes replicas to another Exchange server. A new OAB is created in the active directory and the information store never reads the active directory during its maintenance schedule. This will result in the OAB files never being generated and the Outlook client will fail to download anything. The information store has an invalid EntryID that points to the legacy EX:/ folders. Again there is nothing for the client to download. An outlook client logging in from one domain to another domain and attempting to log in to another users mailbox. The OAB was never generated or some OAB folders are missing from the public folder store. Multiple OAB Version folders exist of the same type. Clients are attempting to download the OAB files from a public folder store that have not received the replicated updates. The offline address book list object has a missing address list. The offline address book list object has an incorrect address list. Send/As changes in the store affect users accounts with no mailbox full rights to another mailbox. If you are seeing this error on an Exchange 2007 server and your OAB is generated by an Exchange 2007 server, please make sure of the following: 1. Make sure that you have added the replicas of OAB to the Exchange 2007 server 2. Make sure public folder replication is working. 3. Make sure the OAB is public folder enabled and you have OAB Version 2, OAB Version 3 and OAB Version 4 checked off so your legacy clients can download the OAB files from the public folder store. 4. Make sure that if you are using an Outlook 2007 client, your OAB is Web Distribution enabled and the OAB files have been replicated over to the Client Access Server. For more information on this process please see this blog: http://blogs.msdn.com/dgoldman/archive/2006/10/23/outlook-client-fails-to-download-the-oab-with-error-0x8004011b.aspx and http://blogs.msdn.com/dgoldman/archive/2006/08/25/How-Exchange-2007-OAB-Files-are-replicated-to-a-Client-Access-Server-for-download.aspx 5. If you are removing your last Exchange 2003 server from the org, please make sure that you follow our documentation on this process. - Dave Goldman
