Exchange 2007 managed services might time out during certificate revocation checks

[The_Exchange_Team]

Introduction

We have been working on a problem that surfaced with the release of Exchange 2007 Rollup 5. A number of customers reported that some of their Exchange 2007 managed services did not start automatically after Rollup application, however they would start manually. In most of these cases the computer in question was not connected to the Internet. Investigation showed that the problem was the timing of the Windows Service Control Manager (SCM) and validation of all of the certificates associated with a service within the SCM timeout. In the case of computers that were connected to the Internet, the problem seemed to be network latency, as the problem in those cases can happen only intermittently.

Why this happens

To sum it up: the problem did not manifest itself until Exchange 2007 managed binaries were signed with two certificates (because the original one was expiring). For Exchange 2007 RTM this occurred when RU5 was released. Because now .Net Framework Common Language Runtime (CLR) attempted to validate two certificates by connecting to http://crl.microsoft.com, the process took longer, to the point where SCM timeout would pass and services would fail to start up automatically. In environments with limited or no Internet connectivity, this might fail every time. In other cases, this might fail intermittently based on current network state and load.

Workarounds

KB article 944752 Exchange 2007 managed code services do not start after you install an update rollup for Exchange 200... is being revised to only contain the recommended workaround, which is to modify the services configuration files. The modification will to prevent the CLR from going onto the Internet in the first place. This is accomplished by adding a section to the managed executable configuration file as outlined in Bypassing the Authenticode Signature Check on Startup (MSDN .NET Security Blog, Shawn Farkas).

Security concerns

The first question that people usually ask about is if making the modification to .config files compromises server security. In case of Exchange Server 2007, this is not relaxing security at all.  From a security point of view, what it's doing is saying "assume that the Authenticode signature is invalid." Let's say that you have these three assemblies:

• An assembly without an Authenticode signature.
• One with an invalid Authenticode signature
• One with a valid signature that has this option set in the Configuration file

All three behave exactly the same.  The CLR will load them, and not give Publisher evidence to the assembly.  Since the assembly doesn't get the Publisher evidence, any trust decisions that were being made based upon the validity of the signature will no longer apply - so if the assembly is only trusted due to its signature it will lose its trust status due to the config switch.  Exchange 2007 assemblies however, do not use this mechanism as the only one to determine if the assembly should be run or not, so disabling it does not compromise Exchange.

To make this .config file modification

The resolution to this problem involves editing the configuration files associated with the Exchange Services to use a switch which was added to CLR 2.0 SP1 (which is by default present in .NET framework V3.5.) You can update .NET Framework 2.0 or 3.0 by installing the fix from http://support.microsoft.com/kb/942027/. That hotfix contains the fix outlined in http://support.microsoft.com/default.aspx/kb/936707. Before continuing with this procedure, save a copy of your existing configuration files to a safe location. In the event of an error in the configuration file, the applicable service will fail to start. Create configuration files for all managed code Exchange 2007 services to resolve this issue. Please note that we do not suggest going to create/modify the .config files unless your server is actually impacted by this problem. To create an application configuration file that contains this configuration setting, follow these steps:

1. Create a file, and then name the file the <ApplicationName>.exe.config file. 2. In a text editor, open the file that you created in step 1. 3. Add the following code to the file:

<configuration> <runtime> <generatePublisherEvidence enabled="false"/> </runtime> </configuration>

4. Save the changes to the file to the applicable directory (see below for a list of files and locations - the .config files should be saved into the same folder where the affected executable is). If the configuration file already exists for a specific service, just add the "<generatePublisherEvidence enabled="false"/>" line to the runtime options section in the file.

Exchange 2007 Services/Apps which come with a .config file that might need to be updated: Bin\EdgeTransport.exe Bin\ExBPA.exe Bin\ExBPACmd.exe Bin\ExTRA.exe Bin\Microsoft.Exchange.Cluster.ReplayService.exe Bin\Microsoft.Exchange.EdgeSyncSvc.exe Bin\Microsoft.Exchange.Monitoring.exe Bin\Microsoft.Exchange.Search.ExSearch.exe Bin\Microsoft.Exchange.ServiceHost.exe Bin\MSExchangeMailboxAssistants.exe Bin\MSExchangeMailSubmission.exe Bin\MSExchangeTransportLogSearch.exe ClientAccess\PopImap\Microsoft.Exchange.Imap4.Exe ClientAccess\PopImap\Microsoft.Exchange.Pop3.Exe Exchange 2007 Services for which you need to create a new .config file (unless it was already created for another reason): Bin\Microsoft.Exchange.AntispamUpdateSvc.exe Bin\MsExchangeFDS.exe Bin\MSExchangeTransport.exe Troubleshooting If a service fails to start after modifying or creating the config files, the most likely reason is an XML syntax error or an incorrect value. In both of these cases the service will fail to start and you'll get an error similar to this example from the Exchange 2007 Edge Transport Service: Event Type:      Error Event Source:    MSExchangeTransport Event Category:  Process Event ID:        14004 Description: The worker process has failed to load application configuration file: System.Configuration.ConfigurationErrorsException: Configuration system failed to initialize ---> System.Configuration.ConfigurationErrorsException: The 'generatePublisherEvidence' start tag on line 4 does not match the end tag of 'runtime'. Line 5, position 6. (C:\Program Files\Microsoft\Exchange Server\Bin\edgetransport.exe.config line 5) ---> System.Xml.XmlException: The 'generatePublisherEvidence' start tag on line 4 does not match the end tag of 'runtime'. Line 5, position 6. at System.Xml.XmlTextReaderImpl.Throw(Exception e) at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag) at System.Xml.XmlTextReaderImpl.ParseEndElement() at System.Xml.XmlTextReaderImpl.ParseElementContent() at System.Xml.XmlTextReaderImpl.Skip() at System.Configuration.XmlUtil.StrictSkipToNextElement(ExceptionAction action) at System.Configuration.BaseConfigurationRecord.ScanSectionsRecursive(XmlUtil xmlUtil, String parentConfigKey, Boolean inLocation, String locationSubPath, OverrideModeSetting overrideMode, Boolean skipInChildApps) at System.Configuration.BaseConfigurationRecord.ScanSections(XmlUtil xmlUtil) at System.Configuration.BaseConfigurationRecord.InitConfigFromFile() --- End of inner exception stack trace --- at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal) at System.Configuration.BaseConfigurationRecord.ThrowIfParseErrors(ConfigurationSchemaErrors schemaErrors) at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey) --- End of inner exception stack trace --- at System.Configuration.ConfigurationManager.GetSection(String sectionName) at System.Configuration.ConfigurationManager.get_AppSettings() at Microsoft.Exchange.Transport.TransportAppConfig.GetConfigBool(String label, Boolean defaultValue) at Microsoft.Exchange.Transport.TransportAppConfig.ResourceManagerConfig.Load() at Microsoft.Exchange.Transport.TransportAppConfig.Load() at Microsoft.Exchange.Transport.Main.Program.Run(String[] args) Additional information We're continuing to investigate this problem and will provide more information as it becomes available. - Ed Beck, Nino Bilic