First published on TECHNET on Jul 31, 2008
The Operations Manager 2007 Management Pack tracks memory usage of the MonitoringHost.exe process via the private bytes counter. By default, the agent will be automatically restarted if memory usage for MonitoringHost.exe is above 100MB. On large Exchange 2007 systems, this threshold may be too low. The recommended threshold is 600MB.
This is why:
On Exchange 20007 servers the .Net runtime (CLR) gets loaded into MonitoringHost.exe. The way Windows performance counters work is that to read a performance counter from an application a DLL provided by that application is loaded into the process. Since some of the Exchange services (for example Hub Transport) are written in managed code, the CLR performance counter client is loaded into the MonitoringHost.exe process for reading those counters. The .net runtime performance counter client implementation loads the CLR to handle reading the counters from the managed service.
The CLR handles memory management via a garbage collection system rather than traditional native code where you explicitly allocate and de-allocate memory. In a garbage collection system the application doesn't explicitly release memory. Periodically the garbage collector will run to see what objects are no longer being used and then free up the memory for those objects. The CLR garbage collector is designed to monitor system load so that if there is no memory pressure then it doesn't need to run as often. When the garbage collector runs it will internally release objects and make space for new objects. However, it may not trim the amount of memory it has requested from the operating system if it detects there is no memory pressure on the system. If the system starts seeing memory pressure then the CLR may start decreasing the amount of memory it has asked for.
On large systems like Exchange 2007 servers, Operations Managers default private byte threshold of 100MB may end up being too small. Since there is limited memory pressure on these systems, the CLR may commit more than 100MB of memory and not de-allocate until there is more memory pressure on the system.
Symptoms
Health Service restarts even after installing the following hotfixes (monitoringhost.exe) on X64 Systems running Exchange 2007.
950853 A memory leak occurs when you monitor Exchange Server 2007 by using the MOM 2007 agent in System Center Operations Manager 2007
http://support.microsoft.com/default.aspx?scid=kb;EN-US;950853
951979 Problems occur on a management server that is running System Center Operations Manager 2007 Service Pack 1 when certain management packs are installed
http://support.microsoft.com/default.aspx?scid=kb;EN-US;951979
You'll see the following event being logged in the Operations Manager Event log.
Event Type: Warning
Event Source: Health Service Script
Event Category: None
Event ID: 6026
Date: 6/26/2008
Time: 8:59:55 PM
User: N/A
Computer: SGBD012512
Description:
LaunchRestartHealthService.js : Launching Restart Health Service. Monitoring Host exceeded ProcessPrivate Bytes threshhold.
To update the agent restart threshold, perform the following steps
1. In the authoring section of the console, find the Monitoring Host Private Bytes Threshold Rule for the Agent class
2. Select to override the value for a group. To override for all Exchange 2007 Servers, select the All Computers in Management Pack: Microsoft.Exchange.2007 Computer Group. It is also possible to override this for single instances of agents or your own groups.
3. Set the Threshold to 629145600 (600MB). Note that you should not store the override in the Default Management Pack
4. Perform the same steps for the Health Service Private Bytes Threshold Monitor (again for the Agent class; this will prevent the agent health state from changing to critical)
(Thanks to Marc Reyhner and Brian Zoucha for most of the content of this article)
