Information Store timeout detection in Exchange 2007 SP3

Exchange Server 2007 Service Pack 3 introduces a new feature that monitors the Information Store service for long running transactions. Transactions are now timed and if a transaction lasts longer than 60 seconds (hardcoded), it's considered to be timed out. The transaction isn’t terminated; it’s just flagged as taking too long.

This monitoring has been added to help report on the health of the Information Store. There is a myriad of reasons for long running transactions. Some of these reasons are explained in Understanding the Performance Impact of High Item Counts and Restricted Views.

Update: Timeout detection and reporting was introduced in Exchange 2010 RTM and backported to Exchange 2007 SP3. See Understanding the Exchange 2010 Store -> Time-Out Detection and Reporting.

In isolation, an individual long running transaction may or may not be of concern. If the transaction doesn’t involve any locking, it will proceed in isolation without harm (assuming CPU and Memory are scaled appropriately). If it does use locking however, it can be quite harmful to the experience of other clients as they wait for the locked resource to be released.

If the prevalence of long transactions increases over time, the monitoring more than likely indicates that there are various problems (data corruption, high item counts, disk performance, memory pressure, CPU pressure).

The new timeout detection logs entries in the Application event log at the following three levels of severity:

• Server Level
• Database Level
• Mailbox Level

Each level is associated with a scope and threshold: Server – any 20 threads; Database – 10 threads per database; Mailbox – 5 threads per mailbox. The following entries are logged in the event log for each level:

[Server Level – 20+ threads]

MessageID=10025
Source=MSExchangeIS
Severity=Error
Facility=General (6)
Language=English
There are %1 RPC requests that take abnormally long time to complete. It may be indicative of performance problems with your server.

• [Database Level – 10+ threads per database]

  MessageID=10026
  Source=MSExchangeIS
  Severity=Error
  Facility=General (6)
  Language=English
  There are %1 RPC requests for the database "%2" that take abnormally long time to complete. It may be indicative of performance problems with your server.

• [Mailbox Level – 5+ threads per mailbox]

  MessageID=10027
  Source=MSExchangeIS
  Severity=Error
  Facility=General (6)
  Language=English
  There are %1 RPC requests for the mailbox "%2" on the database "%3" that take abnormally long time to complete. It may be indicative of performance problems with your server.

When an Event Log entry is logged, the MSExchangeIS\RPC Request Timeout Detected performance counter is incremented:

This feature can be disabled by setting this registry value to 1. The default is enabled (0 or not set).

Path: HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem
Name: DisableTimeoutDetection
Type: DWORD
Value: 1

Jun Yan