INTRODUCTION
This is the second flash about Windows 2003 kernel memory issues and Exchange 2003. The first flash provided technical background. This flash discusses important hardware issues that may affect Windows kernel memory and cause Exchange instability. The next flash (available in about 10 days) will explain the effect of large user security tokens on kernel memory usage.
EXECUTIVE SUMMARY
Servers that implement hot-add memory or that are configured with more than 4 gigabytes of physical RAM can require large amounts of additional kernel memory to support these resources. This reduces the kernel memory available for applications. Even though disk, network and processor bandwidth may be more than adequate, lack of kernel resources can be the bottleneck that limits application scalability.
The impact of this issue for Exchange is primarily on large Exchange 2003 mailbox or public folder servers that host or accept connections from thousands of clients. In some cases, sub-optimal large memory configuration on a large Exchange 2003 server can reduce the server's carrying capacity by half and cause serious server instability.
If kernel memory issues are not adequately addressed, it may be necessary to reduce the server load by removing large numbers of mailboxes or to reduce the number of concurrent client connections to the server.
Symptoms of kernel memory exhaustion include:
- Slow performance
- Server crashes or cluster failovers
- Errors that report complete exhaustion of system page table entries (PTEs) or kernel pool memory
This flash provides recommendations for optimizing the use of kernel memory on large memory hardware running Exchange Server 2003. Other applications may have different requirements and recommendations.
CALL TO ACTION
The Exchange Server Best Practices Analyzer Tool (ExBPA) has been updated in configuration file version 2.9.0.1 to identify and make recommendations for optimizing the majority of hardware platforms affected by these issues. All customers should run ExBPA and implement its memory optimization recommendations for Exchange Server 2003 running on Windows 2003.
Customers may also manually implement the optimization recommendations described below in this flash.
NOTE: Windows 2000 and Exchange 2000 are not greatly impacted by these issues because (1) Windows 2000 does not support these hardware features as fully as does Windows 2003, and (2) Exchange 2000 does not run on Windows 2003.
FREQUENTLY ASKED QUESTIONS (FAQ)
Which hardware configurations make additional demands on kernel memory?
Microsoft has identified two increasingly common hardware features that have significant impact on Windows kernel memory:
- Support for more than 4 gigabytes of physical RAM. Adding RAM above 4 gigabytes allows a server running a 32 bit version of Windows to take advantage of PAE memory extensions. For more information about the advantages and limitations of PAE please refer to this article in the Microsoft Knowledge Base:
Large memory support is available in Windows Server 2003 and in Windows 2000
http://support.microsoft.com/kb/283037
- Hot-add RAM. This feature allows you to add RAM while a server is running. The majority of servers that support hot-add RAM enable this feature by default.
Windows kernel memory must be allocated to support addressability for these new memory features. In the case of hot-add memory, the kernel impact is proportional not just to the actual amount of RAM installed, but to the theoretical maximum that could be installed.
To give you an idea of how significant a problem this can be, a server that enables support for 64 gigabytes of hot-add RAM can reduce a server's initial available kernel paged pool memory by more than 30%.
What should I do to increase the amount of kernel memory available for Exchange?
IMPORTANT NOTE: The following recommendations are in addition to the general memory tuning recommendations described in this Microsoft Knowledge Base article:
How to optimize memory usage in Exchange Server 2003
http://support.microsoft.com/kb/815372
Recommendations 1 and 2 below should be followed for all Exchange servers.
Recommendations 3 and 4 apply only to servers that support hot-add memory. If you are running Windows 2003 without a service pack, you should evaluate both recommendation 3 and recommendation 4 and implement them both if they are applicable. If you are running Windows 2003 with Service Pack 1 or later you do not need to implement recommendation 3. You should evaluate recommendation 4 and implement it if it is applicable.
- Download and run the Exchange Best Practices Analyzer Tool (ExBPA). This tool can be downloaded here:
http://www.microsoft.com/technet/prodtechnol/exchange/downloads/2003/analyzers/default.mspx
ExBPA detects Exchange server configuration problems and recommends solutions and optimizations. ExBPA checks for memory configuration optimizations beyond those described in this flash, and includes best practices and optimizations beyond those recommended in the Microsoft Knowledge Base. It is a good idea to run ExBPA after making configuration changes to ensure that the changes were correctly applied.
- Install no more than 4 gigabytes of physical RAM on any Exchange server. A server dedicated to running Exchange 2003 will not benefit significantly from installation of more than 4 gigabytes of physical RAM.
If the server already has more than 4GB of memory installed, adjust the server settings according to the following documentation: http://technet.microsoft.com/en-us/library/aa997419.aspx
- Disable hot-add memory at the hardware level, if your server meets all of the following conditions:
- The server hardware supports hot-add memory.
- You are running Windows 2003 Enterprise Edition or Datacenter Edition. This recommendation does not apply for Windows 2003 Standard Edition.
- The /3GB switch has been set in the server's boot.ini file
- No service pack for Windows 2003 has been installed.
Contact the manufacturer of the server for specific instructions for disabling hot-add memory. If hot-add memory cannot be disabled, then ensure that the /NOPAE switch is set in the server's boot.ini file.
- Disable hot-add memory at the operating system level, if your server meets all of the following conditions:
- The server hardware supports hot-add memory.
- You are running Windows 2003 Enterprise Edition or Datacenter Edition. This recommendation does not apply for Windows 2003 Standard Edition.
- The /3GB switch has been set in the server's boot.ini file.
This recommendation applies regardless of whether you have installed any service pack.
Setting the following registry key will disable Windows memory reservations for RAM not already installed in the server. You must reboot the server after setting this key before it will take effect. If you install additional RAM in the server, another reboot will be required before the additional RAM is recognized by Windows.
HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
Value: DynamicMemory
Type: REG_DWORD
Value Data: 1
For Windows 2003 with no service pack installed, the DynamicMemory setting will be ignored if the /3GB switch is set. The DynamicMemory setting works if the /3GB switch is absent. This problem has been corrected in Service Pack 1 for Windows 2003.
Because Exchange back-end servers typically run with the /3GB switch, the DynamicMemory setting will be ineffective for many Exchange servers running on Windows 2003 with no service pack installed. In these cases, you must follow recommendation 3 above to control hot-add RAM on Windows 2003 with no service pack installed.
NOTE: We still recommend that you configure the DynamicMemory setting on Windows 2003 with no service pack. Although the setting may have no immediate impact, it will be immediately effective after upgrading to a Windows service pack. You may also set this key if you are unsure whether your server supports hot-add memory. The presence of the key will do no harm on servers that do not support hot-add memory.
How can I monitor kernel memory on my server?
For more information about determining how much initial kernel memory is allocated on your server, and how it is allocated between various kernel memory pools, please see this article in the Microsoft Knowledge Base:
The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003
http://support.microsoft.com/kb/894067
For general information about monitoring and troubleshooting memory usage on an Exchange server, please refer to this white paper:
Troubleshooting Exchange Server 2003 Performance
http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/e2k3perf.mspx
What if I want to leave hot-add memory enabled? Or what if I host other applications on the same server that could make use of additional RAM?
Exchange 2003 is compatible with both hot-add and PAE memory. There is no hard requirement that either of these features be disabled on an Exchange server. The problem is that these features contend with Exchange and other applications and services for kernel resources. Rather than disabling these features, you could also choose to reduce the application load on the server until kernel memory issues are no longer encountered.
If Exchange does not use PAE, then why shouldn't I disable PAE on every Exchange server?
If an Exchange 2003 server is running on Windows 2003 Service Pack 1, you should not disable PAE.
PAE must be enabled to take advantage of the support for hardware-based Data Execution Prevention (DEP) that was introduced in Windows 2003 Service Pack 1. DEP is a security feature that prevents several common buffer overrun attacks from executing, regardless of whether particular software is vulnerable to the attack.
For more information about DEP, please refer to this Microsoft Knowledge Base article:
A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003
http://support.microsoft.com/default.aspx?scid=kb;en-us;875352
Windows 2003 Service Pack 1 will detect AMD or Intel processors that support DEP and enable the PAE kernel even if the /NOPAE switch is set in the server's boot.ini file. (To disable PAE on servers that support DEP, you must also set the /EXECUTE switch in the server's boot.ini file.)
For versions of Windows prior to Windows 2003 Service Pack 1, Microsoft's advice is the opposite: You should not enable PAE on Exchange servers running on previous versions of Windows.
There is negligible benefit for Exchange performance and memory management from enabling PAE on previous versions of Windows, and there may be significant drawbacks in terms of kernel memory management.
If other applications on an Exchange server installed on a previous version of Windows require PAE, then you must install this update before enabling PAE:
http://support.microsoft.com/kb/834628
If your server supports PCI Express chipsets, and your server has 4 GB or more of RAM installed, then PAE must be enabled to prevent 256 megabytes or more of server RAM from being "hidden." This is because PCI Express reserves a large block of memory for its own use. Enabling PAE will allow the operating system to replace this lost memory with RAM mapped above the 4 gigabyte boundary. Enabling this memory will not provide a significant improvement in Exchange performance.
Can I increase kernel memory by removing the /3GB switch from the server's boot.ini file?
Microsoft does not recommend removing the /3GB switch if your server meets the criteria for which this switch is recommended. These criteria are described here:
How to optimize memory usage in Exchange Server 2003
http://support.microsoft.com/kb/815372
Removing the /3GB switch will increase the initial amounts of memory available for kernel memory pools. For example, it will increase the allocation for paged pool memory from approximately 250 megabytes to 350 megabytes. However, this benefit will come at the expense of the loss of 1 gigabyte of user mode virtual memory.
If you remove the /3GB switch for a large Exchange mailbox server, you must also reduce the number of mailboxes hosted on the server below the point where virtual memory fragmentation errors are reported. Removing the /3GB switch reduces the size of the Exchange cache buffers by half and this will impact performance.
You Had Me at EHLO.