Configuring Hot-Add and Large Memory for Exchange 2003

[The_Exchange_Team]

INTRODUCTION

This is the second flash about Windows 2003 kernel memory issues and Exchange 2003. The first flash provided technical background. This flash discusses important hardware issues that may affect Windows kernel memory and cause Exchange instability. The next flash (available in about 10 days) will explain the effect of large user security tokens on kernel memory usage.

EXECUTIVE SUMMARY

Servers that implement hot-add memory or that are configured with more than 4 gigabytes of physical RAM can require large amounts of additional kernel memory to support these resources. This reduces the kernel memory available for applications. Even though disk, network and processor bandwidth may be more than adequate, lack of kernel resources can be the bottleneck that limits application scalability. 

The impact of this issue for Exchange is primarily on large Exchange 2003 mailbox or public folder servers that host or accept connections from thousands of clients. In some cases, sub-optimal large memory configuration on a large Exchange 2003 server can reduce the server's carrying capacity by half and cause serious server instability.

If kernel memory issues are not adequately addressed, it may be necessary to reduce the server load by removing large numbers of mailboxes or to reduce the number of concurrent client connections to the server.

Symptoms of kernel memory exhaustion include:

• Slow performance
• Server crashes or cluster failovers
• Errors that report complete exhaustion of system page table entries (PTEs) or kernel pool memory

This flash provides recommendations for optimizing the use of kernel memory on large memory hardware running Exchange Server 2003. Other applications may have different requirements and recommendations.

CALL TO ACTION

The Exchange Server Best Practices Analyzer Tool (ExBPA) has been updated in configuration file version 2.9.0.1 to identify and make recommendations for optimizing the majority of hardware platforms affected by these issues. All customers should run ExBPA and implement its memory optimization recommendations for Exchange Server 2003 running on Windows 2003.

Customers may also manually implement the optimization recommendations described below in this flash.

NOTE: Windows 2000 and Exchange 2000 are not greatly impacted by these issues because (1) Windows 2000 does not support these hardware features as fully as does Windows 2003, and (2) Exchange 2000 does not run on Windows 2003.

FREQUENTLY ASKED QUESTIONS (FAQ)

Which hardware configurations make additional demands on kernel memory?

Microsoft has identified two increasingly common hardware features that have significant impact on Windows kernel memory:

• Support for more than 4 gigabytes of physical RAM. Adding RAM above 4 gigabytes allows a server running a 32 bit version of Windows to take advantage of PAE memory extensions. For more information about the advantages and limitations of PAE please refer to this article in the Microsoft Knowledge Base:

Large memory support is available in Windows Server 2003 and in Windows 2000

http://support.microsoft.com/kb/283037

• Hot-add RAM. This feature allows you to add RAM while a server is running. The majority of servers that support hot-add RAM enable this feature by default.

Windows kernel memory must be allocated to support addressability for these new memory features. In the case of hot-add memory, the kernel impact is proportional not just to the actual amount of RAM installed, but to the theoretical maximum that could be installed.

To give you an idea of how significant a problem this can be, a server that enables support for 64 gigabytes of hot-add RAM can reduce a server's initial available kernel paged pool memory by more than 30%.

What should I do to increase the amount of kernel memory available for Exchange?

IMPORTANT NOTE: The following recommendations are in addition to the general memory tuning recommendations described in this Microsoft Knowledge Base article:

How to optimize memory usage in Exchange Server 2003

http://support.microsoft.com/kb/815372

Recommendations 1 and 2 below should be followed for all Exchange servers.

Recommendations 3 and 4 apply only to servers that support hot-add memory. If you are running Windows 2003 without a service pack, you should evaluate both recommendation 3 and recommendation 4 and implement them both if they are applicable. If you are running Windows 2003 with Service Pack 1 or later you do not need to implement recommendation 3. You should evaluate recommendation 4 and implement it if it is applicable.

1. Download and run the Exchange Best Practices Analyzer Tool (ExBPA). This tool can be downloaded here:

http://www.microsoft.com/technet/prodtechnol/exchange/downloads/2003/analyzers/default.mspx

ExBPA detects Exchange server configuration problems and recommends solutions and optimizations. ExBPA checks for memory configuration optimizations beyond those described in this flash, and includes best practices and optimizations beyond those recommended in the Microsoft Knowledge Base. It is a good idea to run ExBPA after making configuration changes to ensure that the changes were correctly applied.

2. Install no more than 4 gigabytes of physical RAM on any Exchange server. A server dedicated to running Exchange 2003 will not benefit significantly from installation of more than 4 gigabytes of physical RAM.

If the server already has more than 4GB of memory installed, adjust the server settings according to the following documentation: http://technet.microsoft.com/en-us/library/aa997419.aspx

3. Disable hot-add memory at the hardware level, if your server meets all of the following conditions:

• The server hardware supports hot-add memory.
• You are running Windows 2003 Enterprise Edition or Datacenter Edition. This recommendation does not apply for Windows 2003 Standard Edition.
• The /3GB switch has been set in the server's boot.ini file
• No service pack for Windows 2003 has been installed.

Contact the manufacturer of the server for specific instructions for disabling hot-add memory. If hot-add memory cannot be disabled, then ensure that the /NOPAE switch is set in the server's boot.ini file.

4. Disable hot-add memory at the operating system level, if your server meets all of the following conditions:

• The server hardware supports hot-add memory.
• You are running Windows 2003 Enterprise Edition or Datacenter Edition. This recommendation does not apply for Windows 2003 Standard Edition.
• The /3GB switch has been set in the server's boot.ini file.

This recommendation applies regardless of whether you have installed any service pack.

Setting the following registry key will disable Windows memory reservations for RAM not already installed in the server. You must reboot the server after setting this key before it will take effect. If you install additional RAM in the server, another reboot will be required before the additional RAM is recognized by Windows.

HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management

Value: DynamicMemory

Type: REG_DWORD

Value Data: 1

For Windows 2003 with no service pack installed, the DynamicMemory setting will be ignored if the /3GB switch is set. The DynamicMemory setting works if the /3GB switch is absent. This problem has been corrected in Service Pack 1 for Windows 2003.

Because Exchange back-end servers typically run with the /3GB switch, the DynamicMemory setting will be ineffective for many Exchange servers running on Windows 2003 with no service pack installed. In these cases, you must follow recommendation 3 above to control hot-add RAM on Windows 2003 with no service pack installed.

NOTE: We still recommend that you configure the DynamicMemory setting on Windows 2003 with no service pack. Although the setting may have no immediate impact, it will be immediately effective after upgrading to a Windows service pack. You may also set this key if you are unsure whether your server supports hot-add memory. The presence of the key will do no harm on servers that do not support hot-add memory.

How can I monitor kernel memory on my server?

For more information about determining how much initial kernel memory is allocated on your server, and how it is allocated between various kernel memory pools, please see this article in the Microsoft Knowledge Base:

The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003

http://support.microsoft.com/kb/894067

For general information about monitoring and troubleshooting memory usage on an Exchange server, please refer to this white paper:

Troubleshooting Exchange Server 2003 Performance

http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/e2k3perf.mspx

What if I want to leave hot-add memory enabled? Or what if I host other applications on the same server that could make use of additional RAM?

Exchange 2003 is compatible with both hot-add and PAE memory. There is no hard requirement that either of these features be disabled on an Exchange server. The problem is that these features contend with Exchange and other applications and services for kernel resources. Rather than disabling these features, you could also choose to reduce the application load on the server until kernel memory issues are no longer encountered.

If Exchange does not use PAE, then why shouldn't I disable PAE on every Exchange server?

If an Exchange 2003 server is running on Windows 2003 Service Pack 1, you should not disable PAE.

PAE must be enabled to take advantage of the support for hardware-based Data Execution Prevention (DEP) that was introduced in Windows 2003 Service Pack 1. DEP is a security feature that prevents several common buffer overrun attacks from executing, regardless of whether particular software is vulnerable to the attack.

For more information about DEP, please refer to this Microsoft Knowledge Base article:

A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003

http://support.microsoft.com/default.aspx?scid=kb;en-us;875352

Windows 2003 Service Pack 1 will detect AMD or Intel processors that support DEP and enable the PAE kernel even if the /NOPAE switch is set in the server's boot.ini file. (To disable PAE on servers that support DEP, you must also set the /EXECUTE switch in the server's boot.ini file.)

For versions of Windows prior to Windows 2003 Service Pack 1, Microsoft's advice is the opposite: You should not enable PAE on Exchange servers running on previous versions of Windows.

There is negligible benefit for Exchange performance and memory management from enabling PAE on previous versions of Windows, and there may be significant drawbacks in terms of kernel memory management.

If other applications on an Exchange server installed on a previous version of Windows require PAE, then you must install this update before enabling PAE:

http://support.microsoft.com/kb/834628

If your server supports PCI Express chipsets, and your server has 4 GB or more of RAM installed, then PAE must be enabled to prevent 256 megabytes or more of server RAM from being "hidden." This is because PCI Express reserves a large block of memory for its own use. Enabling PAE will allow the operating system to replace this lost memory with RAM mapped above the 4 gigabyte boundary. Enabling this memory will not provide a significant improvement in Exchange performance.

Can I increase kernel memory by removing the /3GB switch from the server's boot.ini file?

Microsoft does not recommend removing the /3GB switch if your server meets the criteria for which this switch is recommended. These criteria are described here:

How to optimize memory usage in Exchange Server 2003

http://support.microsoft.com/kb/815372

Removing the /3GB switch will increase the initial amounts of memory available for kernel memory pools. For example, it will increase the allocation for paged pool memory from approximately 250 megabytes to 350 megabytes. However, this benefit will come at the expense of the loss of 1 gigabyte of user mode virtual memory.

If you remove the /3GB switch for a large Exchange mailbox server, you must also reduce the number of mailboxes hosted on the server below the point where virtual memory fragmentation errors are reported. Removing the /3GB switch reduces the size of the Exchange cache buffers by half and this will impact performance.

- Mike Lee

[#2]

Two questions on a great post.
#1 - My lack of knowledge - I thought you could use PAE or 3GB, but not both. This is really confusing the consultants out there that I talk to because we understood this to be an either/or scenario. Can you go into a little more depth as to how the two different memory mangement methods can work together on the same box?
FYI - Most servers I run into do have the /NOPAE switch set in the BOOT.INI. So it sounds like that doesn't do anything w/o the EXECUTE command?

#2 - Do you generally recommend that dedicated mailbox servers with 4GB of RAM have the ESE cache set to 1.2 gig (the max recommended)? I have heard differing opinions on this, and I generally leave systems at the default setting until I know for a fact they should be modified. the ExBPA does't really make a recommendation either way (I wish it did). I know we can monitor the server like the KB suggests, but I was interested in a general overall recommendation.

[#3]

All our exchange servers installed on
HP ML370 G4 servers with PCI Express support
and 4 Gb RAM.
We have windows 2003 sp1 and exchange 2003 sp1 installed (both enterprise editions).
Is PAE switch must be enabled in this situation? Is /3GB switch must be enabled too?
Thanks.

[#4]

Dan...

#1 - No, you can set the /3GB switch and have the PAE kernel loaded together. However, both of these squeeze kernel memory, and that's why it's important to tune the server appropriate. The PAE kernel uses wider page table entries (PTE) so that it can accommodate physical memory above the 4GB boundary. As an aside, these wider PTEs are also needed to support the Data Execution Prevention (DEP) feature in Windows 2003 SP1. The /3GB switch modifies the 4GB of virtual address space allocation for each process so instead of each app getting 2GB of user and 2GB of kernel space, the app gets 3GB of user and only 1GB of kernel space. Because Exchange uses a single store.exe process, it requires the larger user address space to scale. You can think of the PAE kernel as changeing physical memory allocation, and the /3GB switch changing virtual memory allocation.

#2 - Great question. As a general rule, we recommend leaving the cache size at its default of ~900M. While it's true that a scale-up server will perform better with a larger cache size (i.e. 1.2GB), you run the risk of depleting the virtual address space which can cause a service outage. As with most tuning parameters, you have to balance performance vs. stability. You are correct that ExBPA doesn't provide a concrete recommendation right now. This is mainly because there continue to be different schools of thought on tuning this parameter. Here at Microsoft, we're running one large server with a 1.2GB cache (>4000 users), whereas all other Exchange servers are running with the default cache size. This one server is running closer 'to the wire' in terms of stability, and we have seen one instance where the lack of virtual memory caused an outage. NOTE: Just because you have 4GB of physical RAM in a server, it doesn't mean that it's safe to increase cache settings. At the end of the day, you'll run out of virtual memory in the process before physical memory. The amount of virtual memory is constrained by the 32-bit OS, which is a big reason why we're looking to 64-bit with E12.

[#5]

Igor. Because the ML370 supports hot-add memory and DEP, you'll find that the PAE kernel will automatically load with Windows 2003 SP1. Therefore, you don't need to explicitly set the /PAE switch. However, assuming that this is a mailbox server, you will need to set the /3GB switch manually in the Boot.ini. Your best bet is to run the ExBPA tool (www.exbpa.com) as this will programmatically analyze many aspects of your server (in fact, it analyzes nearly 2500 settings and variables), and make recommendations for tuning.

[#6]

#1. We should leave the /NOPAE switch in the BOOT.INI files since most customers do not put more than 4GB of RAM in their boxes, and they don't honestly care about DEP on dedicated & locked down servers. I.E. They do not appear to get any benefit from adding it on 4GB machines other than DEP support.

#2. Understood - I think the fact that you all are only running the mod on one server speaks volumes. I would be interested to know if the virtual memory issues you all saw had anything to do with the fact that the server was on a cluster (assuming it was since ITG clusters every production mail server from what I know).

Thanks for the continued advice and support Paul!

[#7]

EXECUTIVE SUMMARY
 
A few weeks ago, we released three CXP flashes on the subject of...