Blog Post

Exchange Team Blog
10 MIN READ

Understanding Exchange 2007 Memory Usage and its use of the Paging File

The_Exchange_Team's avatar
Aug 07, 2008

Lately, we've been getting a lot of questions related to Exchange 2007 memory consumption and why Exchange uses so much of the paging file. Well, there are some valid reasons why and some misconceptions that need to be corrected. Comparing memory usage between Exchange 2007 and Exchange 2003 First, let's start with why Store.exe uses so much RAM. If we take a step back in time to the Exchange 2003 era, this blog was quite active on how to tune memory on an Exchange 2003 server. I am not going to go in to specifics or great detail, but one thing to call out is that on the 32-bit architecture, we were limited to addressing 4GB of virtual memory on any given server. So essentially, any 32bit program could address up to 4GB of virtual memory. This address space is typically split so that applications could address 2GB of memory and 2GB would be for the Windows Executive. By adding the /3GB switch in the boot.ini, applications could now access up to a 3GB virtual address space and lower the Executive down to addressing only 1GB, essentially halving the memory that is can be addressed or is available for kernel drivers, paged/non-paged pool memory, PTE's etc. The larger the load that you put on a server has the potential to exhaust important resources on the server which eventually causes a failure or server outages. Any type of memory leak in these areas could be detrimental to the stability of the server. With the numerous posts on this blog regarding the /3GB, /USERVA and /PAE switches, what drivers to update, how to tweak memory to allow Exchange to run optimally, etc., it was always a juggling act to try to keep Exchange servers online. Cluster servers were especially touchy as http.sys will start rejecting HTTP connections when non-paged pool hit a certain level, thus causing cluster failovers. Due to these constraints of the 32-bit architecture, we needed a way to balance I/O based applications on the server. Essentially, there are two types of I/O for the Windows operating system, Buffered and Unbuffered I/O. For Buffered I/O, only portions of the working set can be contained in memory; those portions not in use are retained in the page file. For Unbuffered I/O, the entire working set can be contained in memory. On any given system, we needed to maintain a balance so that each would get the right resources that they need. This is one of the main reasons why we needed to put a maximum on the ESE database cache to leave us enough room for the system cache. In most implementations, the default of 900MB of database cache was used which was sufficient for most deployments, but some implementations had the need for more cache increasing it to a maximum of 1.2GB of memory on systems of 2GB or more of physical RAM. This maintained the balance that was necessary so that the system cache could perform efficiently. So you may ask, why did I go through all of the above to explain how Exchange 2003 worked? Well, one of the aspects of Exchange 2007 that people ask about is the difference between Exchange 2007 memory usage to Exchange 2003. Indeed, the memory usage is much higher,and there are very good reasons for this. Let me explain further. Exchange 2007 Design Considerations During the design phase of Exchange 2007, we needed to find a way to scale out servers further than Exchange 2003 did due to the rising demands of email usage in companies. For many companies, email is their life line and if an exchange server should go down, they could be losing money at an astronomical rate. One of the  performance bottlenecks in Exchange 2003 was disk latencies which was mostly due to the memory constraints of the 32-bit platform. Ultimately, -, we had to read more from disk than what was available in memory. This required a well performing disk subsystem to handle the sustained I/O necessary to respond to client requests in a timely manner. In many cases where disks were the underlying problem, a re-design of the disk subsystem was necessary to allow the current amount of IOPS that were being generated on the server to be handled by the disks. This could range from adding more spindles to the disk arrays, rebuilding different RAID solutions to provide enough sustained throughput based on current user load. Why we learned to love the 64-bit Platform With the 64-bit architecture being available, we made a conscious decision that Exchange 2007 would only be supported in production on 64-bit hardware. This was mainly done to overcome the limitations that the 32bit platform had. The design goal here was to reduce cost of Exchange deployments in which storage turned out to be the number one driver of cost and support overhead. By moving to the 64-bit platform, we are able to reduce our I/O requirements thus bringing us back to easier balancing of I/O and capacity requirements. Now, one of the results of being able to address more memory is the capability to cache more memory for each application. To allow this to happen for Exchange, the ESE Database Cache Size limit was removed to allow Exchange the ability cache more pages in memory. Accessing pages in memory is extremely fast, so the more data we cache in memory, the less we actually have to read andwrite from the disk. When following our best practice guidance around storage group deployment and memory sizing, Exchange 2007 reduces the amount of I/O required overall. This gave us rather huge performance gains.  For more information regarding I/O improvements on the 64-bit platform for Exchange, see the following blog post http://msexchangeteam.com/archive/2006/09/08/428860.aspx. With no limitation in Database Cache for Exchange 2007, the memory usage for the store process will naturally be much higher compared to what Exchange 2003 used due to the many benefits discussed earlier. Memory allocation for the ESE cache is dynamic, so Jet will use all available memory on a system and return it as needed.  For example, if a server had, let's say, 16GB of physical RAM installed, the database cache could consume approximately 14GB of memory, roughly 2GB less than the total amount of physical RAM in the server, leaving enough memory for the system cache or other applications running on the server. The algorithm is relatively simple for the ESE Database Cache. The DBA (Dynamic Buffer Allocation) will grow the cache by comparing the amount of I/O the databases are doing with the amount of I/O the system is doing in terms of hard page faults (Memory\Transition pages repurposed/sec). So if the database is doing more I/O, and Exchange is hitting hard page faults, we will increase the cache.  If the database is doing less I/O than the hard page faults, then we will decrease the cache. The goal is to keep the ESE cache in balance with the disk cache so we don't end up paging ourselves to death. In some cases, on larger servers, Online maintenance during the night could cause up to a 4GB swing in memory for the Database cache alone due to the aggressive nature of how we touch pages during this process. (It may also be prudent to stagger your OLM schedules to assist with this.) Each server configuration is different, so it is important to have a monitoring solution in place or a Performance monitor log to track memory usage over time if memory pressure is occurring. If memory usage gets to a point where some driver or application attempt to allocate a large block of contiguous memory, and there is not one available, then the OS may decide to start trimming working sets which is a really bad thing performance wise. I've discussed this on my own blog http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx and possible remediation's to work around those type scenarios if you fall in to this bucket. Other Server Memory Considerations So we've talked a lot about memory usage for ESE database cache, but this cache alone is not the only significant consumer of memory on an Exchange server, it is just the majority. From an Exchange perspective, we have other processes going on such as Content Indexing, log shipping in continuous replication and underlying .NET applications that Exchange makes use of now. Most of Exchange has been rewritten in managed code using the .NET framework except for Store, DAV, and DSAccess. Managed code offers some significant advantages over non-managed code such as the ability to compile applications in real-time so that you don't have to worry about running your application on different architectures or platforms and the ability to manage memory efficiently. .NET applications use the Common Language Runtime (CLR) to allow easier coding in different languages because they share the same runtime. Any code that you develop with a language compiler that targets the runtime is called managed code as mentioned here: http://msdn2.microsoft.com/en-us/library/ddk909ch.aspx. Reasons for Memory Paging One of the primary reasons for memory paging in Exchange 2007 is Content Searching, which uses the system cache since this is buffered. Performing more searches increases the file cache you have which in turn reduces the amount of IOPS that are generated. Content Indexing itself does not make use of the file cache as this is designed to work on a memory structure that is not affected by the file cache. Master merges of index data does use a small amount of the file cache, but it very short lived. Database I/O is actually unbuffered, so this not affected by the system cache. This is why the ESE cache doesn't continuously grow (ESE cache I/O is less than the system cache I/O).  It's being balanced out by the content searching I/O (and potentially other applications that use buffered memory I/O) and thus the system cache. CCR log shipping is also unbuffered while log files are being copied over SMB, so this is also not affected by the system cache. Now couple all of this memory management with other 3rd party applications and drivers that are installed on the server which are all competing for the same resources, you can see that this is vastly different than what Exchange 2003 used to be. There is actually no comparison at all. How Exchange makes use of the Paging File So this brings up the second part of this document where I will now talk about the paging file. So you may ask, if I have so much memory in the server, why is a paging file needed at all? Our system requirements talk about needing one for getting a memory dump on the server, but what does Exchange really use it for?  I get this very question asked very frequently. In certain cases, it is true that Windows 2003 does not require a paging file. It is however completely incorrect that Exchange does not need a paging file for normal runtime operations when running on top of that OS. There is always going to be a need to page out unused or rarely used memory to the page file so that we can make maximum use of RAM. For example, there could be a large amount of VM pages used during system startup that are not useful under main load.  Unexpected transient conditions could cause large upward swings in memory that could cause us to spill pages of memory in to the paging file. There are enough processes outside of the cache manager that consume a significant amount of memory in Exchange 2007 (e.g. ESE and all processes that utilize the CLR) that there is a need for a substantially sized paging file as the system balances the memory footprint of the competing processes.  Without a paging file (or with one that is too small) there is a very high chance that applications will encounter OOM (out of memory) conditions and both the performance and the functionality of the server will suffer and may lead to a possible blue screen. The cost of having a 32GB paging file on a server with 32GB of physical RAM on disk is pretty minimal, so having a properly configured paging file (RAM+10MB) will be able to sustain any unexpected growths in memory where we need more commit charge to handle current load. The other benefit is that if we should get ourselves in to a working set trimming issue, we would at least have the ability to flush these pages out to the paging file and then page them back in when a particular page needs to be accessed again. Excessive working set trimming on an Exchange 2007 server is not normal and should be investigated. Again, check out my blog post on working set trimming problems and why this is bad for performance. Why does Exchange allocate so much memory in the paging file? Another question that generally comes up is why does Exchange allocate so much memory in the paging file?  Depending on the amount of memory that is consumed for the store process, we will need to at least allocate enough memory in the paging file to match the current working set that is in RAM on the server. If a system wide working set trim should occur, we would be able to page the current pages in the store working set out to the backing store or paging file. It is important to monitor Memory\Pages/sec on the server to detect if this working set trimming problem is causing any performance related problems on the server. You should be able to see by adding Process\Working Sets a sharp decline in this value along with a sharp increase in Memory\Pages/sec. With the other processes on the server possibly using parts of the system or file cache, there is always a balancing act trying to compete for memory on any given server. The good news is that Windows 2008 has made some significant changes in memory management to help prevent these system wide working set trims and to manage memory more efficiently. With server applications having the need for using more RAM as technology advances, it is imperative that memory management works effectively to prevent any negative performance problems on any given server. For more information on some of these memory management changes in Windows 2008, see http://www.microsoft.com/whdc/system/sysinternals/memmgt.mspx. I hope this helps clarify some of the reasons why Exchange uses more memory overall and provide clarity for these most common questions. A special thanks goes to Andrew Goodsell, Matt Gossage, and Ross Smith IV for their invaluable information in this area and Nick Basile for tech reviewing this document. Mike Lagase Support Escalation Engineer

Published Aug 07, 2008
Version 1.0
  • Thanks for sharing this,
    Triggers me to ask the following:

    I've been tought to calculate the required storage design for E2K3 and E2K7 on W2K3 or W2K8 in basically the same manner to get a better understanding of the calculator sheet developed by dev. team and to be able to validate/tune outcome. This calculation method does not really take into account the Cache Hit Ratio that reduces storage I/O requirements, during normal Exchange operation. There are good reasons for this, because we need to take into account server startup / peak load / and recovery on the storage level (raid rebuilds), etc. It is important to be able to address this "exceptional" situation (assuming your server is running normally most of the time), but this way we are essentially always overestimating the real disk I/O requirements. So, to get back to my question, does it make sense to include some kind of baseline or measured Cache Hit Ratio for Read I/O into your storage calculation (and different figures based on OS and Exchange levels/combination), to determine a more average "Host Read IOPS" requirement (Writes must always be written to disk), resulting in for example the ability to choose a cheaper storage solution? (I.e. raid 5 instead of raid 10, or slower disks) And what would the new function then really look like to determine this.

    So to summarize: We say that with E2K7 we use more RAM for a good reason, we say that we can use cheaper storage solutions, because of reduced disk I/O requirements, but the calculations we use do not seem to be well enough aligned with the statement we make.

    Feel free to correct me, just throwing up a ball.
    Thanks again!
  • Hi Lunik,

    Our storage calcuations do indeed take into account the amount of RAM required to reduce read I/O during normal runtime operations.  We use the following formula:

    ((0.0048 × M) × (D ^ -0.65)) + (0.00152 × M) = total database IOPS

    where M is the number of messages and D is the database cache, per user.

    We do not recommend sizing storage for the rare conditions (like cold boot) as that should be a relatively small window (on systems that follow best practice guidance we see cold boot operations take about 15 minutes for the cache to reach optimum state).

    If you have any other questions, you can send them to the storage calculator alias at strgcalc AT microsoft DOT com.

    Ross
  • Temp and Pagefile
    --------------------
    What do you recommend I do with these two files/directories?

    http://support.microsoft.com/kb/197379 suggests the page file is left alone on the boot volume, but also to create another on a separate volume

    http://technet.microsoft.com/en-us/library/aa998306(EXCHG.80).aspx suggests to keep the temp directory off of transaction log luns and OS luns.

    http://technet.microsoft.com/en-us/library/bb738154(EXCHG.80).aspx says “To improve performance, the TMP folder should not be on the same LUN as the page file and operating system.”

    What does this all mean?  The 3 documents together seem to imply there should be separate drives for the:
    -OS
    -tmp
    -page file
    -transaction log(s)
    -information store(s)

    It also suggests everything should be high performance, so that means so many disks.

    If I load up the server with RAM can I just leave the temp and page file in the default locations on my exchange 2007 servers?
  • Thanks Ross,
    I do understand you calculate the amount of RAM to reduce foremostly Read I/O during runtime. But once you have RAM how do you take into account for example a healthy (high) average Cache Hit Ratio of 95% back into the same storage calculation?
    When i calculate storage with the "complex math" method (taught by Jud Doran) there is RAM or % Cache Hit variable in that equation... And also, i thought we always calculate for 100% concurrency, to also cope with cold boot, rebuilds, peak load, etc.
    I'll mail you a few more underlining examples...
    Lunik
  • Correction: i of course mean there is no RAM or % Cache Hit variable in that complex math equation....
  • Dimitri,

    That statement is great in a perfect world, but there is no way of knowing when a page in a working set might spill over in to the paging file without some hard testing. Exchange does a good job of keeping pages in its own working set, but there are always variables out there that might cause a page to go to the standby list and then eventually be paged out of memory.

    To play it safe, the recommendations that we currently provide is to protect from worst case scenarios, otherwise, if something went wrong, the store would become unstable and possibly crash which is what we don't want. If we can protect our working set backed by a paging file, we could at least recover, albeit not quickly on a slow drive, but in due time the Store process would eventually start responding.

    In certain instances, it is very hard to maintain balance as other 3rd party applications can behave badly on a server and consume it's fair share of memory/handles causing undesired results.

    Most of the problems that I have come across is due to some other application consuming memory causing Exchange to not behave the way you want. It is a perception thing as Exchange is unfortunately the victim of something else going wrong on your server.
  • So if I understand correctly, the page file recommendations come from:
    A) The need to accomodate the potential scenario in which the memory manager is triggered to page the entire working set of all of the processes to disk.
    B) The need to accomodate leaky applications that might run the server out of memory.  By having a huge page file we can fill both the server RAM and the pagefile before we have to deal with the problem.

    Thank you for clarifying exactly what the concerns are that have lead to this recommendation.  That will help me weigh the costs of consuming that much disk space with the potential risks to my environment.