Getting to know the MmSt Pool Tag

First published on TECHNET on Sep 23, 2011

Hello readers,

We have briefly discussed the MmSt pool tag before, but only tangentially. This post is to go much further in depth to the use of this tag and common issues we see related to it.

What is the MmSt pool tag?

The MmSt pool tag is used by the Memory Manager when reserving memory for section prototype PTEs. A prototype PTE is a structure that maps the physical location of a page or set of pages to a memory mapped file. The MmSt allocations reside in Paged Pool memory. MmSt allocations are most commonly seen when the system is under heavy file I/O and has many open files mapped into the memory cache. The Cache Manager should reduce its MmSt allocations when paged pool memory pressure occurs, but this may not always happen quickly enough to prevent some issues.

Common Symptoms

The symptoms for issues dealing with high MmSt paged pool usage are common to most other paged pool depletion issues including:

• Server hangs
• Event ID 2020 – Paged Pool depletion
• Event ID 333 - An I/O operation initiated by the Registry failed
• Users are unable to RDP or access file shares on the server
• Backups or large file copies fail with insufficient resources
  • Error 1450: Insufficient resources exist to complete the requested server
  • Error 1130: Not enough server storage is available to process this command
• SQL backups or consistency checks performing heavy file I/O will fail to complete

Most Common Root Causes for high MmSt pool usage

• The Memory Manager cannot trim paged pool memory quickly enough before allocations start failing
• Heavy file I/O because too many files or large files are opened simultaneously
• Outlook .PST/Lotus Notes .NSF files being stored on the server and opened over the network
• VSS (Volume Shadow Copy) snapshots aren’t dismounting
• Large $MFT due to fragmentation or many files on the volume

Identifying the MmSt pool tag as the issue

You will need to use a tool such as Poolmon, Process Explorer, a memory dump, or a LiveKD dump to first confirm that you have a paged pool issue, and second to determine if MmSt is the tag consuming most of the paged pool memory. If it is in fact the MmSt tag causing the server to run out of paged pool memory then you will typically want to collect data to confirm this and find root cause.

• Poolmon logging
  • Indicates total paged pool usage and which tags are consuming the most
  • How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
  • http://support.microsoft.com/kb/177415
• Performance Monitor log
  • Confirm high pool usage
  • Find the trend in pool memory
  • Determine if the trend is related to high file I/O
• Complete or Kernel memory dump
  • Check if the system failed to allocate paged pool memory and which tags were using the most paged pool memory
• Process Explorer –
  • Check the limits of the paged and nonpaged pool on the machine
• RamMap
  • Check for memory mapped file usage and $MFT size (Note: runs on Vista+ OSes only)
  • Introduction to using RAMMAP
• LiveKD
  • LiveKD can be used to collect a memory dump without bugchecking the machine

Poolmon Logging

Poolmon will be one of the primary tools you use to determine which pool tag is consuming most of the memory. A poolmon log indicates how much pool memory is being used by all paged and nonpaged pool memory tags. The poolmon log should run until the symptoms occur, and should end up looking something like this:

Pool Used:

NonPaged Paged

Tag Allocs     Used               Allocs Used

MmSt 0             0                   8684 293147416 Mm section object prototype ptes , Binary: nt!mm

This indicates that the Paged pool tag MmSt is using approximately 293MB . Depending on the maximum paged pool limit on the machine, this could be too much and cause the system to fail additional pool allocations.

Process Explorer

Process Explorer can be used live on the your machines to determine the maximum paged and nonpaged pool memory limits without needing to dump the machine. It is the least intrusive method of accomplishing this. Internet connectivity to the Public Microsoft Symbol Server is required for this to work successfully.

Memory Dump Files

Collecting a memory dump file from the server while it is in a problematic state or consuming high paged pool memory will confirm if MmSt is the responsible pool tag. The debugger will indicate that there have been pool allocation failures if at some point the machine could not allocate pool memory.

• Requires bugchecking the machine
• LiveKD can be used as a substitute instead of bugchecking the machine
  • Some data cannot always be retrieved from dumps generated by LiveKD due to the manner in which LiveKD collects the dump
• Debugger Commands
  • !vm
    • Displays Virtual Memory values
  • !poolused /t 10 4
    • Displays the top 10 paged pool consumers
• Can be used to check the versions of the core binaries
• Can be used to check $MFT size
• Can be used to check if .PST files are open on the server

Sample from a dump file:

!vm

• *** Virtual Memory Usage ***
• Physical Memory:     8386920 (  33547680 Kb)
• Page File: \??\C:\pagefile.sys
• Current:  16384000 Kb  Free Space:  16346204 Kb
• Minimum:  16384000 Kb  Maximum:     16385024 Kb
• Available Pages:     6756395 (  27025580 Kb)
• ResAvail Pages:      8190718 (  32762872 Kb)
• Locked IO Pages:           0 (         0 Kb)
• Free System PTEs:      48900 (    195600 Kb)
• ******* 40192 system cache map requests have failed ******
• Modified Pages:        89760 (    359040 Kb)
• Modified PF Pages:     89632 (    358528 Kb)
• NonPagedPool Usage:    70527 (    282108 Kb)
• NonPagedPool Max:     523263 (   2093052 Kb)
• PagedPool 0 Usage:     26793 (    107172 Kb)
• PagedPool 1 Usage:     14819 (     59276 Kb)
• PagedPool 2 Usage:     12600 (     50400 Kb)
• PagedPool 3 Usage:     13044 (     52176 Kb)
• PagedPool 4 Usage:     12444 (     49776 Kb)
• PagedPool Usage:       79700 (    318800 Kb)
• PagedPool Maximum:523264 (   2093056 Kb) <-Paged pool limit for this machine
• ********** 33134 pool allocations have failed **********

3: kd> !poolused /t 10 4
Sorting by Paged Pool Consumed

NonPaged                             Paged
Tag             Allocs             Used          Allocs           Used
MmSt             0                     0             3696056   334902032    Mm section object prototype ptes , Binary: nt!mm
MmPd 2 525328 2470 225509872 Mm page table commitment bitmaps , Binary: nt!mm

The values above indicate that the MmSt Paged pool tag had 3696056 memory allocations and is consuming approximately 334MB of Paged Pool memory.

RAMMAP

RamMap , a tool from Sysinternals, can be used to determine why high MmSt pool usage is occurring. RAMMap is an advanced physical memory usage analysis utility for Windows Vista and higher.

• View the Mapped File value to determine if memory mapped files are causing the high MmSt usage
  • Go to the Physical Pages tab and sort the Use column, scroll down to Mapped File to determine which files are being mapped into memory, such as .PSTs

RAMMAP Screenshot showing Mapped File and Metafile (MFT):

RAMMAP screenshot showing memory mapped files.

Troubleshooting Steps and Resolutions

Implementing the steps in the first two bullets below will resolve a majority of these issues:

• Apply the latest hotfixes for core OS files to fix known issues.
  • Ntoskrnl.exe
  • Ntfs.sys
  • Srv.sys
  • Storport.sys
  • Install latest VSS rollup if VSS is being used
• Adjust Maximum Paged Pool Memory and Trimming
  • We often hit the paged pool limit because the trimming process can’t keep up once paged pool hits the default of 80% limit before we start trimming. Adjusting the PoolUsageMaximum from KB312362 changes the default throttle so that we can force trimming to start earlier. Adjusting the PagedPoolSize value will tell the system to reserve as much paged pool memory as possible at boot time. These two registry keys will often alleviate the pressure on the Memory Manager and give it sufficient time to trim paged pool before nearing the maximum limit.
  • Tuning Paged Pool Trimming and Maximum Paged Pool Values
    • 312362 Server is unable to allocate memory from the system paged pool
      http://support.microsoft.com/default.aspx?scid=kb;EN-US;312362
    • PoolUsageMaximum from KB312362 can safely be set to a decimal value of 40 which will cause trimming to start when paged pool reaches 40% of the maximum limit
    • PagedPoolSize should be set to 0xFFFFFFFF to ensure the system reserves as much paged pool memory as possible
  • Remove 4GT Tuning if it is not required
    • Check if /3GB is enabled on the machine and confirm it is necessary for the roles of the server.
    • Removing /3GB (only if it is not required on the server) will increase the paged pool limit dramatically. Adding the /USERVA switch will also slightly increase the maximum amount of paged pool.

Determining Root Cause

In addition to providing the latest binaries and adjusting the registry keys for paged pool memory as stated above, data will need to be collected to determine root cause and additional troubleshooting steps if required.

• PST Files Stored on a File Server
  • When PST files are opened across the network they will consume paged pool memory when the .PST file is loaded into memory.
  • RAMMAP or a memory dump can be used to determine if this is the cause
  • Opening PST files over the network from a file server is an unsupported scenario
    • KB297019 - Limits to using personal folders (.pst) files over LAN and WAN links
    • AskPerf Blog on Network stored PST files
  • Supported solutions per KB297019
    • Exchange 2010
    • Remote Desktop Session Host
    • VDI (Virtual Desktop Infrastructure)
• Heavy File I/O
  • Common Causes
    • Overloaded file servers
    • Antivirus scans
    • Backups
    • Very large number of files being opened or copied
  • Use Performance Monitor logging to confirm if this is the issue
    • Check file I/O and paged pool usage for similar trends
  • Recommendations
    • Set virus scans and backups to run when there is less load on the system
    • It may be necessary to add an additional server to handle the file I/O load if adjusting the PoolUsageMaximum value doesn’t resolve the issue.
    • x64 OS may be the best recommendation if the server is unable to handle the load without being depleted of paged pool
• Large $MFT
  • If the $MFT is too large it can deplete paged pool
    • http://support.microsoft.com/kb/961095
  • The MFT contains information about all of the files on the volume
    • http://msdn.microsoft.com/en-us/library/aa365230(VS.85).aspx
  • RAMMAP or a dump file can be used to determine the size of the $MFT
    • View the Metafile value in RAMMAP to determine if the $MFT is very large
    • Using the Debugger: http://blogs.msdn.com/b/ntdebugging/archive/2008/05/08/tracking-down-mmst-paged-pool-usage.aspx
  • Solutions
    • Move to x64 OS
    • Reduce the number of volumes
    • Spread the load to additional servers
• VSS Snapshots (Volume Shadow Copy)
  • Install the latest VSS updates
  • Reduce the size of the VSS Store
  • Reduce the number of snapshots
  • KB903234 - An update is available to optimize the way that the Shadow Copy Client accesses shadow copies in Windows Server 2003 and in Windows XP
• Custom Applications
  • Application developers often write their applications to open thousands or millions of files simultaneously causing I/O stress
  • Use the unbuffered flag when opening files to reduce pool memory usage
    • http://support.microsoft.com/kb/99794
    • http://msdn.microsoft.com/en-us/library/cc644950(v=VS.85).aspx

Useful Links

Poolmon Overview:
http://technet.microsoft.com/en-us/library/cc737099(WS.10).aspx

Finding Poolmon.exe:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=18546

How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks - http://support.microsoft.com/kb/177415

RAMMAP Askperf Blog:
http://blogs.technet.com/b/askperf/archive/2010/08/13/introduction-to-the-new-sysinternals-tool-rammap.aspx

Process Explorer:
http://technet.microsoft.com/en-us/sysinternals/bb896653

How to generate a kernel dump file or a complete memory dump file in Windows Server 2003:
http://support.microsoft.com/kb/972110

Rami Munder