Several months ago, we wrote a post on
Troubleshooting Server Hangs
. At the end of that post, we provided some basic steps to follow with respect to server hangs. The last step in the list was following the steps in
KB Article 244139
to prepare the system to capture a complete memory dump for analysis. Now that you have the memory dump, what exactly are you supposed to do with it? That will be the topic of today’s post – more specifically, dealing with server hangs due to resource depletion. We discussed various aspects of resource depletion including Paged and NonPaged pool depletion and System PTE’s. Today we’re going to look at Pool Resource depletion, and how to use the Debugging Tools to troubleshoot the issue.
If the server is experiencing Non paged pool (NPP) memory leak or a Paged pool (PP) memory leak you are most likely to see the following event id’s respectively in the System Event log:
Event ID: 2019
Details: The server was unable to allocate from the system nonpaged pool because the pool was empty.
Event ID: 2020
Details: The server was unable to allocate from the system Paged pool because the pool was empty
Let’s load up our memory dump file in the Windows Debugging tool (WINDBG.EXE). If you have never set up the Debugging Tools and configured the symbols, you can find instructions on the
Debugging Tools for Windows Overview
page. Once we have our dump file loaded type
in the prompt to display the Virtual Memory Usage for the system. The output will be similar to what is below:
As you can see, this command provides details about the usage of Paged and NonPaged Pool Memory, Free System PTE’s and Available Physical Memory. As we can see from the output above, this system is suffering from excessive NonPaged Pool usage. There is a maximum of 128MB of NonPaged Pool available and 121MB of this NonPaged Pool is in use:
Our next step is to determine what is consuming the NonPaged Pool. Within the debugger, there is a very useful command called
. We use this command to find the Pool Tag that is consuming our NonPaged Pool. The
command will list out NonPaged Pool consumption, and
lists the Paged Pool consumption. A quick note here; the output from the
commands could be very lengthy as they will list all of the tags in use. To limit the display to the Top 10 consumers, we can use the /t10 switch:
!poolused /t10 2
0: kd> !poolused 2
Sorting by NonPaged Pool Consumed
Tag Allocs Used Allocs Used
R100 3 9437184 15 695744 UNKNOWN pooltag 'R100', please update pooltag.txt
MmCm 34 3068448 0 0 Calls made to MmAllocateContiguousMemory , Binary: nt!mm
LSwi 1 2584576 0 0 initial work context
TCPt 28 1456464 0 0 TCP/IP network protocol , Binary: TCP
File 7990 1222608 0 0 File objects
Pool 3 1134592 0 0 Pool tables, etc.
Thre 1460 911040 0 0 Thread objects , Binary: nt!ps
Devi 337 656352 0 0 Device objects
Even 12505 606096 0 0 Event objects
naFF 300 511720 0 0 UNKNOWN pooltag 'naFF', please update pooltag.txt
Once the tag is identified we can use the steps that we outlined in our previous post,
An Introduction to Pool Tags
to identify which driver is using that tag. If the driver is out of date, then we can update it. However, there may be some instances where we have the latest version of the driver, and we will need to engage the software vendor directly for additional assistance.
That brings us to the end on this post – in Part Three, we will discuss using Task Manager and the Debugging Tools to troubleshoot Handle Leaks which may be causing Server Hangs.