Slow Response Automated Data Collection using FREB and debug diag

[Jawahar Ganesh S]

Recently I have come across many situations where we observe slowness and it is a challenge to collect the right data at the right time either because no one is available when the issue is happening to collect the data or issue is very intermittent.

The normal way we actually follow is as below.

1. Open the Debug Diagnostic tool.

2. Go to the Processes tab, right-click the worker process corresponding to the right application pool and select “Create Full Userdump”.

3. Repeat Step 2. Three times with a gap of 5-10 seconds in between and collect 3 sets of dumps for us.

But the above plan always doesn’t work if the issue is very intermittent and resolves on its own.

Another way to automate the collection as specified in the below blog.

http://blogs.msdn.com/b/benjaminperkins/archive/2013/12/02/using-procdump-and-failed-request-tracing...

http://blogs.msdn.com/b/webtopics/archive/2009/08/04/iis7-and-above-using-freb-to-capture-dumps-for-...

Over here I will be discussing how we can automate this collection to capture multiple dumps in a different way.

Note: Test all the below rules in your QA and testing server before porting the changes to your production box.

Create Freb rule:

1. Go to intemgr.exe, click on the site or application for which you are experiencing slowness or hang and go to Failed Request Tracing rules.

2. Click on Add rule and follow the below steps.

a. Select “All content” and click next. If we know that this is for a specific page, we can mention the page name as well by selecting Custom.

b. Select Time-Taken in seconds option and specify the time (if the page is taking around 30 seconds, you can specify 10 or 15 so that we start capturing the freb and dumps 15 seconds past execution till the end). Click Next

c. Click Finish

3. On the right hand pane at the site level under Configure go to Failed Request Tracing and enable it. You can configure it to the required location or leave it to the default.

Note: Please make sure that we only have the Slowness rule in Failed Request tracing for the specific application or web site or application pool for which we are going to configure Debug Diag rule.(Having multiple rules might make debug diag capture irrelevant data)

Steps to Create Debug Diag Rule:

1. We will use Debug Diagnostic 2.0 update 2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool from https://www.microsoft.com/en-us/download/details.aspx?id=49924 (Please un-install all other versions of Debug Diagnostic tool before installing the latest version)

2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.

a. Download PDBDownloader.exe from https://github.com/rajkumar-rangaraj/PDB-Downloader/releases

b. Run the PDBDownloader.exe and click on open files

c. If this is a 64 bit application/process then navigate to the path C:\Windows\System32\inetsrv\ and select iisfreb.dll. Click on Start

d. If this is a 32 bit application/process then navigate to the path C:\Windows\SysWow64\inetsrv\ and select iisfreb.dll. Click on Start

e. The tool will download the public symbols of iisfreb.dll and copy this to C:\Symbols folder and we are good to proceed with the rule creation

Note: Debug Diag 2.0 Update 2 uses the symbol folder as C:\Symbols. Debug Diag 1.2 or Debug Diag 2.0 Update 1 uses the symbol folder as C:\symcache. If you have Debug Diag 1.2 or Debug Diag 2.0 Update 1 then change the file location in PDBDownloader.exe to C:\symcache. If folder does not exist please create it.

Note: The executable requires internet connection to download the symbols. If you don’t have internet connection on the server, then copy the C:\Windows\System32\inetsrv\iisfreb.dll to the machine which has internet connection and run the tool over there. Once you have the symbols downloaded, copy PDB file to the server’s symbol folder. In case of Debug Diag 1.2 or Debug Diag 2.0 Update 1 copy it to C:\Symcache, else use C:\Symbols

3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:

v Open Debug Diag (Start -> Programs -> Debug Diagnostic Tool 1.2)

v Select "Crash" and click Next

v Choose the option “A specific IIS application pool” and click next and choose the application pool that is crashing from the list and click next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)

v Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint

v Add iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile and change the Action Type to Custom and add the below script in the script window and click on Ok.

Dim StartTimer

CONST interval=2

CONST DumpCount=5

For i=1 to DumpCount Step 1

CreateDump Breakpoint.OffsetExpression, false

DbgState("BP_iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile_ACTION_COUNT") = DbgState("BP_iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile_ACTION_COUNT") + 1

If DbgState("BP_iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile_ACTION_COUNT") >= 5 Then

WriteToLog "Action limit of 5 reached for breakpoint iisfreb!FREB_LOG_FILE_MANAGER::WriteLogFile."

End If

StartTimer=Timer

Do While Timer-StartTimer < interval+0.1

Loop

Next

Exit Sub

v Set Action Limit to 5 and click on Ok.

v Click on Save and Close

v Click Next. Here you can set your "Rule Name" and also the "User dump Location" can be changed here.

v Select "Activate the rule now" and click Finish

v Notice the Status is Active. The User dump Count will increase each time a dump file is created.

v Wait for the issue to happen.

Note: In the above script

'Enter the time interval gap you want the dumps to be generated between in seconds

CONST interval=2

'Number of Dumps you want to get within the specified interval

CONST DumpCount=5

Once the issue occurs

1. The user dump count field in the Crash Rule will increment whenever the tool generates a dump. The dump will be generated in the User Dump Path

2. You can also analyze the Dump by selecting the Debug Diag Analysis tool and clicking Add Data Files. When the .dmp is added, select the Crash/Hang Analyzers script and click Start Analysis. When finished, a report (.mht) will be created in C:\Program Files\DebugDiag\Reports and displayed in Internet Explorer with the results and recommendations. If using custom DLL’s, the Symbol path (Tools menu -> Options and Settings -> Symbol Search Path) to the custom PDB files can be added.

Author: Chiranth Ramaswamy