Collect basics, configuration and logs, when troubleshooting IIS

[Pingulea]

In many cases, when troubleshooting web applications hosted in IIS, we start by collecting the basics for investigation:

• The configuration of IIS and underlying HTTP.SYS, of Asp.Net and of the application, and system info.
• The logs and traces from IIS, HTTP.SYS, Windows Events. If we're lucky, we have FREBs, Failed Request Traces.
• Problem description, to help us figure out how and where to focus during investigation.

The script at http://linqto.me/Grabber or the app at http://aka.ms/LogCatcher should help automate the steps below. And the article at http://linqto.me/IisTS describes why we collect these files.

Start collecting

Many of the files are only accessible to the Administrators on the investigated machine.

So open an administrative command-line console if you're going to use the commands below. Then…

First, create a folder where all collected files will be placed:

MKDIR C:\MsLogs
C:
CD \MsLogs

Collect HTTP.SYS configuration

netsh http show cacheparam > C:\MsLogs\netsh_http_show_cacheparam.txt
netsh http show cachestate > C:\MsLogs\netsh_http_show_cachestate.txt
netsh http show iplisten > C:\MsLogs\netsh_http_show_iplisten.txt
netsh http show servicestate > C:\MsLogs\netsh_http_show_servicestate.txt
netsh http show setting > C:\MsLogs\netsh_http_show_setting.txt
netsh http show sslcert > C:\MsLogs\netsh_http_show_sslcert.txt
netsh http show timeout > C:\MsLogs\netsh_http_show_timeout.txt
netsh http show urlacl > C:\MsLogs\netsh_http_show_urlacl.txt

Master IIS configuration - applicationHost.config

Default location is:

C:\Windows\System32\inetsrv\config\

Note: It's better if you can send the whole folder C:\Windows\System32\inetsrv\config\.

Example:

XCOPY %systemroot%\System32\inetsrv\config\*.* C:\MsLogs\ /E

Note: If IIS is using the Shared Configuration feature, then applicationHost.config should be collected from the network location where the file is shared.

IIS Log files

Only send logs of the site containing the problematic application. Default location:

C:\inetpub\logs\LogFiles\W3SVC[siteID]\

Determine the Site ID from IIS Manager console, selecting the Sites node.

Only include latest files, if too many logs are there; we only need the investigated timeframe. For instance, logs in the last 10 days.

Example:

MKDIR C:\MsLogs\IIS-logs
ROBOCOPY C:\inetpub\logs\LogFiles\W3SVC1\ C:\MsLogs\IIS-logs\ /MAXAGE:10

FREBs, Failed Request Traces…

…if any were collected at all, related to the issue being reported.

Only send traces for the site containing the problematic application. Default location:

C:\inetpub\logs\FailedReqLogFiles\W3SVC[siteID]\

Determine the Site ID from IIS Manager console, selecting the Sites node.

Example:

MKDIR C:\MsLogs\FREBs
COPY C:\inetpub\logs\FailedReqLogFiles\W3SVC1\*.* C:\MsLogs\FREBs\

HttpErr logs

Events and errors logged by the HTTP.SYS driver, which IIS relies on. The default location:

C:\Windows\System32\LogFiles\HTTPERR\

Only include latest files, if too many logs are there; we only need the investigated timeframe. For instance, logs in the last 10 days.

Example:

MKDIR C:\MsLogs\HTTPERR
ROBOCOPY %systemroot%\System32\LogFiles\HTTPERR\ C:\MsLogs\HTTPERR\ /MAXAGE:10

The .NET configuration and root Web.config

Before executing an app, IIS is building its configuration by merging trees with settings - read "config files": .NET FX config, then Asp.NET root Web.config, applicationHost.config of IIS, application's local Web.config cascade (root site Web.config, then sub-application's Web.config etc).

The .NET Framework (FX) configuration:

machine.config

The root ASP.NET configuration:

Web.config

Depending on application bitness, these can be found at:

C:\Windows\Microsoft.NET\Framework(64)\[version]\Config

Example:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config

It's better if you can send the whole folder.

Example:

MKDIR C:\MsLogs\ASP.NET-Config
XCOPY %systemroot%\Microsoft.NET\Framework64\v4.0.30319\Config\*.* C:\MsLogs\ASP.NET-Config\ /E

Application's local configuration

The Web.config file(s) of the root site - and sub-application(s), if that's the case - being diagnosed.

Or at least send the application's effective configuration, compiled by IIS from the above config files:

C:\inetpub\temp\appPools\[appPoolName]\[appPoolName].config

Example:

COPY C:\inetpub\temp\appPools\DefaultAppPool\DefaultAppPool.config C:\MsLogs\

Windows Events Viewer entries

We'll take the following 3 logs from Windows Events Viewer

1. Application Event logs
2. System Event logs  
3. Security Event logs
4. Setup Event logs

Select each of them and then…

1. On the right side, pick Filter Current Log…, then only last 7 or 30 days
      (make sure the period covers moment when issue was manifesting).
2. Then, on the right side again, pick Save Filtered Log File As… EVTX,
      (make sure you include 'Display information for these languages: English')

To collect entries in last 10 days, you can export them by running the following command lines:

WEVTUTIL export-log System C:\MsLogs\WinEvents-System.evtx /query:"*[System[TimeCreated[timediff(@SystemTime) <= 864000000]]]"
WEVTUTIL export-log Application C:\MsLogs\WinEvents-Application.evtx /query:"*[System[TimeCreated[timediff(@SystemTime) <= 864000000]]]"
WEVTUTIL export-log Security C:\MsLogs\WinEvents-Security.evtx /query:"*[System[TimeCreated[timediff(@SystemTime) <= 864000000]]]"
WEVTUTIL export-log Setup C:\MsLogs\WinEvents-Setup.evtx /query:"*[System[TimeCreated[timediff(@SystemTime) <= 864000000]]]"

Note that timediff function returns milliseconds from now. One day means 24 hours x 60 minutes x 60 seconds x 1.000 milliseconds = 86.400.000.

IIS installation logs

Collect Component-Based Setup log file - the packages that Windows installed.
Then IIS setup logs - how the IIS instance installation went.

COPY C:\Windows\Logs\CBS\cbs.log C:\MsLogs\
COPY C:\Windows\iis.log C:\MsLogs\

System information

Export the output of MsInfo32, or simply run the command:

MsInfo32 /nfo C:\MsLogs\System-Info.nfo
MsInfo32 /report C:\MsLogs\System-Info.txt

Export the Registry keys telling about .NET Framework version(s) installed on machine:

Reg.exe export "HKLM\SOFTWARE\Microsoft\NET Framework Setup\NDP" C:\MsLogs\NET-Frameworks-versions.reg.txt

Archive the resulting data collection folder C:\MsLogs\ in a ZIP file, then upload the archive in the file transfer space for the case.

You'll need to send the collected data and/or share it with the engineer helping you do the troubleshooting.

Application issue details

You have to provide as much descriptive information as you can around what happens.

• Please provide the name of the site and/or application with the problem and also the Application Pool name;
• The affected URLs, possibly with screenshots or HTTP traces from client;
• Observed behavior, such as long response time, high-memory consumption or high-CPU;
• What the application is supposed to do, or what are the URLs exerting the problem are doing;
• Reproducing steps, if applicable;
• Any pattern of usage that triggers the behavior: certain time of day, certain users, certain system conditions etc.