HTTP 500 Internal Server Errors: Understanding and Log Collection for Effective Analysis

Introduction

The HTTP 500 Internal Server Error is one of the most common errors faced by developers and administrators when hosting web applications in IIS. This error indicates that the server encountered an issue preventing it completing the request, but it doesn’t provide much detail on what went wrong. To effectively troubleshoot the problem understanding of 500 status code and detailed log collection are essential. In this article, we’ll explore the 500 Internal Server Error, why it happens, and the various methods for collecting useful diagnostic logs.

Overview

The 500 Internal Server Error is a generic server error that occurs when the server cannot process the request due to an unexpected condition. IIS 7.0 and later versions define the following HTTP status codes that indicate a more specific cause of an error 500. The below table illustrate the details sub status code with descriptions.

To learn more about http status codes grab this article - HTTP status code overview - Internet Information Services | Microsoft Learn.

Log Collections

Because 500 Internal Server Errors a general error, pinpointing the root cause requires in-depth investigation. This is where log collection plays a crucial role. You need to collect the following logs -

1. IIS Logs
2. Failed Request Tracing Logs
3. Event logs for application and system.
4. ETW traces
5. Memory dump

1. IIS Logs

It is the starting point for troubleshooting any server error. These logs provide a record of all incoming HTTP requests and the server’s response, including status codes, timestamps, and more. Follow the below steps to enable IIS logs -

• Open IIS Manager.
• Navigate to your website in the Connections pane.
• Open Logging. Logs are typically saved in C:\inetpub\logs\LogFiles. Update the location as per your convenience. 
• And click apply.

To collect the logs, navigate to the log file directory and find the folder name that ends with site number. To know the sites number, Go to Sites from the Connection pane. And look for the ID column. For example, if the site id is 2 then the folder name would be W3SVC2.

2. Failed Request Tracing Logs

Failed Request Tracing Logs or FREB offer more granular insight than IIS logs by breaking down the request processing pipeline. They track which part of the pipeline causes the failure. Follow the below steps to configure FREB.

• Open IIS Manager
• Navigate to your website in the Connections pane.
• Open Failed Request Tracing
• Click Add from right-hand action pane.

• Select All content(*)
• Click Next
• Check Status codes(s) checkbox.
• Status code should be 500-600
• And click Next.

• Check all provides check boxes.
• And click Finish.
• Select the newly created rules and click on Edit Site Tracing.
• Check Enable check box
• Update a directory path(or leave it to default)
• And click Ok.

To collect the logs, navigate to the log file directory and find the folder name that ends with site number. For example, if the site id is 1 then the folder name would be W3SVC1.

3. Event Log

The windows Event Log is another valuable source of information, especially when dealing with application crashes, service restarts, or other server-related issues that might trigger the 500 error.

• Open Event Viewer
• Export Application and System logs under Windows Logs section by saving all without any filters in .evtx format.

4. ETW Traces

You need to use PerfView tool, it is a powerful diagnostic tool for collecting performance traces and analyzing .NET applications. It’s especially helpful for investigating performance bottlenecks or runtime issues that can lead to a 500 error.

• Download the Perfview tool from Releases · microsoft/perfview (github.com)on the Server. It’s a profiler tool that captures ETW events (no installation required).
• Open Perfview tool and
• Go to Collect menu and select Collect option.
• Select Zip, Merge, Thread Time check boxes as below.
• Expand the Advanced Options
• Select Kernel Base, .NET, CPU Samples, IIS, and Tasks(TPL) checkboxes as show below.
• In the additional provider add the string *Microsoft-Extensions-Logging  and in case of .Net Core add *Microsoft.AspNetCore.Hosting [do not miss * at the beginning]

• Click Start Collection button.
• Replicate the issue
• Stop collection by clicking Stop collection,
• Wait for some time to allow it to Merge, it will generate Perfview.etl.zip file.

5. Memory dump

When your application crashes unexpectedly, a memory dump captures the state of the application at the moment of the crash, including active threads, memory usage, and error states. This is useful for diagnosing severe issues that lead to 500 errors. Since the actual error is yet to be identified, you need to capture dumps on first chance exception. Follow the below steps -

• Download and install DebugDiag tool from this official download link - Download Debug Diagnostic Tool v2 Update 3 from Official Microsoft Download Center
• Open DebugDiag 2 Collection.
• Click on Add Rule
• And select Crash and click Next.
• Select A specific IIS web application pool
• Click Next
• And choose the application pool of the application in question.

• Select Action type to Full Userdump
• And action limit count to 3.
• Click Next.
• Choose an output location to store the dump files.
• And Save
• After that Activate the Rule.

Conclusion

HTTP 500 Internal Server Errors can be elusive and challenging to troubleshoot, but with the right log collection techniques, you can narrow down the causes and find a solution more efficiently. Start with basic IIS logs for high-level information, dive deeper with Failed Request Tracing logs for detailed request failures, use PerfView for performance bottlenecks, and, if necessary, collect memory dumps for deeper analysis of crashes and unhandled exceptions.  If you are comfortable analyzing this data then you should be able to draw insights from the collected logs. If you want us to do that, please contact us with a case and we will do it for you.