IIS performance issues after more than 100 concurrent request.

[dayanandsagar]

We are using more than 10 sites on a server. We are facing issue after increase user work load. Sites becomes not responding more than 100 concurrent request.

[#2]

Here are some steps you can take to diagnose and address the issue:
CPU and Memory Usage: Check the server's CPU and memory usage during peak load times. If either is near maximum capacity, this could be causing the performance issues.
Disk I/O: High disk usage or latency can also affect IIS performance. Ensure that the disk is not a bottleneck.
Network Utilization: Ensure that the network bandwidth is not being saturated, especially if your sites are serving large files or media.
Check IIS Configuration Settings
Application Pool Settings: Each site is typically assigned to an application pool. Check if the application pools are recycling too frequently, have low limits on the number of concurrent requests, or are configured to use fewer worker processes than needed.
Connection Limits: IIS has a setting called "Connection Limit" under "Performance" settings for each site. Make sure this setting is not too low.
Request Queue Length: Increase the queue length for your application pools. This setting determines how many requests IIS will queue for processing before it starts rejecting new requests.
Optimize Application Pools and Worker Processes
Increase the Number of Worker Processes: Consider increasing the number of worker processes in the application pool (Web Garden configuration). This allows IIS to handle more concurrent requests.
Enable 32-bit Applications: If your applications are not memory-intensive and can run in a 32-bit mode, enabling this might free up memory usage per application pool.
Recycling Settings: Review recycling settings to prevent application pool recycling during peak times, which can cause delays.
Enable IIS Logging and Monitoring
IIS Logs: Check IIS logs for any errors or warning messages that could indicate the source of the problem
Failed Request Tracing: Enable and configure Failed Request Tracing in IIS to capture detailed error information for requests that fail or take a long time to process.
Performance Monitor (PerfMon): Use Performance Monitor to track IIS-specific counters like Requests Current, Requests Queued, Request Execution Time, Processor Time, and Memory Usage.
Database Performance
If your sites rely heavily on a database (e.g., SQL Server), performance issues with the database could also cause IIS to slow down. Make sure to:

Optimize Database Queries: Review and optimize database queries to reduce load time.
Index Optimization: Ensure that your database tables have the appropriate indexes to speed up queries.
Connection Pooling: Use database connection pooling to reduce the overhead of establishing connections to the database.
Check for Any Network Bottlenecks
Firewall/Antivirus: Sometimes, firewalls or antivirus software on the server can cause delays by scanning each request. Ensure that these are configured optimally for IIS traffic.
To address the issue effectively, you may need to take a combination of the above steps. Start by identifying the most significant bottleneck (CPU, memory, disk, or network) and focus on optimizing that area first. Regular monitoring and tuning of your IIS configuration, applications, and server hardware can help prevent similar issues in the future.

[#3]

I think I am using session and can't increase workers so I think I need to work on Increase thread limit of one worker and maxConcurrentRequestsPerCpu. So It will be great to suggest how to modify default count for this

[#4]

Optimize Worker Thread and I/O Thread Limits
Modify the machine.config or Web.config to increase the worker threads and I/O threads for your sites:

In Web.config (for individual sites) or machine.config (for global settings):

<system.web>
<processModel maxWorkerThreads="100" maxIoThreads="100" />
</system.web>

Modify maxConcurrentRequestsPerCpu
To allow more requests per CPU, you need to modify the IIS settings for maxConcurrentRequestsPerCpu:

Update aspnet.config or Web.config:
<system.webServer>
<serverRuntime appConcurrentRequestLimit="5000" />
</system.webServer>

Increase ASP.NET Queue Length
Adjust the request queue length to allow the server to queue more requests during high traffic times.

In Web.config:
<system.web>
<httpRuntime executionTimeout="300" maxRequestLength="4096" requestQueueLimit="5000" />
</system.web>

Session Management
If you're using in-process sessions, they might be a bottleneck. Consider moving sessions to Out-of-Process or SQL Server session state to better manage memory and worker thread resources.

Move Session to SQL Server (in Web.config):
<sessionState mode="SQLServer" sqlConnectionString="your-sql-connection-string" cookieless="false" timeout="30" />

Modify IIS Application Pool Settings
Increase Queue Length: In IIS, go to Application Pools > Advanced Settings for each pool, and increase the Queue Length setting (default is 1000, you can increase it to 5000 or higher based on your needs).
Idle Timeout: Ensure the Idle Timeout of the application pool is set to an appropriate value to keep workers alive during periods of low traffic.

Increase CPU Cap for Application Pool
In the Advanced Settings of the IIS Application Pool, adjust the CPU Limit if you’ve set a maximum limit on CPU usage per application pool. This can help prevent worker starvation during high loads.

Consider Load Balancing
If performance issues persist after tuning the above settings, consider distributing the load across multiple servers using a load balancer.

[#5]

Showing error when added
<system.web>
<processModel maxWorkerThreads="100" maxIoThreads="100" />
</system.web>

Parser Error Message: It is an error to use a section registered as allowDefinition='MachineOnly' beyond machine.config.

[#6]

@dayanandsagar 

Solution:

If you need to configure the process model, you must edit the machine.config file instead of web.config.

1. Locate machine.config:
• For a 64-bit machine:
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config
• For a 32-bit machine:
C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config
2. Edit the machine.config file:
Add the <processModel> section under the <system.web> section in this file.

<system.web>
<processModel maxWorkerThreads="100" maxIoThreads="100" />
</system.web>

3. Restart IIS after making the changes by running iisreset from the command line.

Note: Changes in machine.config apply globally to all .NET applications running on the machine. Make sure you fully understand the impact of these changes on your environment.