A .NET web app hosted on Azure App Service can be built on top of the Windows Communication Foundation (WCF) framework. We've seen a kind of performance/slow response/timeout issue of such web app, which is caused by the MaxConcurrentSessions limit of WCF is hit.
The symptom might include that after restarting the web app for a few hours, a percentage of the total requests (e.g. 1/3) get no response from the web server and timeout after 230 seconds. 230 seconds is the Azure default timeout value. The web app's memory usage may be observed to grow continuously since the issue starts.
The default MAX concurrent session setting of WCF server is 5000. If the WCF clients don’t close channels properly, a lot of sessions which have been completed will not be able to get closed at the WCF server side. This will result in the MaxConcurrentSessions limit of WCF is always hit. Future requests might get 230 seconds timeout results from the WCF server.
We could configure App Service's Auto-Heal rule to collect memory dump for the w3wp instances once the issue happens. You could find Auto-Heal tool from "Diagnose and Solve Problems" on the Azure App Service portal. Then we can analyze the memory dump in the following way.
- Download the NetExt extension from here Releases · rodneyviana/netext · GitHub
- Copy the NetExt.dll file to WinDbg installation folder, e.g. C:\Program Files\Debugging Tools for Windows (x64)\
- Run WinDbg, open the memory dump file, load the NetExt dll by 0:000> .load NetExt
- Run 0:000> !windex
- Run 0:000> !wservice
When issue happens, you may see result like blow
You can see the current session count is 5000 and the Max session count is also 5000.
You can run !dumpheap -type System.ServiceModel.Security.SecuritySessionServerSettings to check Wcf objects.
SecuritySessionServerSettings is the important object of the WCF session/connection. You can view the activeSessions object within the SecuritySessionServerSettings object. The "Count" value of the activeSessions object contains the number of the active WCF sessions. It also equals to the number of the System.ServiceModel.Security.SecuritySessionServerSettings+SecurityReplySessionChannel objects.
A comparison of two memory dumps of w3wp when issue happened and not happened.
|
Dump 1# - Working (Issue not replicated) |
Time |
Tue Nov 9 08:34:53.000 2021 (UTC + 8:00) |
WCF sessions |
0:000> !wservice Sessions/Max 0n2230/0n5000
|
Number and Size of objects related with WCF sessions |
MT Count TotalSizeClass Name
00007ffdb2d18940 2238 304368 System.ServiceModel.Channels.TransactionChannelListener`1+TransactionReplySessionChannel[[System.ServiceModel.Channels.IReplySessionChannel, System.ServiceModel]]
00007ffdb2912208 8988 2516640 System.ServiceModel.Channels.ServiceChannel
00007ffdb2cf8188 2238 572928 System.ServiceModel.Security.SecuritySessionServerSettings+SecurityReplySessionChannel
|
Number of Active Sessions |
|
|
Dump 2# - Issue (replicated) |
Time |
Tue Nov 9 18:03:55.000 2021 (UTC + 8:00) |
WCF sessions |
0:000> !wservice Sessions/Max 0n5000/0n5000 |
Number and Size of objects related with WCF sessions |
MT Count TotalSizeClass Name
00007ffdb2d18940 5001 680136 System.ServiceModel.Channels.TransactionChannelListener`1+TransactionReplySessionChannel[[System.ServiceModel.Channels.IReplySessionChannel, System.ServiceModel]]
00007ffdb2912208 11530 3228400 System.ServiceModel.Channels.ServiceChannel
00007ffdb2cf8188 5124 1311744 System.ServiceModel.Security.SecuritySessionServerSettings+SecurityReplySessionChannel
|
Number of Active Sessions |
|
To solve this problem, you need to double check the application code of your WCF clients and ensure your WCF clients are closing the created Channels properly after job done.
Updated May 04, 2022
Version 2.0Weizhen_Sun
Microsoft
Joined November 01, 2021
Apps on Azure Blog
Follow this blog board to get notified when there's new activity