Note: Please see this blog post for the most updated recommendations on this subject.
EDIT: On 3/12/2008, we have posted a new blog post that talks about the hotfix that we have released for this problem: http://msexchangeteam.com/archive/2008/03/12/448421.aspx
Previously I did not mention what network cards were affected, but the majority of the cases that we have seen have dealt with Broadcom cards or network cards that have a Broadcom chipset in them.
Broadcom has provided an update to their drivers and can be downloaded from http://www.broadcom.com/support/ethernet_nic/netxtremeii.php that helps resolves the issues with these offloading feature problems. To determine if you have the correct update, you need to check the version numbers to ensure they are 3.7.19 or later. Anything lower than that does not have the fixes in them. ExBPA is being updated with a new rule to detect if the Scalable Networking pack features are enabled on the server.
NOTE: Before updating directly from Broadcom's site, I would recommend checking with your Server manufacturer first before applying the update directly from Broadcom as this may affect your ability to apply updates directly from each vendor in the future using their integrated update utilities.
Currently there are only a handful of manufacturers that have updated the drivers on their sites, but over time, you will see the updates available for downloading.
I have personally dealt with cases that having the Scalable Networking features enabled with the latest drivers have not caused any connectivity issues to/from Exchange servers, so you can now take advantage of these features that may help increase overall network performance on your servers.
Possibly impacted Operating System versions:
- Windows 2003 SP1 with KB 912222: The Microsoft Windows Server 2003 Scalable Networking Pack Release installed
- Windows 2003 SP2
With the release of the Scalable Networking Pack that is included with Windows 2003 SP2, we in Exchange support have been seeing some connectivity issues once the new networking features are enabled. These new features are enabled by default and are only used if your network card driver supports them. Some of the new architectural additions that were introduced with the Scalable Networking Pack are TCP Chimney Offload, Receive-side Scaling (RSS) and NetDMA. These were introduced because of the Microsoft Scalable Networking Initiative that was designed to help reduce OS bottlenecks caused by network packet processing. More information regarding the Scalable Networking initiative can be found at www.microsoft.com/snp.
What this is does essentially is to offload TCP/IP packet processing to the network card, thus freeing up valuable CPU cycles for your applications. The throughput increases that you can get from having these enabled are quite significant.
To support these new features, the NDIS miniport driver had to be redesigned to handle this, thus NDIS 6.0 was born. For more information regarding the updated NDIS miniport driver, see http://msdn2.microsoft.com/en-us/library/ms798546.aspx from the Windows Driver Kit NDIS Miniport Driver Reference. With the NDIS 5.1 driver, the Operating System was limited to processing network traffic on a single CPU which impacted CPU performance quite significantly. The new NDIS driver design allows for processing this same information across multiple processors which will improve performance quite significantly.
This appears like this would actually increase the performance of Operating System, but what does this have to do with Exchange? Well, some of the issues surrounding the problems are documented in 936594 and a short list of what may affect Exchange is listed below.
- You cannot create a Remote Desktop Protocol (RDP) connection to the server.
- You cannot connect to shares on the server from a computer on the local area network.
- You cannot connect to Microsoft Exchange Server from a computer that is running Microsoft Outlook.
- You can only connect to Web sites that are hosted on the server or on the Internet by using a secure sockets layer (SSL) connection. In this scenario, you cannot connect to a Web site that does not use SSL encryption.
- You experience slow network performance.
- You cannot create an outgoing FTP connection from the server.
- You experience intermittent RPC communications failures.
- Some Outlook clients may be unable to connect to Exchange.
- You cannot run the Configure E-mail and Internet Connection Wizard successfully.
- Microsoft Internet Security and Acceleration (ISA) Server blocks RPC communications.
- You cannot browse Internet Information Services (IIS) Virtual Directories.
In Support Services, we are also seeing some of the following behaviors when clients are trying to connect to the Exchange server.
- 32 MAPI sessions exceeded (9646 errors) causing the inability for Outlook clients to connect to the Information Store. This can occur more frequently with VPN connected clients and we have also seen scenarios where Exchange 2007 is affected with Windows 2003 SP2 installed as well.
- Non-paged pool memory leaks caused by having the Chimney feature enabled. Sometimes you can't even start the IIS services when Non-paged pool gets below 20MB.
- The Inability to logon to Outlook Web Services or even IIS for that matter, either locally or remotely.
- Networking throughput is decreased when these features are turned on. This is the opposite of what the Networking Pack is supposed to do.
- Cluster ISAlive checks fail randomly
- TCP Connections are reset when RSS is enabled.
- TCP port exhaustion
Wow, now that is a lot of things that could possibly go wrong, so why have this enabled by default? The default assumes that the network card drivers are up to date or have the latest driver release that supports the new networking features. Most of the issues that we see are due to outdated network card drivers and simply updating the network card driver to the latest release has provided relief in most situations. Other cause that has been seen is outdated Storport/SAN drivers consuming higher than normal non-paged pool memory, so ensure that you have the latest storport.sys and SAN drivers installed on your servers to help mitigate this problem. If you are still having problems, then I would highly recommend opening a case with Support Services for further assistance.
So how does one go about troubleshooting something like this? One would think that Netmon may show you what is going on, but the truth to the matter is that once the data is handed off to the Network card, the netmon filter can never log that request. This makes it harder to troubleshoot unfortunately.
You can actually see what connections are offloaded by using the netstat -t command. This -t switch is only available if the networking pack is installed on a server. An offloaded network connection can be in one of the following states:
- In Host - the network connection is being handled by the host CPU
- Offloading - the network connection is in the process of being transferred to the offload target
- Uploading - the network connection is in the process of being transferred back to the host CPU
- Offloaded - the network connection is being handled by the offload target.
One of the more common issues we have seen is working with the TCP Chimney offload feature and most of the resolutions to date that we provide to customers is to disable that feature to see if that helps your specific scenario.
To disable TCP Chimney, you can do this one of two ways. I prefer the latter as it is instantaneous and does not require a reboot.
To Disable TCP Chimney, Navigate to the following registry key and set the value to 0. Note: You have to reboot the server after this registry change.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableTCPChimney"=dword:00000000
Or you can use the netsh command which I prefer without having to reboot:
Netsh int ip set chimney DISABLED
If that does help with the situation, you could also try disabling the following offloading keys under the above registry hive to disable the RSS features.
"EnableTCPA"=dword:00000000
"EnableRSS"=dword:00000000
If any of the above options help with the issue, this may be as simple as updating your network card drivers to the latest version and then re-enabling those features again to see if the problem still occurs. If there are still questions regarding whether or not your network card supports these new features, contact the hardware vendor for your card for more information.
To help with some of the HTTP resource constraints that may occur when nonpaged pool memory is less than 20MB on an Exchange Server with these features enabled, you can enable the AggressiveMemoryUsage registry key per 934878 to help HTTP services continue to function in low nonpaged pool memory conditions. Setting this same registry key may also help with allowing your cluster HTTP IsAlive polls to function in these low memory conditions that normally result in a failover. Of course, this is a workaround for a short period of time, but root cause of paged pool memory issues should be identified and fixed by following KB articles 177415, 912376, 317249, to help determine what might be causing the problem.
One other thing to mention here is that TCP Chimney offload and NetDMA will not work with the following features enabled:
- Windows Firewall
- Internet Protocol security (IPsec)
- Internet Protocol Network Address Translation (IPNAT)
- Third-party firewalls
- NDIS 5.1 intermediate drivers
If any one of these features is turned on, TCP Chimney offload and NetDMA will not work regardless of the registry settings.
Using offloading with any Microsoft or 3rd party NAT solution will also cause known connectivity issues with Exchange Servers, SBS, ISA servers, etc.
As you can see, there are many variables that can affect the performance and stability of your network card, so you just need to keep in mind the new networking features during your troubleshooting efforts.
I hope this helps shed some light on these new features and some of the issues that we in support are seeing here today. It may even prevent a future support call somewhere down the road.
For a list of partners that have tested Scalable Networking with their networking products, see Scalable Networking Partners.
References:
You cannot host TCP connections when Receive Side Scaling is enabled in Windows Server 2003 with Service Pack 2
http://support.microsoft.com/default.aspx?scid=kb;EN-US;927695
You may experience network-related problems after you install Windows Server 2003 SP2 or the Scalable Networking Pack on a Windows Small Business Server 2003-based computer that has an advanced network adapter
http://support.microsoft.com/default.aspx?scid=kb;EN-US;936594
TCP traffic stops after you enable both receive-side scaling and Internet Connection Sharing in Windows Vista or in Windows Server 2003 with Service Pack 1
http://support.microsoft.com/default.aspx?scid=kb;EN-US;927168
