A reminder on real life performance impact of Windows SNP features
Published Jul 30 2013 11:37 AM 21.3K Views

I think we are due for a reminder on best practices related to Windows features collectively known as “Microsoft Scalable Networking Pack (SNP)”, as it seems difficult to counter some of the “tribal knowledge” on the subject. Please also see our previous post on the subject.

Recently we had a customer that called in on the subject and this particular case stressed the importance of the Scalable Networking Pack features. The background of this case was that the customer was Running Exchange 2010 SP2 RU6 on Windows 2008 R2 SP1, and they had multiple physical sites with a stretched DAG. This customer had followed our guidance from Windows 2003 times and disabled all of the relevant options on all of their servers, similar to below:

Receive-Side Scaling State : disabled
Chimney Offload State : disabled
NetDMA State : disabled
Direct Cache Acess (DCA) : disabled
Receive Window Auto-Tuning Level : disabled
Add-On Congestion Control Provider : ctcp
ECN Capability : disabled
RFC 1323 Timestamps : disabled

The current problem was that the customer was trying to add copies of all their databases to a new physical site, so that they could retire an old disaster recovery site. The majority of these databases were around 1 to 1.5TB in size, with some ranging up to 3TB. The customer stated that the databases took five days to reseed, which was unacceptable in his mind, especially since he had to decommission this site in two weeks. After digging into this case a little bit more and referencing this article, we first started by looking at the network drivers. With all latency issues or transport issues overs a WAN or LAN, we should always make sure that the network drivers are updated. Since the majority of the servers in this customer’s environment were virtual servers running the latest version of the virtual software, we switched our focus over to physical machines. When we looked at the physical machines we saw they had a network driver with a publishing date of December 17, 2009.

At this point I recommended to update the network driver to a newer version with at least a driver date of 2012 or newer. We then tested again, and still saw the transfer speeds roughly similar to those before updating the drivers. At that point I asked the customer to change the scalable network pack items from above to:

Receive-Side Scaling State : enabled
Chimney Offload State : automatic
NetDMA State : enabled

(Here is how you change these items.)

The customer changed the SNP features and then rebooted the machines in question. At that time he started to reseed a 2.2TB database across their WAN at around 12pm. The customer sent me an email later that night that stated the database would now take around 12 hours to reseed. The next morning he sent me another email and the logs were copied over before he showed up for work at 7am. This time the reseed took 19 hours to complete compared to 100+ hours with SNP features disabled. Customer stated that he was very happy, and started planning how to upgrade network drivers on all other physical machines in his environment. Once that was done he was going to change RSS, TCP Chimney, and NetDMA to the recommended values on all of his other Windows 2008 R2 SP1 machines.

The following two articles show the current recommendations for the Scalable Networking Pack features:

  1. Here is the document reference above that shows the correct settings for each version
  2. Even though this article specifies SQL, this is still relevant to the operating system that Exchange sits on.

So, what exactly is our point?

Friends don’t let friends run their modern OS-servers with old network drivers and SNP features turned off! As mentioned in our previous blog post on the subject, please make sure that you update network-level drivers first, as many vendors made various fixes in their driver stacks to make sure that SNP features function correctly. The above is just one illustration of issues that incorrect settings in this area can bring to your environment.

David Dockter

17 Comments
Not applicable

WOW!!

This is AWESOME! I'm going to try this. Seeding is a major issue across the Atlantic.. specifically after mailbox moves.

Thanks for Sharing!

Dame

thelifestrategist.wordpress.com

Not applicable

technet.microsoft.com/.../gg162716%28WS.10%29.aspx says "You cannot use TCP Chimney Offload and NetDMA together."

Not applicable

Why disable "Receive Window Auto-Tuning Level" ? Wasn't it one of the main improvements of the new TCP/IP stack ?

Not applicable

@td77 - This is a default setting within Windows 2008 R2.  In order for NetDMA to be used, it requires hardware to be enabled.  So if the hardware isn't there, or it is not enabled, it won't get used.  I just placed RSS, TCP Chimney and NetDMA in this article, as these are the three that we see are changed regularly.

Not applicable

So why hasn't the recommendation for Server 2003 SP2 been changed?

Not applicable

 @ Yuhong Bao  - the guidance probably still makes sense for Windows 2003 (anyone still using it should already have their configurations pretty standardized by now!)

Specifically these settings were turned off in the Server 2003/Exchange 2003 days due to non-paged pool memory leaks. These would bring any medium sized server to its knees. Not an issue with newer Server 2008 drivers and the switch to 64-bit OS

Not applicable

Good One David.

Thanks for sharing.

Not applicable

Do these recommendations apply to Server 2012 too? I noticed that NetDMA is not available in Server 2012.

And I would suggest to add the recommendations in the official documentation.

Not applicable

got me thinking... does the ExBPA check this? Why can't it check before setup ?

Not applicable

what if we are running under ESX environment?  Should we enable SNP features as well?

Not applicable

@Yuhong Bao   - The guidance has not changed because we can not be for certain that drivers written for Windows 2003 have the SNP features embedded within them.  So that is why our guidance is to turn them off to prevent issues from happening.

Not applicable

@Frank T - EXBPA does not check for this, but it is something we are looking at adding for future products.  We have also created rules within our diagnostics that we send to customers to check for this.

Not applicable

@hwc - The guidance in this document applies to ESX as well.  Please make sure your drivers are updated on your host, and then follow the guidelines above.

Not applicable

@David Dockter: Huh? I never thought old drivers that do not know SNP at all was the problem.

Not applicable

I'm with Jetze -- would like to know if this would apply for Exchange 2013 on Windows 2012.

Not applicable

@Yuhong Bao - I see my last post did not get added.  Prior to Windows 2008, we recommended turning RSS and TCP Chimney off because the Operating System had the ability to configure these options, and the hardware did not know how to handle these new processes.  Shortly after we recommended turning these options off, the hardware manufacture provided driver updates that allowed these options to be configured from the hardware side.  So instability came about due to the mismatch between the operating systems configuration and the networking cards configuration.  

Not applicable

In regards to Server 2012,

Chimney is optional, and is disabled by default

RSS should be on, and is enabled by default

NetDMA should be on, and is enabled by default

Version history
Last update:
‎Jul 01 2019 04:14 PM
Updated by: