Do you have a sleepy NIC?
Published Oct 22 2013 10:56 AM 159K Views

I continue to run into this issue over and over in the field so I wanted people to be aware of this possible problem. In a Database Availability Group (DAG), if your databases are randomly mounting or flipping from one server to another, for no apparent reason (including across datacenters) you may be suffering from your network interface card (NIC) going to sleep. And that’s not a good thing.

Power Management on the NIC

In the power management settings for the NIC on Windows Server, make sure you are not allowing the NIC to go into power save mode. Why is this important? It seems like at least once a month I’ve run into customers who have this power management setting turned on and more than one of them even had it turned on for their replication network. They were seeing some odd behavior - for example, their databases randomly flipping from one DAG node to another for no apparent reason. And yes, they were all on physical machines.

Here are the steps to look at this configuration: use Device Manager to change the power management settings for a network adapter.

To disable all Power Management settings in Device Manager, expand Network Adapters, right-click the adapter > Properties > Power Management, and then clear the Allow the computer to turn off this device to save power check box.

Screenshot: Network adapter properties | Power Management tab
Figure 1: Disable power management for the network adapter from the Power Management tab

Some of your network adapters may not have the Power Management tab available. This is a good thing, as your NIC is not able to go to sleep. This means there is one less item to worry about in your setup!

CAUTION Be careful when you change this setting. If it's enabled and you decide to disable it, you must plan for this modification as it will likely interrupt network traffic. It may seem odd that by just making a seemingly non-impacting change that the NIC will reset itself, but it definitely can. Trust me; I had a customer ‘test’ this during the day by accident… oops!

PowerShell to the rescue

In addition, now that PowerShell is able to be used for just about everything, there is this page that has a PS script available to make this change. There are additional links and related forum threads to review with supplementary information near the bottom of the script download page.

This modifying script will run against all physical adapters to the machines you deploy it to, and you can also modify the script to disable wireless NIC’s. With PS, don’t forget that you can use this script to blast down these changes to all of your Exchange servers with a single step.

GPO and regedit

For those of you that are more comfortable with regedit and creating GPO’s to help control these settings, that option is also available. This page has information on both ‘one off’ fixes that you can download a .reg file and manually deploy, or using GPO Preferences, you can edit the values in a GPO and apply those changes to an Exchange Server OU (Organizational Unit).

The one step to note with the regedit process is which NIC you are working with and how many NIC’s your server has. The registry only knows of the first, second, third, etc. number of NIC’s. Now if you have identical builds between all of your servers, then this option certainly will ensure that all current and any future servers placed into an OU with the GPO applied will adhere to the proper registry settings.

Also don’t forget, you can record all of your changes on a Windows Server 2008R2 or higher OS, by using the Problem Steps Recorder (PSR) tool.

There you have it: if your DAG Databases are randomly becoming active from one server to another with no apparent reason, you may have a sleepy NIC. Please confirm that you have avoided this setting as you build out not only your DAG environment, but all Exchange related servers. Thank you.

Mike O'Neill

37 Comments
Not applicable
With the help of Lalita Jat, another PFE team member, we have an additional answer about the 'high performance' setting within the Windows Server OS:

• If the server is set to High Performance - Windows places the system in the highest performance state and disables the dynamic scaling of performance in response to varying workload levels. Therefore, special care should be taken before setting the power plan to High Performance as this can increase power consumption unnecessarily when the system is underutilized.

• We will have to disable “Allow the computer to turn off this device to save power” option of Power management from NIC. Setting the server in High Performance would not stop the NIC to go in sleep mode whenever there is no activity, as that is a setting at an OS level. The setting on NIC will take preference in this situation.

Hopefully this answers the question about the high performance setting.

Not applicable
@Anonymous. There are 3 things I always check when I roll into a customer with a DAG.

1. Turn off sleepy NIC's.

2. Ensure there is an IP set for the DAG in each subnet/AD Site where DAG members live. Occasionally, engineers just put one in and not an IP for the secondary site, which is typically another subnet range.

3. Ensure your 2010 DAG network is collapsed: http://blogs.technet.com/b/timmcmic/archive/2011/09/26/exchange-2010-collapsing-dag-networks.aspx

Any one of these could cause inconsistent DAG behavior. Hope that helps.

Not applicable
Great artical but, it didn't work in my environment. I came to the same conclusion Steve did before scrolling down... We unchecked the NIC power save mode on all Exchange 2010 servers, rebooted each server and 5 hrs later our DAG failed over again.
Not applicable
This is a key item to check while deploying Windows Servers for any collaboration platforms. Great article, Mike.
Not applicable

Good catch Mike, one of the most important reason of DAG Wars!

Not applicable

Mike does it needs to be don on the CAS/HT servers as well?

Not applicable

If the box is checked, is there any log that would show the computer turning off the NIC to save power?

Not applicable

I think this might be my gremlin.  Kudos if this works.

Not applicable

I have a similiar issue with one customer, but in my case just few databases randomly become active from one server to another. I could note yesterday that just 2 DBs were changed. If it is related to the power management all DBs should be moved, right?

Not applicable

@ Samir Marwaha: While I have seen this DAG Wars issue occur on Mailbox servers, it would be advisable to ensure all servers (not only Exchange ones) have this setting disabled. When would I want a server NIC to go to sleep? Never would be my guess.

@ Heath Graves: This is one of the first 3 things I check rolling onsite. NIC settings, DAG IP's for each subnet that has a DAG member, and collapse of the DAG network. Any or all 3 can cause gremlin issues on randomly activating DB's in a DAG.

@ Tiago Ferreira de Souza: I have not dug into specifics if all or just some DB's are impacted. If the NIC only drops a few packets, then wakes back up, I could see that only some DB's could be impacted and not necessarily the entire server with all DB's

bouncing.

Again, ensure that all 3 steps are covered if you are having random DB's mounting for what appears to be no reason at all.

1. Disable NIC sleep mode

2. Ensure the DAG network is collapsed.

blogs.technet.com/.../exchange-2010-collapsing-dag-networks.aspx

3. Ensure that there is an IP assigned to the DAG where each DAG member lives in that subnet:

technet.microsoft.com/.../dd351172(v=exchg.150).aspx

Hope that helps.

Not applicable

@ Samir Marwaha: While I have seen this DAG Wars issue occur on Mailbox servers, it would be advisable to ensure all servers (not only Exchange ones) have this setting disabled. When would I want a server NIC to go to sleep? Never would be my guess.

@ Heath Graves: This is one of the first 3 things I check rolling onsite. NIC settings, DAG IP's for each subnet that has a DAG member, and collapse of the DAG network. Any or all 3 can cause gremlin issues on randomly activating DB's in a DAG.

@ Tiago Ferreira de Souza: I have not dug into specifics if all or just some DB's are impacted. If the NIC only drops a few packets, then wakes back up, I could see that only some DB's could be impacted and not necessarily the entire server with all DB's

bouncing.

Again, ensure that all 3 steps are covered if you are having random DB's mounting for what appears to be no reason at all.

1. Disable NIC sleep mode

2. Ensure the DAG network is collapsed.

blogs.technet.com/.../exchange-2010-collapsing-dag-networks.aspx

3. Ensure that there is an IP assigned to the DAG where each DAG member lives in that subnet:

technet.microsoft.com/.../dd351172(v=exchg.150).aspx

Hope that helps.

Not applicable

@ Samir Marwaha: While I have seen this DAG Wars issue occur on Mailbox servers, it would be advisable to ensure all servers (not only Exchange ones) have this setting disabled. When would I want a server NIC to go to sleep? Never would be my guess.

@ Heath Graves: This is one of the first 3 things I check rolling onsite. NIC settings, DAG IP's for each subnet that has a DAG member, and collapse of the DAG network. Any or all 3 can cause gremlin issues on randomly activating DB's in a DAG.

@ Tiago Ferreira de Souza: I have not dug into specifics if all or just some DB's are impacted. If the NIC only drops a few packets, then wakes back up, I could see that only some DB's could be impacted and not necessarily the entire server with all DB's

bouncing.

Again, ensure that all 3 steps are covered if you are having random DB's mounting for what appears to be no reason at all.

1. Disable NIC sleep mode

2. Ensure the DAG network is collapsed.

blogs.technet.com/.../exchange-2010-collapsing-dag-networks.aspx

3. Ensure that there is an IP assigned to the DAG where each DAG member lives in that subnet:

technet.microsoft.com/.../dd351172(v=exchg.150).aspx

Hope that helps.

Not applicable

Great Article...The sleepy NIC can be applied to all Microsoft application server as best practice...

Not applicable

Thank you Mike. In my case is missing only disable NIC sleep mode.

Not applicable

I always untick that on all my NIC-s. Have been wondering for years why they are ticked on all NIC-s by default on Windows Servers as that can only cause issues and lead to server not being available. I suspect an oversight on the MS Windows Server team as it appears to be the case no matter what drivers or make of NIC you have.

Not applicable

Well, is this the default on a Server 2012 box? If yes, then sorry to say MS, you can do better! ;)

GOOD catch indeed, Mike!

Not applicable

It should also be noted that the boxes on the power management tab can easily get check again during server maintenance, e.g. when you or your operations staff upgrade the driver to a newer release.  :)

I have always unchecked these options on every server I've ever worked on.  I don't see why it always defaults to being checked or even why these options need to exist at all for a server class machine or OS.

Not applicable

Also, please have a look to your general power saving settings:

Degraded overall performance on Windows Server 2008 R2

support.microsoft.com/.../2207548

Poor virtual machine application performance may be caused by processor power management settings

kb.vmware.com/.../search.do

And don't forget to check the host BIOS power settings, too.

keyword: C-States

Not applicable

Hi Mike,

We are investigating the risk around this issue, there seems to be a lack information around what will cause a NIC with this setting enabled to go into sleep mode? I'm assuming either a lack of traffic across the interface or something else? Any info would be appreciated.

Cheers

Not applicable

I agree with Josh.  I don't have these issues in Exchange, but I have been having issues with a client NIC that has been disconnecting periodically throughout the day.  I ultimately suspect a bad driver, but for kicks, I tried this setting.  Sure enough, no issues so far -- knock on wood.

Not applicable

According to this article (support.microsoft.com/.../2740020), it only effects the NIC when the computer goes into sleep. Why would any server be going into a sleep state?

Not applicable

Fully agree with Steve. Why should I do this when running a Server in "High Performance" Mode. So the Server operates always in S0 state!? and never goes to sleep. Something I like cause it's a Server ^^ and not a Client. The details of my NIC also say that in S1 (which is sleep) the NIC goes to Energy Saving D3. Conclusion: No S1 -> No D3 of the NIC

Not applicable

I have read the MS articles several times and still do not fully understand why a NIC in server production environments, including DAG's, would ever go to sleep. However, I see it monthly with customers having their DAG DB's activate randomly for no apparent reason. They stop the random behavior after we implement this change. Something somewhere must be triggering it; different drivers, different vendors, different patches...

Would love to find the end all solution to this issue.

Not applicable

I have read the MS articles several times and still do not fully understand why a NIC in server production environments, including DAG's, would ever go to sleep. However, I see it monthly with customers having their DAG DB's activate randomly for no apparent reason. They stop the random behavior after we implement this change. Something somewhere must be triggering it; different drivers, different vendors, different patches...

Would love to find the end all solution to this issue.

Not applicable

I have read the MS articles several times and still do not fully understand why a NIC in server production environments, including DAG's, would ever go to sleep. However, I see it monthly with customers having their DAG DB's activate randomly for no apparent reason. They stop the random behavior after we implement this change. Something somewhere must be triggering it; different drivers, different vendors, different patches...

Would love to find the end all solution to this issue.

Not applicable

The PowerShell script says you need to reboot the server for it to take affect? Is this true or can it be done on the "fly" knowing that it will cause a reset of the NIC.

Not applicable

Thanks for the heads up Mike, We don't have any issues here but it won't hurt too turn it off anyway.

I'd be interested to know if all these cases are with physical servers?

Josh

Not applicable

@ Joshua Bines: I actually only found it on virtual servers at first (running on VMWare), then, ran into more setups and found it on any/all types of servers. I think it depends on the driver, as I have seen several (and others pointed out) NIC's don't even have the 'power management' tab. Which is ideal. I don't know of a compiled list of which drivers/versions have/do not have the power management tab.

@ Will Shepherd: I doubt it would require a reboot, as the setting takes effect upon making the change. You are correct, the NIC will reset itself. Unfortunately, I can't test this, as my hyper-v server NIC's do not have the power management tab available. Maybe someone can verify if they use the PS that it does not require a reboot? I just don't have current access to a machine that I can test.

Not applicable

@ Joshua Bines: I actually only found it on virtual servers at first (running on VMWare), then, ran into more setups and found it on any/all types of servers. I think it depends on the driver, as I have seen several (and others pointed out) NIC's don't even have the 'power management' tab. Which is ideal. I don't know of a compiled list of which drivers/versions have/do not have the power management tab.

@ Will Shepherd: I doubt it would require a reboot, as the setting takes effect upon making the change. You are correct, the NIC will reset itself. Unfortunately, I can't test this, as my hyper-v server NIC's do not have the power management tab available. Maybe someone can verify if they use the PS that it does not require a reboot? I just don't have current access to a machine that I can test.

Not applicable

@ Joshua Bines: I actually only found it on virtual servers at first (running on VMWare), then, ran into more setups and found it on any/all types of servers. I think it depends on the driver, as I have seen several (and others pointed out) NIC's don't even have the 'power management' tab. Which is ideal. I don't know of a compiled list of which drivers/versions have/do not have the power management tab.

@ Will Shepherd: I doubt it would require a reboot, as the setting takes effect upon making the change. You are correct, the NIC will reset itself. Unfortunately, I can't test this, as my hyper-v server NIC's do not have the power management tab available. Maybe someone can verify if they use the PS that it does not require a reboot? I just don't have current access to a machine that I can test.

Not applicable

@ Mike_O'Neill - The permanent fix would probably be a GPO/DSC (PowerShell 4.0's Desired State Configuration) setting that tells all NICs on the host to disable Power Saving; then you set that GPO on your Server Group(s).

Here's the Official Method to doing it, under 'Fix It Myself' it lists the Registry entry - there is one per-device, so one would need a way to enumerate all devices under that:

support.microsoft.com/.../2740020

Here is someone who made a script to do it (his is set for Intel NICs only, but could be modified fairly easily):

community.spiceworks.com/.../243073-we-need-a-gpo-or-script-to-adjust-power-management-on-nics

Not applicable

Mike, Appreciated for highlighting this issue -

But, if this is enabled by default at OS level then this is wrong as per my view, MS should disbale this by default in next rollup or SP release ...I know power saving is also important but it should not create this type of unnecessary issue in the email service of organization..

Not applicable

Excellent article!  Changing that setting stopped the constant flipping of my DAG DBs.  Mine were all ESX VMs but apparently it happens with physicals as well.  Definitely a must for an Exchange server build.

Not applicable

Thank you Terrance for the feedback, glad it helped your situation. And thank you for helping me pin point this error.

Also, I'm getting other feedback from engineers and it appears that (most/all?) HP NIC's do not have the power management tab available. Which is a good thing.

Not applicable

Thank you Terrance for the feedback, glad it helped your situation. And thank you for helping me pin point this error.

Also, I'm getting other feedback from engineers and it appears that (most/all?) HP NIC's do not have the power management tab available. Which is a good thing.

Not applicable

Thank you Terrance for the feedback, glad it helped your situation. And thank you for helping me pin point this error.

Also, I'm getting other feedback from engineers and it appears that (most/all?) HP NIC's do not have the power management tab available. Which is a good thing.

Copper Contributor

Well, apparently some of you aren't seeing your comments come across right away and are clicking send 2 or 3 times... but no one has removed those comments.  Just dropping in to say WTF is noone saying ANYTHING about this BOGUS green AGENDA.  The entire thing started plenty of years ago and I think it STINKS.  They have it in USB connections of COURSE, so I know all about it.  I presume this is the reason that my WIFI disconnects 20 seconds after connecting.  If you have WINDOWS PRO you can use Group policy to change certain WINDOWS CONNECTION MANAGER protocols, and yes on some of my machines I can do it.  But not all.  So I found this ALSO is in the Network Adapters properties- configure (above the list that you used to use to change your ipv4 settings, is a button that says CONFIGURE) so you can use this CONFIGURE button and YOU WILL SEE THE POWER MANAGEMENT in the upper row of tabs and SURE ENOUGH IT IS SET TO HAVE THIS CONFOUNDED COMPANY DISABLE YOUR NETWORK ADAPTER at the bequest of the GOD DMND WEF who KILL people for malthusian reasons AS WELL as internet connections!!!!!!! 

 

Editing to note that after I unselected "allow windows to turn off this to save power" the appearance of my network adapters pane changed from medium icons to a very short list containing bluetooth

wifi

ethernet

Which took aback me a minute, but double clicking each of these 'details' type listings gets the same properties box as the usual icons gets.

Version history
Last update:
‎Jul 01 2019 04:15 PM
Updated by: