Recently I was involved in a troubleshooting on a SQL Always-On failover cluster on a Windows Server 2012R2, and let me say… ”A SPECIAL ALWAYS ON CLUSTER!”
The customer has a branch office in a remote site, with a physical SQL Always On Cluster. He wants to move this physical cluster from the remote site to the central site but on a different hardware. This can be achieved with third party tools and data replication.
When the customer try to start the SQL Always On cluster, on the central site, he receive this strange error: ==========================================================================
Log Name: System Source: Service Control Manager Date: 29.1.2018 18:24:28 Event ID: 7001 Task Category: None Level: Error Keywords: Classic User: N/A Computer: SERVERNAME.Contoso.com Description: The Cluster Service service depends on the Csv File System Driver service which failed to start because of the following error: The system cannot find the file specified.
and here you can see the relevant part of the cluster log: ============================================================ 00002de8.00002d34::2018/01/29-12:28:27.589 INFO -----------------------------+ LOG BEGIN +---------------------------- 00002de8.00002d34::2018/01/29-12:28:27.589 INFO [CS] Starting clussvc as a service 00002de8.00002d34::2018/01/29-12:28:27.589 INFO [CS] cluster service logging level is 5 00002de8.00001038::2018/01/29-12:28:27.589 INFO [CS] Creating cluster node <vector len='1'> 00002de8.00001038::2018/01/29-12:28:27.589 INFO <item>ClusSvc</item> 00002de8.00001038::2018/01/29-12:28:27.589 INFO </vector> 00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Initializing. 00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Local node name: BANZAI-NODE1. 00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Initializing NetFT adapter management information. 00002de8.000016f8::2018/01/29-12:28:27.605 ERR [CS] Service CreateNodeThread Failed, (4319)' because of 'Network interface for NetFT adapter not found.' ……………………. ===========================================================
From the cluster log, we can see that the cluster Service was not able to find the NetFT Adapter (alias the “Microsoft Failover cluster Virtual Adapter”), you can see this adapter from the devmgmt.msc by selecting the “Show hidden devices” option:
In my case from the list of network adapters I wasn’t able to find this network card, so probably something went wrong with the procedure to move the cluster from the branch site to the central site or…..someone removed it, honestly I don’t believe in ghosts yet :).
To resolve this issue I have tried to install manually the “Microsoft Failover cluster Virtual Adapter” by using an old and legacy procedure that I didn’t used for a long time:
After this simple procedure, the cluster was able to start.
TO BE HONEST
I have tried to reproduce the same exact issue on my lab, to try to identify the correlation between the CSV File System error event and the “Microsoft Failover cluster Virtual Adapter”, but I haven’t been able to reproduce it…..I have always received this useful error message:
Log Name: System Source: Microsoft-Windows-FailoverClustering Date: 1/02/2018 13:51:38 Event ID: 1289 Task Category: Cluster Virtual Adapter Level: Critical Keywords: User: SYSTEM Computer: BANZAI-NODE1.contoso.com Description: The Cluster Service was unable to access network adapter 'Microsoft Failover Cluster Virtual Miniport'. Verify that other network adapters are functioning properly and check the device manager for errors associated with adapter 'Microsoft Failover Cluster Virtual Miniport'. If the configuration for adapter 'Microsoft Failover Cluster Virtual Miniport' has been changed, it may become necessary to reinstall the failover clustering feature on this computer.