By
Published
Mar 15 2019 01:34 PM
5,270
Views
First published on MSDN on Feb 23, 2009
Windows Server Failover Clustering nodes can communicate with each other across a router (this is often referred to as the multi-subnet or cross-subnet support), which is key functionality to enable stretched clustering scenarios. To do this, the cluster service builds a list of communication routes to every other node in the cluster and provides those routes to the Microsoft Failover Cluster Virtual Adapter (NetFT.sys) driver, which is clustering’s fault-tolerant network driver.
Let’s assume you have a 2-node cluster. Both nodes are in the same site, and both are connected to three networks A, B, and C. Each node should end up with three routes in the NetFT driver – from Node 1’s perspective we will see: A1->A2, B1->B2, and C1->C2. But now let’s assume that networks A, B, and C are all connected to the same routing infrastructure, and thus interfaces on all networks are mutually reachable. The default behavior is that each node will end up with the same three routes: A1->A2, B1->B2, and C1->C2.
This default behavior is configurable with the cluster common property PlumbAllCrossSubnetRoutes, if you run PowerShell as an administrator, you can see the cluster properties by running:
Get-Cluster | fl *
Here you will find the PlumbAllCrossSubnetRoutes property:
By setting PlumbAllCrossSubnetRoutes to a value of 1 then nodes will also attempt to establish routes which cross subnets.
(Get-Cluster). PlumbAllCrossSubnetRoutes = 1
With this configuration setting Node 1 would end up with all these routes: A1->A2, B1->B2, C1->C2, A1->B2, A1->C2, B1->A2, B1->C2, C1->A2, C1->B2 – nine routes. Why is this bad? Well it uses a lot more heartbeats, especially if you have more than two nodes and more than two networks – think how quickly this could exponentially scale up. Well, why is this good? This can enable cluster communication to survive some real pathological failures. For instance, let’s say network A completely fails, and B1 fails, and C2 fails. Your cluster is still up because you have C1->B2. We didn’t think all the heartbeat and fault isolation complexity cost was worth it for most customers who are responsive to the first failure that occurs, so we left PlumbAllCrossSubnetRoutes off by default. I haven’t yet heard of a customer that experienced that pathological failure and decided to enable PlumbAllCrossSubnetRoutes, however we wanted to extend the option to our customers if they so choose.
In a fully dissimilar multi-subnet cluster (where nodes have no subnets in common), the cluster service always searches for all cross-subnet routes and PlumbAllCrossSubnetRoutes has no effect.
Do not attempt to find cross subnet routes if local routes are found (default value)
(Get-Cluster). PlumbAllCrossSubnetRoutes = 0
Always attempt to find routes that cross subnets
(Get-Cluster). PlumbAllCrossSubnetRoutes = 1
Disable the cluster service from attempting to discover cross subnet routes after node successfully joins
(Get-Cluster). PlumbAllCrossSubnetRoutes = 2
Thanks,
David Dion
Principal Development Lead
Clustering & High Availability
Windows Server Failover Clustering nodes can communicate with each other across a router (this is often referred to as the multi-subnet or cross-subnet support), which is key functionality to enable stretched clustering scenarios. To do this, the cluster service builds a list of communication routes to every other node in the cluster and provides those routes to the Microsoft Failover Cluster Virtual Adapter (NetFT.sys) driver, which is clustering’s fault-tolerant network driver.
Let’s assume you have a 2-node cluster. Both nodes are in the same site, and both are connected to three networks A, B, and C. Each node should end up with three routes in the NetFT driver – from Node 1’s perspective we will see: A1->A2, B1->B2, and C1->C2. But now let’s assume that networks A, B, and C are all connected to the same routing infrastructure, and thus interfaces on all networks are mutually reachable. The default behavior is that each node will end up with the same three routes: A1->A2, B1->B2, and C1->C2.
This default behavior is configurable with the cluster common property PlumbAllCrossSubnetRoutes, if you run PowerShell as an administrator, you can see the cluster properties by running:
Get-Cluster | fl *
Here you will find the PlumbAllCrossSubnetRoutes property:
By setting PlumbAllCrossSubnetRoutes to a value of 1 then nodes will also attempt to establish routes which cross subnets.
(Get-Cluster). PlumbAllCrossSubnetRoutes = 1
With this configuration setting Node 1 would end up with all these routes: A1->A2, B1->B2, C1->C2, A1->B2, A1->C2, B1->A2, B1->C2, C1->A2, C1->B2 – nine routes. Why is this bad? Well it uses a lot more heartbeats, especially if you have more than two nodes and more than two networks – think how quickly this could exponentially scale up. Well, why is this good? This can enable cluster communication to survive some real pathological failures. For instance, let’s say network A completely fails, and B1 fails, and C2 fails. Your cluster is still up because you have C1->B2. We didn’t think all the heartbeat and fault isolation complexity cost was worth it for most customers who are responsive to the first failure that occurs, so we left PlumbAllCrossSubnetRoutes off by default. I haven’t yet heard of a customer that experienced that pathological failure and decided to enable PlumbAllCrossSubnetRoutes, however we wanted to extend the option to our customers if they so choose.
Clusters with No Common Subnets
In a fully dissimilar multi-subnet cluster (where nodes have no subnets in common), the cluster service always searches for all cross-subnet routes and PlumbAllCrossSubnetRoutes has no effect.
Configuration Settings
Value 0
Do not attempt to find cross subnet routes if local routes are found (default value)
(Get-Cluster). PlumbAllCrossSubnetRoutes = 0
Value 1
Always attempt to find routes that cross subnets
(Get-Cluster). PlumbAllCrossSubnetRoutes = 1
Value 2
Disable the cluster service from attempting to discover cross subnet routes after node successfully joins
(Get-Cluster). PlumbAllCrossSubnetRoutes = 2
Thanks,
David Dion
Principal Development Lead
Clustering & High Availability