Failover Clustering articles

Rack-Local Reads in Storage Spaces Direct Campus Cluster / Rack Aware Cluster

Rob-Hindman — Wed, 25 Mar 2026 02:41:48 GMT

Author: Parsan Saffaie

Locality‑Aware Read Optimization for Modern Multi‑Rack and Campus Deployments

With Windows Server 2025, Storage Spaces Direct (S2D) expanded its resiliency capabilities with support for Campus Clusters, allowing Failover Cluster deployments to span racks, chassis, and even sites using explicit fault‑tolerance domains. If you haven’t seen it yet, check out Rob’s announcement here:

Announcing Support for S2D Campus Cluster on Windows Server 2025.

Building on that foundation, Windows Server 2025 introduces Rack‑Local Reads, a new optimization that allows S2D to use the cluster topology during read selection, ensuring mirrored reads are served from the closest healthy copy whenever possible.

Before Rack‑Local Reads

Before this feature, when a mirrored space was deployed in an S2D cluster, data copies were placed across fault‑tolerant domains (nodes, racks, or sites) to ensure resiliency.

During read operations, all healthy copies were treated as equal. When a read I/O arrived, S2D could satisfy it from any available copy containing the requested data offset — regardless of where that copy was located in the cluster.

This behavior was correct and safe, but it was topology‑agnostic.

Example: topology-agnostic read selection

This diagram shows a conceptual view of read behavior in a multi‑rack cluster, where a workload may read from a remote rack even though a local copy is available.

Read selection flow:

A read request arrives on Node 1 in Rack 1, where the workload is running.
Storage Spaces Direct identifies all healthy copies of the requested data offset.
The local copy on Node 1 (Copy A on Drive 1) is unavailable.
Remaining healthy copies exist on Node 2, Node 3, and Node 4.
Because read selection is topology‑agnostic, rack locality is not considered.
A healthy copy on Node 3 (Rack 2) is selected.
The read is serviced across racks, traversing the inter‑rack fabric.

Why the Old Design Was Suboptimal

In larger S2D deployments, this topology‑agnostic read selection can result in unnecessary cross‑rack or cross‑site traffic during steady‑state operations. While resiliency placement is correct, read performance becomes dependent on fabric conditions rather than locality, increasing latency variability and making the network part of the critical read path.

After Rack‑Local Reads

With the new Rack‑Local Reads, write behavior is unchanged. What changes is read selection. Instead of treating all healthy copies as equal, S2D now prefers the closest healthy copy based on topology. Reads are served from the nearest available copy in the fault domain—node, chassis, rack, or site—with remote locations used only when no closer copy exists.

As a result, read traffic remains local during steady‑state operations, even in multi‑rack and campus deployments.

Example: topology-aware read selection

This diagram shows the same cluster topology as the previous example, but with Rack‑Local Reads enabled. Read selection now accounts for rack locality when choosing among healthy copies.

Read selection flow:

A read request arrives on Node 1 in Rack 1, where the workload is running.
Storage Spaces Direct identifies all healthy copies of the requested data offset.
The local copy on Node 1 (Copy A on Drive 1) is unavailable.
Remaining healthy copies exist on Node 2, Node 3, and Node 4.
Rack‑Local Reads evaluates the proximity of each healthy copy based on the fault domain.
A healthy copy on Node 2 (Rack 1) is selected as the closest available copy.
The read is serviced within the local rack, avoiding cross‑rack traffic.

Why the New Design Is Better

By incorporating topology awareness into the read path, Rack‑Local Reads keep steady‑state read traffic local whenever a healthy copy is available. This preserves existing resiliency guarantees while reducing cross‑rack traffic, lowering latency, and making read performance less dependent on fabric conditions.

Summary:

Rack‑Local Reads separate resiliency from performance: rack‑aware mirroring determines where data is placed for protection, while Rack‑Local Reads determine which copy is used for performance.

Nothing changes for writes or resiliency. Data placement remains exactly the same. What changes is read selection—S2D now prefers the closest healthy copy in the fault domain, keeping steady‑state reads local when possible.

The result is lower and more predictable read latency, reduced east‑west storage traffic, and better scalability in multi‑rack and campus deployments—without sacrificing fault tolerance.

Microsoft runs on trust. We’d like to thank our customers and MVPs for that trust, and for the feedback that directly shaped Rack‑Local Reads. Your real‑world deployments continue to push Storage Spaces Direct to scale to larger and more complex environments.

Note: Posted by Rob for Parsan Saffaie.

Using Failover Clustering Cloud Witness with Managed Identity in Windows Server 2025

Rob-Hindman — Sun, 22 Mar 2026 00:41:39 GMT

Failover Clustering has a strong quorum model that is always used to prevent partition in space (AKA Split Brain, Network partition, Cluster partition). We require a cluster quorum resource (cluster witness) to be used on each failover cluster. Using a cluster quorum resource not only adds protection but also means that small two-node clusters can provide high availability for Hyper-V VMs, SQL Server Availability Sets, SQL Server Failover Cluster Instance, Scale-out File Server (SoFS), etc. workloads.

The Cloud Witness quorum resource was first introduced in the Failover Clustering feature in Windows Server 2016. It was a low-cost variation of the File Share Witness quorum resource that enabled effective low-cost two node failover clusters in scenarios where connectivity to Azure is reliable. The implementation of both the Cloud Witness quorum resource and the File Share Witness quorum resource is a Paxos tag, the date-time stamp of the Paxos tag, and filename is the GUID for the cluster is used. No other information is needed. While the Disk Witness quorum resource contains a full copy of the cluster database, the File Share Witness quorum resource and the Cloud Witness quorum resource only contain the Paxos tag, which is used as a tiebreaker when there is a partition in space, so that the cluster can continue to function.

The Cloud Witness quorum resource is created in an Azure storage account, and originally secured using a SAS token, called the StorageAccountAccessKey:

Set-ClusterQuorum -CloudWitness -AccountName <StorageAccountName> -AccessKey <StorageAccountAccessKey>

Previously, the SAS Token (StorageAccountAccessKey) for the Azure Storage Account was stored in the cluster database so that the cluster could access the for storage account. For details, see https://learn.microsoft.com/en-us/windows-server/failover-clustering/deploy-quorum-witness.

As a best practice, we are now recommending that the Azure storage account should be accessed using a Managed Identity instead of instead of using a SAS token. Only the name of the Managed Identity will be stored in the cluster database, making this practice more secure.

Steps to create a Cloud Witness quorum resource using managed identity while creating a new cluster:

Before creating the cluster, create a storage account resource, in this case we will create a storage account called cloudwitnessdemo.
Using the Azure Portal, create VMs in Azure IaaS running Windows Server 2025, and add the Failover Clustering feature. For physical on-premises servers running Windows Server 2025, add the Failover Clustering feature.
Install the latest updates for Windows Server 2025 from Windows Update on each server (AKA cluster node).
Connect each server (AKA cluster node) to Azure Arc – this will create a Managed Identity for the servers:

Using the Azure Portal Access Control pages, assign the Storage Blob Data Contributor role to node's managed identities:

Create the cluster using the New-Cluster cmdlet, for example:

New-Cluster -Name ExampleCluster -Node TOAD03H09-VM24,TOAD03H09-VM25,TOAD03H09-VM26,TOAD03H09-VM27 -NOSTORAGE

Create cloud witness using the cluster nodes’ managed identity:

Set-ClusterQuorum -CloudWitness -AccountName cloudwitnessdemo -UseManagedIdentity -Cluster ExampleCluster

Cluster QuorumResource

------- --------------

ExampleCluster Cloud Witness

Steps to create a Cloud Witness quorum resource using managed identity to an existing cluster:

Using the Azure Portal, create a storage account resource, in this case we will create a storage account called cloudwitnessdemo.
Install the latest updates for Windows Server 2025 from Windows Update on each server (AKA cluster node).
Connect each server (AKA cluster node) to Azure Arc – this will create a Managed Identity for the servers:
Assign Storage Blob Data Contributor role to node's managed identities.

Create cloud witness using the cluster nodes managed identities. This will delete the existing cloud witness and create the new one with managed identities configuration.

Set-ClusterQuorum -CloudWitness -AccountName cloudwitnessdemo -UseManagedIdentity -Cluster ExampleCluster

Cluster QuorumResource

------- --------------

ExampleCluster Cloud Witness

(Optional) Check the witness assignment and the use of Azure Managed Identity:

Get-ClusterResource -Cluster ExampleCluster -Name "Cloud Witness" | Get-ClusterParameter

Object Name Value Type

------ ---- ----- ----

Cloud Witness AccountName cloudwitnessdemo String

Cloud Witness EndpointInfo core.windows.net String

Cloud Witness ContainerName msft-cloud-witness String

Cloud Witness IsManagedIdentity 1 UInt32

Announcing Support for S2D Campus Cluster on Windows Server 2025

Rob-Hindman — Thu, 29 Jan 2026 17:51:53 GMT

Applies to: Windows Server 2025 Failover Clustering with the 2025-12 Security Update (KB5072033) (26100.7462) applied on each Failover Cluster node.

First, I need to THANK the Microsoft MVPs - external consultants that we work closely with – who proposed this configuration, advocated for it, and helped to validate it! MVPs, we listen to you! Thank You Very Much! Coffee is on me!

While this article applies to Windows Server 2025, please note that our friends and colleagues on the Azure Local product team call this configuration “Rack Aware Clustering” – see their articles here: Overview of Azure Local rack aware clustering (Preview) - Azure Local | Microsoft Learn

Note that the S2D Campus Cluster configuration supports two RACKs and is different than the S2D Stretch Cluster configuration that supports two geographically distant SITES. The S2D Stretch Cluster relies on Storage Replica to replicate volumes between sites, while the S2D Campus Cluster does not use Storage Replica and uses S2D replication between the cluster nodes.

What is a Campus?

Factories
Business Parks / Office Parks
Hospitals
School Campuses, College Campuses
Vessels / Cruise Ships
Stadiums
Any location with two strands of dark fibre cable between two rooms or buildings that can be defined as RACK fault domains in the failover cluster.

We think that there are many opportunities - especially in Europe - that require two separate data rooms to meet NIS2 requirements.

Included in the 2025-12 Security Update (KB5072033) (26100.7462) for Windows Server 2025 is the Rack Level Nested Mirror (RLNM) for S2D, which improves resiliency for the S2D Campus Cluster by keeping the same number of copies in each rack – one copy in each rack for a two-copy volume, and two copies per rack for a four copy volume. With a 2+2 S2D Campus Cluster – meaning two failover clustering nodes in each rack, a new level of resiliency can be achieved with a four copy volume: the RLNM will place a copy of data on each node, increasing resiliency to “Rack + Node” – meaning that you can lose a Rack AND a node and still have one copy of the data. We think that the 2+2 S2D Campus Cluster is a good tradeoff between cost and performance for many applications.

In the diagram below, note that seven servers are used: four servers are used for the S2D Campus Cluster (VMs, applications, storage), and there is an AD and DNS server in each rack, and a File Share Witness server.

In designing S2D Campus Cluster solutions, it is very important to find the correct balance between cost and hardware redundancy. Achieving low Recovery Time Objective (RTO) and Recovery Point Objective (RPO) can require investing in redundant hardware. The business cost of VM and application downtime needs to be clearly understood. Businesses like hotels and colleges will have different RTOs and RPOs than businesses like airports and oil refineries.

It’s also important to "practice failure" when validating S2D Campus Cluster solutions: test node failure, test rack failure, rack and node failure, test network switch failure, test disk failure – is the failover behavior acceptable? Was data corruption and data loss avoided?

What are the support requirements for the S2D Campus Cluster on Windows Server 2025?

Install the Windows Server 2025 12B CU (KB5072033) on every node in the failover cluster.
“Flat” S2D storage – all capacity drives, all flash meaning SSD or NVMe drives – avoid using a caching tier, avoid using HDDs.
Define exactly two RACK cluster fault domains – and place the cluster nodes in these two racks.
Follow hardware OEM guidelines.
Cluster quorum resource (File Share Witness, Disk Witness, Cloud Witness, or USB Witness) should be placed in a third room, separate from the data room containing the racks.
Recommended: Each rack should have a separate network path to the cluster quorum resource.
Recommended: Redundant TOR switches, CORE switches, and dedicated networks for S2D storage traffic to minimize Single Points Of Failure (SPOF) and maximize workload uptime and durability.
Recommended: 1ms latency (or less) between racks.
Recommended: RDMA NICs and switches - because RDMA NICs and switches can achieve 30% CPU savings!

Frequently Asked Questions (FAQs):

How can I measure network latency?

Answer: We recommend downloading the PsPing utility: PsPing - Sysinternals | Microsoft Learn

With the Rack Level Nested Mirror, can I automatically convert an existing campus cluster with a three copy volume to a four copy volume?

Answer: No, VMs will need to be stopped and data will need to be copied to an external location, RACK fault domains must be defined, and a new storage pool must be created to use the Rack Level Nested Mirror (RLNM). After the new storage pool is created, and volumes created from that storage pool, then data and VMs can be copied onto the new volumes.

Will you support 1+1, 2+2, 3+3, 4+4, and 5+5 S2D Campus Clusters configurations?

Answer: Yes. Note that Rack+Node resiliency is a special case with a 1+1 S2D Campus Cluster using a two-copy volume and a 2+2 S2D Campus Cluster using a four-copy volume.

Can I create as many two-copy and four-copy volumes as I need, or are there limits?

Answer: Yes, you can create as many two-copy and four-copy volumes as you need, given the capacity of the storage pool. There is a trade-off between resiliency and capacity: a two-copy volume delivers 50% capacity, and a four-copy volume delivers 25% capacity. We recommend that valuable VMs, applications, and data should be stored on four-copy volumes.

For the S2D Campus Cluster on Windows Server 2025, are redundant core switches required? Are two TOR switches per RACK required?

Answer: Core switches are recommended but optional. A single TOR switch per RACK is acceptable, but we note that it’s a single point of failure. Network infrastructure investments (switches, NICs, cabling) corresponding to the Recovery Time Objective (RTO) and Recovery Point Objective (RPO) requirements of the business.

If you would like to see additional scenarios supported, please let us know so that we can prioritize development and validation and address your business needs!

Please send your questions, comments, and feature requests about S2D and SAN Coexistence to: wsfc_s2dcampuscluster@microsoft.com

PowerShell Script for Deploying S2D Campus Cluster:

#Create a test cluster but do not create storage:

New-Cluster -Name TestCluster -Node Node1, Node2, Node3, Node4 -NoStorage

#Define the fault domains for the cluster – two nodes are in “Room1” and two nodes are in “Room2”:

Set-ClusterFaultDomain -XML @”<Topology><Site Name=”Redmond”><Rack Name=”Room1”><Node Name=”Node1”/><Node Name=”Node2”/></Rack><Rack Name=”Room2”><Node Name=”Node3”/> <Node Name=”Node4”/></Rack></Site></Topology>”@@

#Alternatively you can define the fault domains using New-ClusterFaultDomain and Set-ClsuterFaultDomain cmdlets:

#New-ClusterFaultDomain -Name Room1 -FaultDomainType Rack

#New-ClusterFaultDomain -Name Room2 -FaultDomainType Rack

#Set-ClusterFaultDomain -Name Room1 -FaultDomain Redmond

#Set-ClusterFaultDomain -Name Room2 -FaultDomain Redmond

#Set-ClusterFaultDomain -Name Node1 -FaultDomain Room1

#Set-ClusterFaultDomain -Name Node2 -FaultDomain Room1

#Set-ClusterFaultDomain -Name Node3 -FaultDomain Room2

#Set-ClusterFaultDomain -Name Node4 -FaultDomain Room2

#Note that you can check your fault domains using the Get-ClusterFaultDomain cmdlet.

#Add Storage Spaces Direct (S2D) Storage to the cluster – note that Enable-ClusterS2D cmdlet can also be used:

Enable-ClusterStorageSpacesDirect

#Update the Storage Pool

Get-StoragePool S2D* | Update-StoragePool

#Check that the StoragePool version is 29:

(Get-CimInstance -Namespace root/microsoft/windows/storage -ClassName MSFT_StoragePool -Filter 'IsPrimordial = false').CimInstanceProperties['Version'].Value

#Check that the Storage Pool’s FaultDomainAwareness property is set to StorageRack:

Get-storagepool -FriendlyName <S2DStoragePool> | fl

#Note that resiliency is specified when the volume is created – our idea with the S2D Campus cluster is that IT Admins can create as many two-copy and four-copy volumes as needed. Valuable workload (VMs) and data should go on the 4-copy volumes.

#Create a four-copy volume on the storage pool, fixed provisioned:

New-Volume -FriendlyName “FourCopyVolumeFixed” -StoragePoolFriendlyName S2D* -FileSystem CSVFS_ReFS –Size 500GB -ResiliencySettingName Mirror -PhysicalDiskRedundancy 3 -ProvisioningType Fixed -NumberOfDataCopies 4 –NumberOfColumns 3

#Optional - Create a four-copy volume on the storage pool, thinly provisioned:

New-Volume -FriendlyName “FourCopyVolumeThin” -StoragePoolFriendlyName S2D* -FileSystem CSVFS_ReFS –Size 500GB -ResiliencySettingName Mirror -PhysicalDiskRedundancy 3 –ProvisioningType Thin -NumberOfDataCopies 4 –NumberOfColumns 3

#Optional - Create a two-copy volume on the storage pool, fixed provisioned:

New-Volume -FriendlyName “TwoCopyVolumeFixed” -StoragePoolFriendlyName S2D* -FileSystem CSVFS_ReFS –Size 500GB -ResiliencySettingName Mirror -PhysicalDiskRedundancy 1 -ProvisioningType Fixed

#Optional - Create a two-copy volume on the storage pool, thinly provisioned:

New-Volume -FriendlyName “TwoCopyVolumeThin” -StoragePoolFriendlyName S2D* -FileSystem CSVFS_ReFS –Size 500GB -ResiliencySettingName Mirror -PhysicalDiskRedundancy 1 -ProvisioningType Thin

How can you increase the S2D Campus Cluster from 1+1 to 2+2 to 3+3 to 4+4 to 5+5?

Add one node at a time to a rack and then to the cluster - first to the Rack fault domain using New-ClusterFaultDomain, then to the cluster using Add-ClusterNode, for example:

$nodeName = ‘Node5’

$rackFaultDomainName = ‘Room1’

New-ClusterFaultDomain -Type Node -Name $nodeName -FaultDomain $rackFaultDomainName

Add-ClusterNode -Name $nodeName

$nodeName = ‘Node6’

$rackFaultDomainName = ‘Room2’

New-ClusterFaultDomain -Type Node -Name $nodeName -FaultDomain $rackFaultDomainName

Add-ClusterNode -Name $nodeName

Here are some additional articles on the S2D Campus Cluster for WS 2025:

Luka Manojlovic (MVP) Blogs: Windows Server 2025 – Storage Spaces Direct (S2D) Campus Cluster – part 1 – Preparation and deployment | Luka Manojlovic

Windows Server 2025 – Storage Spaces Direct (S2D) Campus Cluster – How does it handle problems / disaster scenarios – part 2 – sequence of failures – N2 in Rack 2, disk in N1 in Rack 2, N1 in Rack 2 (whole Rack 2 offline), N2 in Rack 1 – cluster quorum failure … | Luka Manojlovic

Windows Server 2025 – Storage Spaces Direct (S2D) Campus Cluster – How does it handle problems / disaster scenarios – part 3 – Bring online the only remaining node – FixQuorum, Force Cluster Start … | Luka Manojlovic

Darryl van der Peijl (MVP) Blog: Windows Server Campus Clusters - Splitbrain

Weithenn Wang (MVP) Blog: S2D Campus Cluster on Windows Server 2025 ~ 不自量力の Weithenn

Announcing Support for S2D and SAN Coexistence

Rob-Hindman — Fri, 06 Mar 2026 00:09:39 GMT

Applies to: Windows Server 2022 and Windows Server 2025 Failover Clustering

Failover Clustering has supported SAN storage since Windows NT 4.0 Server, Enterprise Edition was released in 1997!

In addition, Failover Clustering has supported Storage Spaces Direct (S2D) since Windows Server 2016!

Many customers have been asking if they can use (or reuse) their existing iSCSI and Fibre Channel SANs together with S2D – and now the answer is Yes!

Now Windows Server 2022 and Windows Server 2025 Failover Clusters can use both S2D Cluster Shared Volumes (CSVs) and separate SAN CSVs (Fiber Channel, iSCSI connected LUN’s) together in the same single rack cluster.

This is more evidence that we listen to your feedback! 😊Customers with existing SANs – many of them looking for a new virtualization platform – have told us that they want to use their SANs with Hyper-V and S2D.

Note that Hyper-V VM Storage Live Migration (“Move Virtual Machine Storage”) works great for moving VMs between S2D and SAN storage without any VM disruption!

Why are we adding support for S2D and SAN coexistence in the same single rack failover cluster?

If you want to use your SAN(s) to migrate data and VMs to and from S2D storage. For example, over time, workload requirements can change, and you may use your SAN(s) to move data and VMs between clusters, or you might choose to move data and VMs from SAN storage to S2D for faster data access for the workloads depending on usage or demand.
If you want to use your SANs for backup and restore data and VMs, which can add protection against ransomware.
If you want to use S2D storage and/or SAN storage for data ingestion for AI and ML training and inference. For example, AI model training may be faster using S2D storage in some cases, but SAN storage could be used to store multiple versions of the model as it’s being developed and validated. More choices for data management!

What are the support requirements for using SAN storage (SAN LUNs) (Fibre Channel, iSCSI, and iSCSI Target) together with S2D storage CSVs in the same single rack cluster?

SAN disks are not included in a Storage Spaces Direct storage pool. Ensure that only Direct Attached Storage (DAS) drives are used in S2D:
1. - PowerShell: Get-PhysicalDisk -CanPool $True
SAN volumes should be formatted as NTFS before being added to CSV.
S2D volumes should be formatted as ReFS before being added to CSV.

How can I create a single rack failover cluster with both S2D and SAN storage?

Create a failover cluster that initially doesn’t use storage:
- - PowerShell: New-Cluster -Name <clustername> -Node <node1name>,<node2name>,<node3name> -NoStorage
Enable Storage Spaces Direct – this will add eligible non-boot DAS drives to the S2D Storage Pool:
- - PowerShell: Enable-ClusterS2D
Create volumes from the S2D Storage Pool, format using ReFS, and add the Cluster Shared Volumes:
1. - PowerShell: New-Volume -StoragePoolFriendlyName S2D* -FriendlyName S2DVDisk01 -FileSystem CSVFS_REFS -Size 200GB
Configure MPIO (if required) for SAN LUNs, add them to the failover cluster’s Available Storage (“Add Disk”), and then add them to CSV (“Add to Cluster Shared Volumes”)

How can I validate both S2D and SAN storage in the same single rack failover cluster? It’s important to check that all the nodes in the cluster are connected to the SAN LUNs and that S2D is configured correctly. Here’s how you can check:

Schedule a maintenance window when VMs and other roles running on the SAN LUNs can be stopped – OR - move the VMs and other roles off the CSV disks based on SAN LUNs and onto the CSV disks based on S2D.
Take CSV disks based on the SAN LUNs Offline, I.E., “Take Offline”
CSV disks using S2D can remain online:
Run cluster validate for both “Storage” and “Storage Spaces Direct” validation suites:
1. - PowerShell: Test-Cluster –Include “Storage”, “Storage Spaces Direct”
Both SCSI-3 PR tests for SAN storages and S2D tests will be performed and a validation report generated.
Bring SAN LUNs Online, I.E., “Bring Online”, and restart VMs or other roles if they were stopped.

Note – remember to avoid renaming or evicting cluster nodes using S2D storage.

We recognize that some management features are different between S2D and SAN LUNs, but we think that most users will benefit from being able to combine S2D and SAN storage.

For more details on Failover Cluster storage architectures, see this article: Failover Clustering Storage Architectures in Windows Server | Microsoft Learn

To see SAN Storage systems that have qualified for the Windows Server 2022 or Windows Server 2025 logo, browse the Windows Server Catalog Hardware: Hardware | Windows Server Catalog

If you would like to see additional scenarios supported, please let us know so that we can prioritize development and validation and address your business needs!

Please send your questions, comments, and feature requests about S2D and SAN Coexistence to: WSFC_S2DandSAN@microsoft.com

-Rob.

Pleased to see that Hitachi Vantara is supporting S2D and SAN Coexistence: WSFC 2025 (Hyper-V) — Blended Storage Coexistence (S2D + Hitachi VSP One Block SAN) (also on LinkedIn here: (33) WSFC 2025 (Hyper-V) — Blended Storage Coexistence (S2D + Hitachi VSP One Block SAN) | LinkedIn)

-RH.

New Cluster-Wide Control For Virtual Machine Live Migrations In Windows Server and Azure Stack HCI

Steven Ekren — Thu, 05 Jan 2023 17:57:36 GMT

Applies to: Windows Server 2022, Azure Stack HCI, version 21H2 and later versions of both

Overview:

There is a new enhancement in the ability to manage the number of parallel live migrations within a cluster, making it easier to change and ensuring consistency. Previously, changing it required setting it on each node of the cluster, and remembering to set it when a new server is added to the cluster. This meant it was easy to have inconsistencies across the nodes.

Hyper-v has a setting to limit the number of live migrations that a server can participate in. If an administrator wanted to change this value to be optimized for their systems, they would have to go to each node of a failover cluster and change the per-server Hyper-V property. They would also have to remember to set this property for any new node added to the cluster. This meant that it was difficult to ensure consistency over time.

With the installation of the September 2022 Windows Update package or later, The new cluster property MaximumParallelMigrations was added that allows an administrator to set the value once and have each node of the cluster inherit the setting. When new servers are added to the cluster, the cluster value will be inherited. This ensures consistency and makes it easy to adjust the system.

Using the new property:

The default when the update is applied is MaximumParallelMigrations=1.

To get the current value the cluster property:

(Get-Cluster).MaximumParallelMigrations

The value can be changed by using the command:

(Get-Cluster).MaximumParallelMigrations=<value>

To view the nodes property:

Get-VMHost -ComputerName <name1>,<name2>,<name3> | FT Name, MaximumVirtualMachineMigrations

More Information

The first time a monthly update with this change is applied, the local MaximumParallelMigrations setting will be converted to a cluster level setting and set to 1. Based on testing this is the recommended default with the safest and most reliable value as far as reliability with live migrations across the various types of systems deployed today. If this parameter is changed by an administrator after it is added to the system via a monthly update, the new value will persist.

We recommend you test and validate the reliability of more than one parallel live migration with your hardware and in your environment before setting to a higher number.

References

Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure - Microsoft Tech Community

Set-VMHost (Hyper-V) | Microsoft Learn

2022-09 Cumulative Update for Azure Stack HCI, version 20H2 and Windows Server 2019 Datacenter: Azure Edition for x64-based Systems (KB5017311)

New features of Windows Server 2022 Failover Clustering

John Marlin — Wed, 01 Sep 2021 20:02:49 GMT

Greetings again Windows Server and Failover Cluster fans!! John Marlin here and I own the Failover Clustering feature within the Microsoft product team. In this blog, I will be giving an overview of the new features in Windows Server 2022 Failover Clustering. Some of these will be talked about at the upcoming Windows Server Summit. One note that I will say is that this particular blog post will not cover the new features for Azure Stack HCI version 21H2. That is another blog for another time.

So let's get this started.

Clustering Affinity and AntiAffinity

Affinity is a rule you would set up that establishes a relationship between two or more roles (i,e, virtual machines, resource groups, and so on) to keep them together. AntiAffinity is the same but is used to try to keep the specified roles apart from each other. In Azure Stack HCI version 20H2, we added this and now brought it over to Windows Server as well. In previous versions of Windows Server, we only had AntiAffinity capabilities. This was with the use of AntiAffinityClassNames and ClusterEnforcedAntiAffinity. We took a look at what we were doing and made it better. Now, not only do we have AntiAffinity, but also Affinity. You can configure this new Affinity and AntiAffinity with PowerShell commands and have four options.

Same Fault Domain
Same Node
Different Fault Domain
Different Node

The below doc discusses the feature in more detail including how to configure it.

Cluster Affinity

https://docs.microsoft.com/en-us/azure-stack/hci/manage/vm-affinity

For those that still use AntiAffinityClassNames, we will still honor it. Which means, upgrading to Windows Server 2022

AutoSites

AutoSites is another feature brought over from Azure Stack HCI. AutoSites is basically what is says. When you configure Failover Clustering, it will first look into Active Directory to see if Sites are configured. For example:

If they are and the nodes are included in a site, we will automatically create site fault domains and put the nodes in the fault domain they are a member of. For example, if you had two nodes in a Redmond site and two nodes in a Seattle site, it would look like this once the cluster is created.

As you can see, we will create the site fault domain name the same as what it is in Active Directory.

If sites are not configured within Active Directory, we will then look at the networks to see if there are differences as well as networks common to each other. For example, say you had the nodes with this network configuration:

Node1 = 1.0.0.11 with subnet 255.0.0.0

Node2 = 1.0.0.12 with subnet 255.0.0.0

Node3 = 172.0.0.11 with subnet 255.255.0.0

Node1 = 172.0.0.12 with subnet 255.255.0.0

We will see this as multiple nodes in one subnet and multiple nodes in another. Therefore, these nodes are in separate sites and it will configure sites for you automatically. With this configuration, it will create the site fault domains with the names of the networks. For example:

This will make things easier when you want to create a stretched Failover Cluster. Please note that Storage Spaces Direct cannot be stretched in Windows Server 2022 as it can be in Azure Stack HCI.

Granular Repair

Since we just mentioned Storage Spaces Direct, one of the talked about features of it is repair. As a refresher, as data is written to drives, it is spread throughout all drives on all the nodes.

When a node goes down for maintenance, crashes, or whatever the case may be, once it comes back up, there is a "repair" job run where data is moved around and onto the drives, if necessary, of the node that came back. A repair is basically a resync of the data between all the nodes. Depending on the amount of time the node was down, the longer it could take for the repair to complete. A repair in previous versions would take the extent (block of data) that is normally 1 gigabyte or 256 megabyte in size and resync it in its entirety. It did not matter how much of the extent was changed (for example 1 kilobyte), the entire extent is copied.

In Windows Server 2022, we have changed this thinking and now work off of "sub-extents". A sub-extent is only a portion of the entire extent. This is normally set at the interleave setting which is 256 kilobytes. Now, when 1 kilobyte of a 1 gigabyte extent is changed, we will only move around the 256 kilobyte sub-extent. This will make repair times much faster and quicker to complete.

One other thing we considered was, when a repair/resync occurs, it can affect production due to the CPU resources it must use. To combat that, we also added the capability to throttle the resources up or down, depending on when it may be done. For example, if you need a repair/resync to run during production hours, you need to keep performance of your production needs to remain up. Therefore, you may want to set it on low so it more runs in the background. However, if you were to do it overnight on a weekend, you can afford to crank it up to a higher setting so it completes faster.

The storage speed repair settings are:

Setting	Queue depth	Resource allocation
Very low	1	Most resources to active workloads
Low	2	More resources to active workloads
Medium (default)	4	Balances workloads and repairs
High	8	More resources to resync and repairs
Very high	16	Most resources to resync and repairs

For more information regarding resync speeds, please refer to the below article:

Adjustable storage repair speed in Azure Stack HCI and Windows Server

https://docs.microsoft.com/en-us/azure-stack/hci/manage/storage-repair-speed

Cluster Shared Volumes and Bitlocker

Cluster Shared Volumes (CSV) enable multiple nodes in a Windows Server Failover Cluster or Azure Stack HCI to simultaneously have read-write access to the same LUN (disk) that is provisioned as an NTFS volume. BitLocker Drive Encryption is a data protection feature that integrates with the operating system and addresses the threats of data theft or exposure from lost, stolen, or inappropriately decommissioned computers.

BitLocker on volumes within a cluster are managed based on how the cluster service "views" the volume to be protected. BitLocker will unlock protected volumes without user intervention by attempting protectors in the following order:

Clear Key
Driver-based auto-unlock key
ADAccountOrGroup protector
- Service context protector
- User protector
Registry-based auto-unlock key

Failover Cluster requires the Active Directory-based protector option (#3 above) for a cluster disk resource or CSV resources. The encryption protector is a SID-based protector where the account being used is Cluster Name Object (CNO) that is created in Active Directory. Because it is Active Directory-based, a domain controller must be available in order to obtain the key protector to mount the drive. If a domain controller is not available or slow in responding, the clustered drive is not going to mount.

With this thinking, we needed to have a "backup" plan. With Windows Server 2022, when a drive is enabled for Bitlocker encryption while it is a part of Failover Cluster, we will now create an additional key protector just for cluster itself. By doing this, it will still go out to a domain controller first to get the key. If the domain controller is not available, it will then use the locally kept additional key to mount the drive. The default will always be to go to the domain controller first. We have also built in the ability to manually mount a cluster drive using new PowerShell cmdlets and passing the locally kept recovery key.

Another thing about this is that it now opens up the ability to Bitlocker drives that are a part of a workgroup or cross-domain cluster where a Cluster Name Object does not exist.

SMB Encryption

Windows Server 2022 SMB Direct now supports encryption. Previously, enabling SMB encryption disabled direct data placement, making RDMA performance as slow as TCP. Now data is encrypted before placement, leading to relatively minor performance degradation while adding AES-128 and AES-256 protected packet privacy. You can enable encryption using Windows Admin Center, Set-SmbServerConfiguration, or a Universal Naming Convention (UNC) Hardening group policy. Furthermore, Windows Server Failover Clusters now support granular control of encrypting intra-node storage communications for Cluster Shared Volumes (CSV) and the storage bus layer (SBL). This means that when using Storage Spaces Direct and SMB Direct, you can decide to encrypt the east-west communications within the cluster itself for higher security.

New Cluster Resource Types

Cluster resources are categorized by type. Failover Clustering defines several types of resources and provides resource DLLs to manage these types. In Windows Server 2022, we have added three new resource types.

HCS Virtual Machine

Building a great management API for Docker was important for Windows Server Containers. There's a ton of really cool low-level technical work that went into enabling containers on Windows, and we needed to make sure they were easy to use. This seems very simple, but figuring out the right approach was surprisingly tricky. Our first thought was to extend our existing management technologies (e.g. WMI, PowerShell) to containers. After investigating, we concluded that they weren’t optimal for Docker, and started looking at other options.

After a bit of thinking, we decided to go with a third option. We created a new management service called the Host Compute Service (HCS), which acts as a layer of abstraction above the low level functionality. The HCS was a stable API Docker could build upon, and it was also easier to use. Making a Windows Server Container with the HCS is just a single API call. Making a Hyper-V Container instead just means adding a flag when calling into the API.

Looking at the architecture in Linux:

Looking at the architecture in Windows:

HCS Virtual Machine lets you create a virtual machine using the HCS APIs rather than the Virtual Machine Management Service (VMMS).

NFS Multi Server Namespace

If you are not familiar with an NFS Multi Server Namespace, think of a tree with several branches. An NFS Multi Server Namespace allows for the single namespace to extend out to multiple servers by the use of a referral. With this referral, you can integrate data from multiple NFS Servers into a single namespace. NFS Clients would connect to this namespace and be referred to a selected NFS Server from one of its branches.

Storage Bus Cache

In all things transparent, this one is a little bit of a reach, but Failover Clustering is needed (sort of).

The storage bus cache for Storage Spaces on standalone servers can significantly improve read and write performance, while maintaining storage efficiency and keeping the operational costs low. Similar to its implementation for Storage Spaces Direct, this feature binds together faster media (for example, SSD) with slower media (for example, HDD) to create tiers. By default, only a portion of the faster media tier is reserved for the cache.

What makes this the bit of a reach is that in order to use storage bus cache, the Failover Clustering feature must be installed but the machine cannot be a member of a Cluster. I.E. Add the feature and move on.

More information on Storage Bus Cache can be found here:

Tutorial: Enable storage bus cache with Storage Spaces on standalone servers

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-storage-bus-cache

Thanks

John Marlin

Senior Program Manager

Twitter: @Johnmarlin_MSFT

Failover Clustering in Azure

John Marlin — Fri, 16 Jul 2021 18:38:13 GMT

Azure is a cloud computing platform with an ever-expanding set of services to help you build solutions to meet your business goals. Azure services range from simple web services for hosting your business presence in the cloud to running fully virtualized computers for you to run your custom software solutions. With over 60 regions globally, 200+ products, and over 17,000 services and applications, Azure has everything you need in a cloud.

One of the products that can server as the compute infrastructure for our service or application is Failover Clustering. Failover Clustering can be a traditional cluster or it can be running Storage Spaces Direct. No matter the choice, there are a few configuration changes that must be made post cluster creation to ensure connectivity can be made. Starting in Windows Server 2019, and moving forward, we have added detection into the cluster creation process that will automatically do some of this configuration for you.

Let's first talk about the Cluster Network Name. The Cluster Network Name is used to provide an alternate computer name for an entity that exists on a network. When it is created, it will also create a Cluster IP Address resource that provides an identity to the group, allowing the group to be accessed by network clients. When in Azure, an additional Azure Load Balancer must be created with separate a IP Address so that it can be reached.

In Windows Server 2019, and moving forward, we have added detection during the cluster creation process to look to see if it is being created in Azure. A new parameter has been added to Clustering to help you determine what we have detected. To view it and see the output, the command to run would be:

Get-Cluster | fl DetectedCloudPlatform

DetectedCloudPlatform : Azure

As a side note, if it detects it is on-premises or any other cloud provider, the response will be None.

If so, there are several configurations it will add and the first is with the Cluster Name. Instead of the traditional Cluster Name and Cluster IP Address, it will now create the Cluster Name as a distributed network name (DNN) automatically. If you have worked with Scale Out File Servers (SOFS), it is the same type distributed name. A Distributed Network Name is a name in the Cluster that does not use a clustered IP Address. It is a name that is published in DNS using the IP Addresses of all the nodes in the Cluster. Since it uses the IP Addresses of the nodes, a load balancer is not needed. So it would look like this from Failover Cluster Manager.

As a side note, the automatic creation of the name as a DNN is only when the machines are in Azure. However, we have added the ability to create it as a DNN on-premises if it is so desired. When creating the Cluster using Failover Cluster Manager or Windows Admin Center on-premises, it will create it with the name and IP Address. However, using PowerShell, you have a new switch –ManagementPointNetworkType that can be used with New-Cluster that will create it as a DNN. –ManagementPointNetworkType has several parameters to define the type of name it will be.

New-Cluster -ManagementPointNetworkType:x

Singleton : Traditional Cluster Name and Cluster IP Address
Distributed : Create as DNN and use node IP Addresses
Automatic : Detect if on-premises or Azure (default)

Moving on, one of the next things we will change is the network communication thresholds. Communication between nodes is crucial in keeping them up and talking to ensure high availability. As a refresher, you have several settings that control the length of wait times and number of failures before we determine a node to be down and it is removed from cluster membership. As a refresher, these are those settings.

Parameter	Windows 2019 / Azure Stack HCI Default	Maximum
SameSubnetDelay	1 second	2 seconds
SameSubnetThreshold	10 heartbeats	120 heartbeats
CrossSubnetDelay	1 second	4 seconds
CrossSubnetThreshold	20 heartbeats	120 heartbeats
CrossSiteDelay	1 second	4 seconds
CrossSiteThreshold	20 heartbeats	120 heartbeats

It is important to understand that both the delay and threshold have a cumulative effect on the total health detection. For example setting SameSubnetDelay to send a heartbeat every 1 seconds and setting the SameSubnetThreshold to 10 heartbeats missed before taking recovery, means that the cluster can have a total network tolerance of 10 seconds before recovery action is taken. The higher the numbers, the longer it will take to detect a node is not responding. In general, continuing to send frequent heartbeats but having greater thresholds is the preferred method. The primary scenario for increasing the Delay, is if there are ingress / egress charges for data sent between nodes. When we have detected that the cluster in in Azure, we will auto increase the thresholds to their maximum values.

Please refer to the Tuning Failover Cluster Network Thresholds blog to change these values.

The last thing I want to talk about is Azure host maintenance. Maintenance on an compute host is something you cannot get around as patches, driver/firmware updates, etc need to be done periodically. Same goes for those hosts in Azure or any other cloud provider. So what to do with the virtual machines running on those hosts is something that needs to be considered by the Azure administrators. There is basically only a couple of things that they can do which is leave the VMs where they are or move them off. The decision to move or stay can simply come down to how long is it going to take and does it need a reboot. No matter how quick it may take to apply, if a reboot is needed, the VMs are going to move off. However, if whatever maintenance being done doesn't need a reboot an is quick, simply freezing the virtual machine is done.

As a client, you very well never know anything ever happened and that is the goal. But there could be times when you notice it as you cannot connect, you are hung, a cluster node drops out of membership, etc. From a client perspective there is not a way of knowing what had happened. You must trust that the administrators have no issues and they make the right decisions.

But what if you as an administrator received a heads up of impending host maintenance and you could make the decision. Well, that leads to the other new feature we added. With Windows Server 2019, we added integration and awareness of Azure Host Maintenance and improved experience by monitoring for Azure Scheduled Events. For this to fully be done, all clustered VMs must be in the same Azure Availability Zone.. When a host has maintenance scheduled, we will now detect it and throw an event into the virtual machine's FailoverClustering/Operational channel. We have also included actions that you can configure based on the event.

First, let's talk about the events you could see. This is an example of one of those events.

Log: FailoverClustering/Operational
Level: Warning
Event ID: 1139
symbol="NODE_MAINTENANCE_DETECTED”
Description: The cluster service has detected an Azure host maintenance event has been scheduled. This maintenance event may cause the node hosting the virtual machine to become unavailable during this time.

Node: VMNode1
Approximate Time: 2021/07/16-17:30:00.000
Details: ' EventId = 4FE57A76-7754-48FD-9B45-48387A36CD19
EventStatus = Scheduled Event
Type = Freeze Resource
Type = VirtualMachine

As you can see, this event triggered as a host maintenance event has been scheduled. It provides several other things of interest.

1. The time the event is to occur

2. What the event will be someone from the Azure Team could look up if a support ticket were raised

3. What it will do with the virtual machine

There are actually 3 events you could see.

Event ID 1136: Host maintenance is imminent and about to occur

Event ID 1139: Host maintenance has been detected

Event ID 1140: Host maintenance has been rescheduled

Now that you have the events, the next thing is to decide if you want to define an action. We have created two new cluster properties of DetectManagedEvents and DetectManagedEventsThreshold. DetectManagedEvents is for the action you wish to have occur when it detects an event is scheduled. DetectManagedEventsThreshold The options for each of these are as follows:

DetectManagedEvents

0 = Do not Log Azure Scheduled Events <-- default for on-premises
1 = Log Azure Scheduled Events <-- default in Azure
2 = Avoid Placement (don’t move roles to this node)
3 = Pause and drain when Scheduled Event is detected
4 = Pause, drain, and failback when Scheduled Event is detected

DetectManagedEventsThreshold

60 seconds <-- default
Amount of time before taking action

Note: These settings only apply when the virtual machine is in Azure. It does not take effect on any other platform (I.E. a third party cloud provider, Hyper-V, Azure Stack HUB/HCI/Edge, etc).

In closing, we recognized that there are some configurations needed when a Failover Cluster is in Azure. By adding these new features, we have taken some of the burden away from you as an administrator and automatically making these changes for you.

Thanks

John Marlin

Senior Program Manager

Twitter: @Johnmarlin_MSFT

Security Settings for Failover Clustering

John Marlin — Mon, 06 Jun 2022 23:19:48 GMT

Security is at the forefront of many administrator's minds and with Failover Clustering, we did some improvements with Windows Server 2019 and Azure Stack HCI with regards to security.

Since the beginning of time, Failover Clustering has always had a dependency on NTLM authentication. As the versions came and went, a little more of this dependency was removed. Now, with Windows Server 2019 Failover Clustering, we have finally removed all of these dependencies. Instead Kerberos and certificate-based authentication is used exclusively. There are no changes required by the user, or deployment tools, to take advantage of this security enhancement. It also allows failover clusters to be deployed in environments where NTLM has been disabled.

This goes for the bootstrapping of the cluster to the starting of the resources and drives. With the bootstrapping process, the need of an Active Directory domain controller is also no longer needed. As explained in this blog, we have a local user account (CLIUSR) that is used for various things now. In conjunction of this account as well as the use of certificates:

Cluster Service starts and forms the cluster
Other nodes will join the cluster
Drives (including Cluster Shared Volumes) will come online
Groups and resources start coming online.

This is especially beneficial if you have a domain controller is virtualized running on the cluster, preventing the "chicken or the egg" scenario.

Another security concern that administrators have is what is out on the wire. There are a couple of security settings to consider with regards to communications between the nodes and and storage. From a storage perspective, there is Cluster Shared Volume (CSV) traffic for any redirected data and Storage Bus Layer (SBL) traffic, if using Storage Spaces Direct.

Let's first talk about cluster communications. Cluster communications could contain any number of things and what an admin would like is to prevent anything from picking it up on the network. As a default, all communication between the nodes are sent signed, making the use of certificates. This may be fine when all the cluster nodes reside in the same rack. However, when nodes are separated in different racks or locations, an admin may wish to have a little more security and make use of encryption.

This setting is controlled by the Cluster property SecurityLevel and has three different levels.

0 = Clear Text

1 = Signed (default)

2 = Encrypted

If the desire is to change this to encrypted communications, the command to run would be:

(Get-Cluster).SecurityLevel = 2

The other bit of communication between the nodes would be with the storage. Both Cluster Shared Volumes (CSV) has traffic on the wire and if using Storage Spaces Direct, you have the Storage Bus Layer (SBL) traffic. For these bits of traffic, the default is to send everything in clear text. Admins may decide they wish to secure this type of data traffic to lock it down and prevent sniffer traces from picking anything up.

This setting is controlled by the Cluster property SecurityLevelToStorage and has three different levels.

0 = Clear Text (default)

1 = Both CSV and SBL traffic are signed

2 = Both CSV and SBL traffic are encrypted

If the desire is to change this to encrypted communications, the command to run would be:

(Get-Cluster).SecurityLevelToStorage = 2

One caveat to the SecurityLevel and SecurityLevelToStorage that must be taken into consideration. These forms of communication are using SMB. When using a form of encryption on the network with SMB, RDMA is not used. Therefore, if you are using this on RDMA network cards, RDMA is not used and can cause a performance impact. Microsoft is aware of this impact and working on correcting this for a later version. For more information on this, please refer to the following document.

Reduced networking performance after you enable SMB Encryption or SMB Signing in Windows Server 2016

Reduced performance after SMB Encryption or SMB Signing is enabled - Windows Server | Microsoft Docs

Thanks

John Marlin

Senior Program Manager

Twitter: @Johnmarlin_MSFT

Failover Clustering Networking Basics and Fundamentals

John Marlin — Thu, 24 Sep 2020 01:19:20 GMT

My name is John Marlin and I am with the High Availability and Storage Team. With newer versions of Windows Server and Azure Stack HCI on the horizon, it’s time to head to the archives and dust off some old information as they are in need of updating.

In this blog, I want to talk about Failover Clustering and Networking. Networking is a fundamental key with Failover Clustering that sometimes is overlooked but can be the difference in success or failure. In this blog, I will be hitting on all facets from the basics, tweaks, multi-site/stretch, and Storage Spaces Direct. By no means should this be taken as a “this is a networking requirement” blog. Treat this as more of general guidance with some recommendations and things to consider. Specific requirements for any of our operating systems (new or old) will be a part of the documentation (https://docs.microsoft.com) of the particular OS.

In Failover Clustering, all networking aspects are provided by our Network Fault Tolerant (NetFT) adapter. Our NetFT adapter is a virtual adapter that is created with the Cluster is created. There is no configuration necessary as it is self-configuring. When it is created, it will create its MAC Address based off of a hash of the MAC Address of the first physical network card. It does have conflict detection and resolution built in. For the IP Address scheme, it will create itself an APIPA IPv4 (169.254.*) and IPv6 (fe80::*) address for communication.

Connection-specific DNS Suffix . :

Description . . . . . . . . . . . : Microsoft Failover Cluster Virtual Adapter

Physical Address. . . . . . . . . : 02-B8-FA-7F-A5-F3

DHCP Enabled. . . . . . . . . . . : No

Autoconfiguration Enabled . . . . : Yes

Link-local IPv6 Address . . . . . : fe80::80ac:e638:2e8d:9c09%4(Preferred)

IPv4 Address. . . . . . . . . . . : 169.254.1.143(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.0.0

Default Gateway . . . . . . . . . :

DHCPv6 IAID . . . . . . . . . . . : 67287290

DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-26-6B-52-A5-00-15-5D-31-8E-86

NetBIOS over Tcpip. . . . . . . . : Enabled

The NetFT adapter provides the communications between all nodes in the cluster from the Cluster Service. To do this, it discovers multiple communication paths between nodes and if the routes are on the same subnet or cross subnet. The way it does this is through “heartbeats” through all network adapters for Cluster use to all other nodes. Heartbeats basically serve multiple purposes.

Is this a viable route between the nodes?
Is this route currently up?
Is the node being connected to up?

There is more to heartbeats, but will defer to my other blog No Such Thing as a Heartbeat Network for more details on it.

For Cluster communication and heartbeats, there are several considerations that must be taken into account.

Traffic uses port 3343. Ensure any firewall rules have this port open for both TCP and UDP
Most Cluster traffic is lightweight.
Communication is sensitive to latency and packet loss. Latency delays could mean performance issues, including removal of nodes from membership.
Bandwidth is not as important as quality of service.

Cluster communication between nodes is crucial so that all nodes are currently in sync. Cluster communication is constantly going on as things progress. The NetFT adapter will dynamically switch intra-cluster traffic to another available Cluster network if it goes down or isn’t responding.

The communications from the Cluster Service to other nodes through the NetFT adapter looks like this.

Cluster Service plumbs network routes over NIC1, NIC2 on NetFT

Cluster Service establishes TCP connection over NetFT adapter using the private NetFT IP address (source port 3343)
NetFT wraps the TCP connection inside of a UDP packet (source port 3343)
NetFT sends this UDP packet over one of the cluster-enabled physical NIC adapters to the destination node targeted for destination node’s NetFT adapter
Destination node’s NetFT adapter receives the UDP packet and then sends the TCP connection to the destination node’s Cluster Service

Heartbeats are always traversing all Cluster enabled adapters and networks. However, Cluster communication will only go through one network at a time. The network it will use is determined by the role of the network and the priority (metric).

There are three roles a Cluster has for networks.

Disabled for Cluster Communications – Role 0 - This is a network that Cluster will not use for anything.

Enabled for Cluster Communication only – Role 1 – Internal Cluster Communication and Cluster Shared Volume traffic (more later) are using this type network as a priority.

Enabled for client and cluster communication – Role 3 – This network is used for all client access and Cluster communications. Items like talking to a domain controller, DNS, DHCP (if enabled) when Network Names and IP Addresses come online. Cluster communication and Cluster Shared Volume traffic could use this network if all Role 1 networks are down.

Based on the roles, the NetFT adapter will create metrics for priority. The metric Failover Cluster uses is not the same as the network card metrics that TCP/IP assigns. Networks are given a “cost” (Metric) to define priority. A lower metric value means a higher priority while a higher metric value means a lower priority.

These metrics are automatically configured based on Cluster network role setting.

Cluster Network Role of 1 = 40,000 starting value

Cluster Network Role of 3 = 80,000 starting value

Things such as Link speed, RDMA, and RSS capabilities will reduce metric value. For example, let’s say I have two networks in my Cluster with one being selected and Cluster communications only and one for both Cluster/Client. I can run the following to see the metrics.

PS > Get-ClusterNetwork | ft Name, Metric

Name Metric

---- ------

Cluster Network 1 70240

Cluster Network 2 30240

The NetFT adapter is also capable of taking advantage of SMB Multichannel and load balance across the networks. For NetFt to take advantage of it, the metrics need to be < 16 metric values apart. In the example above, SMB Multichannel would not be used. But if there were additional cards in the machines and it looked like this:

PS > Get-ClusterNetwork | ft Name, Metric

Name Metric

---- ------

Cluster Network 1 70240

Cluster Network 2 30240

Cluster Network 3 30241

Cluster Network 4 30245

Cluster Network 5 30265

In a configuration such as this, SMB Multichannel would be used over Cluster Networks 2, 3 and 4. From a Cluster communication and heartbeat standpoint, multichannel really isn’t a big deal. However, when a Cluster is using Cluster Shared Volumes or is a Storage Spaces Direct Cluster, storage traffic is going to need higher bandwidth. SMB Multichannel would fit nicely here so an additional network card or higher speed network cards are certainly a consideration.

In the beginning of the blog, I mentioned latency and packet loss. If heartbeats cannot get through in a timely fashion, node removals can happen. Heartbeats can be tuned in the case of higher latency networks. The following are default settings for tuning the Cluster networks.

Parameter	Windows 2012 R2	Windows 2016	Windows 2019
SameSubnetDelay	1 second	1 second	1 second
SameSubnetThreshold	5 heartbeats	10 heartbeats	20 heartbeats
CrossSubnetDelay	1 second	1 second	1 second
CrossSubnetThreshold	5 heartbeats	10 heartbeats	20 heartbeats
CrossSiteDelay	N/A	1 second	1 second
CrossSiteThreshold	N/A	20 heartbeats	20 heartbeats

For more information on these settings, please refer to the Tuning Failover Cluster Network Thresholds blog.

Planning networks for Failover Clustering is dependent on how it will be used. Let’s take a look at some of the common network traffics a Cluster would have.

If this were a Hyper-V Cluster running virtual machines and Cluster Shared Volumes, Live Migration is going to occur. Clients are also connecting to the virtual machines.

Cluster Communications and heart beating will always be on the wire. If you are using Cluster Shared Volumes (CSV), there will be some redirection traffic.

If this were Cluster that used ISCSI for its storage, you would have that as a network.

If this was stretched (nodes in multiple sites), you may have the need for an additional network as the considerations for replication (such as Storage Replica) traffic.

If this is a Storage Spaces Direct Cluster, additional traffic for the Storage Bus Layer (SBL) traffic needs to be considered.

As you can see, there is a lot of various network traffic requirements depending on the type of Cluster and the roles running. Obviously, you cannot have a dedicated network or network card for each as that just isn’t always possible.

We do have a blog that will help with the Live Migration traffic to get some of the traffic isolated or limited in the bandwidth it uses. The blog Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure goes over some tips to set up.

The last thing I wanted to talk about is with stretch/multisite Failover Clusters. I have already mentioned the Cluster specific networking considerations, but now I want to talk about how the virtual machines react in this type environment.

Let’s say we have two datacenters and a four-node Failover Cluster with 2 nodes in each datacenter. As with most datacenters, they are in their own subnet and would be similar to this:

The first thing you want to consider is if you want security between the cluster nodes on the wire. As a default, all Cluster communication is signed. That may be fine for some, but for others, they wish to have that extra level of security. We can set the Cluster to encrypt all traffic between the nodes. It is simply a PowerShell command to change it. Once you change it, the Cluster as a whole needs to be restarted.

PS > (Get-Cluster).SecurityLevel = 2

0 = Clear Text

1 = Signed (default)

2 = Encrypt (slight performance decrease)

Here is a virtual machine (VM1) that has an IP Address on the 1.0.0.0/8 network and clients are connecting to it. If the virtual machine moves over to Site2 that is a different network (172.0.0.0/16), there will not be any connectivity as it stands.

To get around this, there are basically a couple options.

To prevent the virtual machine from moving from a Cluster-initiated move (i.e. drain, node shutdown, etc), consider using sites. When you create sites, Cluster now has site awareness. This means that any Cluster-initiated move will always keep resources in the same site. Setting a preferred site will also keep it in the same site. If the virtual machine was to ever move to the second site, it would be due to a user-initiated move (i.e. Move-ClusterGroup, etc) or a site failure.

But you still have the IP Address of the virtual machine issue to deal with. During a migration of the virtual machine, one of the very last things is to register the name and IP Address with DNS. If you are using a static IP Address for the virtual machine, a script would need to be manually run to change the IP Address to the local site it is on. If you are using DHCP, with DHCP servers in each site, the virtual machine will obtain a new address for the local site and register it. You then have to deal with DNS replication and TTL records a client may have. Instead of waiting for the timeout periods, a forced replication and TTL clearing on the client side would allow them to connect again.

If you do not wish to go that route, a virtual LAN (VLAN) could be set up across the routers/switches to be a single IP Address scheme. Doing this will not have the need to change the IP Address of the virtual machine as it will always remain the same. However, stretching a VLAN (not a recommendation by Microsoft) is not always easy to do and the Networking Group within your company may not want to do this for various reasons.

Another consideration is implementing a network device on the network that has a third IP Address that clients connect to and it holds that actual IP Address of the virtual machine so it will route clients appropriately. For example:

In the above example, we have a network device that has the IP Address of the virtual machine as 30.0.30.1. It will register this with all DNS and will keep the same IP Address no matter which site it is on. Your Networking Group would need to involved with this and need to control it. The chances of them not doing it is something to also consider if it can even done within your network.

We talked about virtual machines, but what about other resources, say, a file server? Unlike virtual machine roles, roles such as a file server have a Network Name and IP Address resource in the Cluster. In Windows 2008 Failover Cluster, we added he concept of “or” dependencies. Meaning, we can depend on this "or" that.

In the case of the scenario above, your Network Name could be dependent on 1.0.0.50 “or” 172.0.0.50. As long as one of the IP Address resources is online, the name is online and what is published in DNS. To go a step further for the stretch scenario, we have two parameters that can be used.

RegisterAllProvidersIP: (default = 0 for FALSE)

Determines if all IP Addresses for a Network Name will be registered by DNS
TRUE (1): IP Addresses can be online or offline and will still be registered
Ensure application is set to try all IP Addresses, so clients can connect quicker
Not supported by all applications, check with application vendor
Supported by SQL Server starting with SQL Server 2012

HostRecordTTL: (default = 1200 seconds)

Controls time the DNS record lives on client for a cluster network name
Shorter TTL: DNS records for clients updated sooner
Disclaimer: This does not speed up DNS replication

By manipulating these parameters, you will have quicker connection times by a client. For example, I want to enable to register all the IP Addresses with DNS but I want the TTL to be 5 minutes. I would run the commands:

PS > Get-ClusterResource FSNetworkName | Set-ClusterParameter RegisterAllProvidersIP 1

PS > Get-ClusterResource FSNetworkName | Set-ClusterParameter HostRecordTTL 300

When setting the parameters, recycling (offline/online) of the resources is needed.

There is more I could go into here with this subject but need to signoff for now. I hope that this gives you some basics to consider when designing your Clusters while thinking of the networking aspects of it. Networking designs and considerations must be carefully thought out.

Happy Clustering !!

John Marlin

Senior Program Manager

High Availability and Storage

Follow me on Twitter: @johnmarlin_msft

Disaster Recovery in the next version of Azure Stack HCI

John Marlin — Thu, 30 Jul 2020 20:22:30 GMT

Disaster can hit at any time. When thinking about disaster and recovery, I think of 3 things

Be prepared
Plan on not involving humans
Automatic, not automated

Having a good strategy is a must. You want to be able to have resources automatically move out of one datacenter to the other and not have to rely on someone to "flip the switch" to get things to move.

As announced at Microsoft Ignite 2019, we are now going to be able to stretch Azure Stack HCI systems between multiple sites in the next version for disaster recovery purposes. This blog will give you some insight into how you can set it up along with videos showing how it works.

To set this up, the configuration I am using is basic and common to how many networks are configured.

As you can see from the above, I have:

Two sites (Seattle and Redmond)
Two nodes in each site
A domain controller in each site
Different IP Address schemes at each site
Each site goes through a router to connect to the other site

When putting this scenario together, we considered multiple things. One of the main considerations is ease of configurations. In the past, when setting up a Failover Cluster in a stretched environment could have some inadvertent misconfigurations. We wanted to ensure, where we could, that misconfigurations can be averted. We will be using Storage Replica as our replication method between the sites. Everything you need from a software perspective for this scenario will be in-box.

One of the first things we looked at was the sites themselves. What we are doing is detecting if nodes are in different sites when Failover Clustering is first created. We do this with two different methods.

Sites are configured in Active Directory
Nodes being added have different IP Address schemes

If we see sites are configured in Active Directory, we will create a site fault domain with the name of the site and add nodes to this fault domain. If sites are not configured but the nodes are in different IP Address schemes, we will create a site fault domain with the IP Address scheme as the name with the nodes. So taking the above configuration:

Sites configured in Active Directory would be named SEATTLE and REDMOND
No sites configured in Active Directory, but different IP Address schemes would be named 0.0.0/8 and 172.0.0.0/16

In this first video, you can see that Active Directory Sites and Services do have sites configured and the creation of the site fault domains are created for you.

For more information on how to set up sites within Active Directory, please refer to the following blog:

Step-by-Step: Setting Up Active Directory Sites, Subnets, and Site-Links

https://blogs.technet.microsoft.com/canitpro/2015/03/03/step-by-step-setting-up-active-directory-sites-subnets-site-links/

The next item we took to ease configuration burdens are when Storage Spaces Direct is enabled. In Windows Server 2016 and 2019 Storage Spaces Direct, we supported one storage pool. In the stretch scenario with the next version, we are now supporting a pool per site. When Storage Spaces Direct is now enabled, we are going to automatically create these pools and name them with the site they are created in. I.E.:

Pool for site SEATTLE and Pool for site REDMOND
Pool for site 1.0.0.0/8 and Pool for site 172.0.0.0/16

The other item we are detecting for is the presence of the Storage Replica service. We will go out to each node to detect if Storage Replica has been installed on each of the nodes specified. If it is missing, we will stop and let you know it is missing and let you know which node.

As an FYI, keep in mind that this is a pre-released product at the time of this blog creation. The stopping of the Storage Spaces Direct enablement is subject to change to more of a warning at a later date.

Once everything is in place and all services are present, you can see from this video, the successful enablement of Storage Spaces Direct and the separate pools.

To point out here, in order to get this to work, we did a lot of work with the Health Service to allow this. The Health Service keeps track of the health of the entire cluster (resources, nodes, drives, pools, etc). With multiple pools and multiple sites, it needs to keep track of all this. I will go more into this a bit later and show you how it works in a few other scenarios with this configuration.

Now that we have pools, we need to create virtual disks at the sites. Since this is a stretch and we want replication with Storage Replica, two disks need to be created at each site (one for data and one for logs). I use New-Volume to create them, but there is a caveat. When the virtual disk is created, Failover Clustering will auto add them into the Available Storage Group. Therefore, you need to ensure Available Storage is on the node you are creating the disks on.

NOTE: We are aware of this caveat and will be working on getting a better story for this as we go along.

In the examples below, I will use:

site names of SEATTLE and REDMOND
pool names of Pool for site SEATTLE and Pool for site REDMOND
NODE1 and NODE2 are in site SEATTLE
NODE3 and NODE4 are in site REDMOND

So, let's create the disks. The first thing is to ensure the Available Storage group is on NODE1 in SITEA so we can create those disks.

From Failover Cluster Manager:

Go under Storage to Disks
In the far-right pane, click Move Available Storage and Select Node
Select NODE1 and OK

From PowerShell:

Move-ClusterGroup -Name "Available Storage" -Node NODE1

Now we can create the two disks for site SEATTLE. I will be creating a 2-way mirror From PowerShell you can run the commands:

New-Volume -FriendlyName DATA_SEATTLE -StoragePoolFriendlyName "Pool for Site SEATTLE" -FileSystem ReFS -NumberOfDataCopies 2 -ProvisioningType Fixed -ResiliencySettingName Mirror -Size 125GB

New-Volume -FriendlyName LOG_SEATTLE -StoragePoolFriendlyName "Pool for Site SEATTLE" -FileSystem ReFS -NumberOfDataCopies 2 -ProvisioningType Fixed -ResiliencySettingName Mirror -Size 10GB

Now, we must create the same drives on the other pool. Before doing so, take the current disks offline and move the Available Storage group is moved to NODE3 in site REDMOND.

Move-ClusterGroup -Name "Available Storage" -Node NODE1

Once there, the commands would be:

New-Volume -FriendlyName DATA_REDMOND -StoragePoolFriendlyName "Pool for Site REDMOND" -FileSystem ReFS -NumberOfDataCopies 2 -ProvisioningType Fixed -ResiliencySettingName Mirror -Size 125GB

New-Volume -FriendlyName LOG_REDMOND -StoragePoolFriendlyName "Pool for Site REDMOND" -FileSystem ReFS -NumberOfDataCopies 2 -ProvisioningType Fixed -ResiliencySettingName Mirror -Size 10GB

We now have all the disks we want in the cluster. You can then move the Available Storage back to NODE1 and copy your data on it if you have is already. Next thing is to set them up with Storage Replica. I will not go through the steps here. But here is the link to the document on setting it up in the same cluster as well as another video here showing it being set up.

https://docs.microsoft.com/en-us/windows-server/storage/storage-replica/stretch-cluster-replication-using-shared-storage

Everything is all set and resources can be created. If you have not seen what it looks like, here is a video of a site failure. Everything will move over and run. Storage Replica takes care of ensuring the right disks are utilized and halting replication until the site comes back.

TIP: If you watch the tail end of the video, you will notice that the virtual machine automatically live migrates with the drives. We added this functionality back in Windows 2016 where the VM will chase the CSV so they are not on different sites. Again, nother extra you have to set up, just something we did to ease administration burdens for you.

I talked earlier about all the work we did with the health service. We also did a lot of work with the way we autopool drives together by site. In the next couple of videos, I want to help show you.

This first video is more of the brownfield scenario. Meaning, I have an existing cluster running and want to add nodes from a separate site. In the video, you will see that we add the nodes from the different site into the cluster and create a pool from the drives in that site.

In this video, I will show how we react to disk failures. I have removed a disk from each pool to simulate a failure. I then replace the disks with new ones. We detect which drive is in which site and pool them into the proper pool.

I hope this gives you a glimpse into what we are doing with the next version of Azure Stack HCI. This scenario should be in the public preview build when it becomes available. Get a preview of what is coming and help us test and stabilize things. You also can suggest how it could be better.

Thanks

John Marlin

Senior Program Manager

High Availability and Storage Team

Twitter: @johnmarlin_msft

Talking Failover Clustering and Azure Stack HCI Ignite 2019 Announcements

John Marlin — Mon, 18 Nov 2019 20:03:53 GMT

Microsoft Ignite 2019 at the Orange County Convention Center in Orlando, Florida was a huge success with approximately 26,000 in attendance. We had a great time meeting and talking with our partners and customers. We also had a few announcements regarding features coming in the next Windows Server LTSC regarding Azure Stack HCI and Failover Clustering that I wanted to highlight.

In this blog, I will try to separate out each of the features announced. In some of the sessions, there were multiple announcements. Since this is the case, I want to make sure you are aware of each announcement. Due to this, a particular session may be listed in multiple sections. I will also try to point out what things apply only to Windows Server vNext and what would also include using on Windows Server 2016/2019.

Ensure you go through this entire blog as there are numerous announcements including one of the biggest asks from our customers at the end.

I will first start out with this one. It doesn't apply necessarily to Failover Clustering or Azure Stack HCI, but it is an important one as the date is right around the corner.

Windows 2008/2008R2 End of Life

Windows Server 2008/2008R2 are both reaching end of life, these two sessions talk about options of planning for this day.

Plan for Z-Day 2020: Windows Server 2008 end of support is coming

https://myignite.techcommunity.microsoft.com/sessions/82850

It's 2019 and your servers are 2008

https://myignite.techcommunity.microsoft.com/sessions/89294

Quick list

For a quick list of announcements being made, you should catch this session listing 45 things in 45 minutes. This will give you a brief overview of things so that you can delve a little deeper into the other session announcements below. So if there is only one session you review from this list, this is the one. These will cover items for both Windows Server 2016/2019 as well as Windows Server vNext.

What's new for Azure Stack HCI: 45 things in 45 minutes

https://myignite.techcommunity.microsoft.com/sessions/82905

Azure Stack is now one portfolio

We have expanded Azure Stack into a PORTFOLIO of products, including Azure Stack Edge, Azure Stack HCI, and Azure Stack Hub.

Discover Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/82907

Get started with Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/89352

Azure Iaas VM Guest Cluster support for Premium File Shares and Shared Azure Disk

This is a preview of file shares that act like shared disks for your Azure IaaS Failover Clusters. This is not Storage Spaces Direct, this is traditional Failover Clusters for SQL Server. If you do not wish to run Storage Spaces Direct, but would rather run a traditional Failover Cluster, Shared Azure Disk is now in limited preview for your shared storage. This one is also for all versions currently released as well as Windows Server vNext.

Windows Server on Azure Overview: Lift-and-Shift migrations for Enterprise Workloads

https://myignite.techcommunity.microsoft.com/sessions/81956

Windows Admin Center setup wizard for Storage Spaces Direct and Software Defined Networking

That's right. With the latest release of Windows Admin Center, you now have a walk through of creating an Azure Stack HCI system as well as SDN. PowerShell not needed !! Note: what is in black is there now while items greyed out are coming. This one is not limited to only vNext, but also works with currently released Windows Server 2016/2019.

Jumpstart your Azure Stack HCI Deployment

https://myignite.techcommunity.microsoft.com/sessions/82906

Get started with Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/89352

Using Azure Services to manage and monitor on-premises clusters

See how various Azure Services can be used from the cloud to your on-premises clusters all using Windows Admin Center. The "hybrid" way of doing things now. When talking about using these services, you can use them against your current Windows 2016/2019 HCI and traditional clusters as well as Windows Server vNext.

Windows Server: What's new and what's next

https://myignite.techcommunity.microsoft.com/sessions/81704

Modernize your retail stores or branch offices with Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/82904

Clustering in the age of HCI and Hybrid

https://myignite.techcommunity.microsoft.com/sessions/83946

Shut down safeguard

Say you are needing to reboot your Azure Stack HCI systems. You first reboot your first node and bring it back up. Once it comes back up, it needs to go through a re-sync of the data. However, if you were to then reboot the second node before it finishes, bad things "could" happen to your data. We will now have a built-in safeguard to prevent this from happening. If here are storage jobs currently running, we will warn you not to. Sorry, but this is where we get into the "it's only available in Windows Server vNext" areas of the list.

Modernize your retail stores or branch offices with Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/82904

Increase to 16 petabytes raw storage

In Windows Server 2019, we announced 4 petabyte raw storage availability. In the next Windows Server LTSC, we have increased it even further to 16 petabytes raw storage. This is a big win for backup server scenarios (and others).

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

Switchless Clusters

Here again, we announced 2-node switchless clusters for Windows Server 2019 Azure Stack HCI and Failover Clustering. For Windows Server vNext, we are now going to support more than 2 nodes with full mesh connectivity. So how many network adapters can you add?

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

Get started with Azure Stack HCI

https://myignite.techcommunity.microsoft.com/sessions/89352

Repair/Resyc Times are much faster and can throttle it.

When a reboot is necessary, resync times when it comes back can take a while. We have changed the way we are doing these repairs/resyncs and made it much faster. We have also introduced a way to throttle the repair/rysync. So you can now control if you need it to complete even faster or run more in the background. This is something that will only be available in Windows Server vNext Azure Stack HCI.

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

New Affinity/AntiAffinity rules

In the past, we have only had antiaffinity as an option to keep roles apart. Introducing new rules for affinity and antiaffinity was a must. With nodes in different sites, we also had to make it site aware. So now, you can keep roles together or apart as well as storage affinity to keep virtual machines on the same node as the storage. These new rules will only be available in Windows Server vNext Azure Stack HCI and traditional failover clusters.

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

Bitlocker

This one had no slides as snuck it in there with some time available. Bitlocker has been available for Clusters for quite some time. The requirement was the cluster nodes must be all in the same domain as the bitlocker key is tied to the Cluster Name Object (CNO). However, for those clusters at the edge, workgroup clusters, and multidomain clusters, Active Directory may not be present. With no Active Directory, there is no CNO. These cluster scenarios had no data at-rest security. With Windows Server vNext, we introduced our own bitlocker key stored locally (encrypted of course) for cluster to use. Now you can feel safer as at-rest security is now in place. This feature will only be available in Windows Server vNext Azure Stack HCI and traditional failover clusters.

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

New Network Validation tests

We come up with new things all the time and networking continues to be an innovation. We are not standing pat and continue to add new tests. These new tests will only be available in Windows Server vNext Azure Stack HCI and traditional failover clusters.

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

Windows Admin Center

I mentioned in previous descriptions here about Windows Admin Center. The Windows Admin Center Team works tirelessly to continue to add more and more in. For the general availability (GA) announcement of the latest Windows Admin Center, you can see just how much has been added and new. Everything from continued on-premises work, hybrid, third party extensions, and more. The work being done here is not just for the future, but all existing Azure Stack HCI and traditional failover clustering systems.

Windows Server deep dive: Demopalooza

https://myignite.techcommunity.microsoft.com/sessions/81949

Live Q&A: Manage your hybrid server environment with Windows Admin Center

https://myignite.techcommunity.microsoft.com/sessions/89341

Windows Admin Center: Unlock Azure Hybrid value

https://myignite.techcommunity.microsoft.com/sessions/81952

Automatically monitor, secure and update your on-premises servers from Azure with Windows Admin Center

https://myignite.techcommunity.microsoft.com/sessions/83942

Be a Windows Admin Center expert: Best practices for deployment, configuration, and security

https://myignite.techcommunity.microsoft.com/sessions/83943

Get more done with Windows Admin Center third-party extensions

https://myignite.techcommunity.microsoft.com/sessions/83944

Windows Admin Center: Better together with System Center and Microsoft Azure

https://myignite.techcommunity.microsoft.com/sessions/83945

New Windows Performance Monitor

Everyone is concerned with performance. We have in-box Performance Monitor, but it hasn't been updated in a long time (other than some counters with each release). In Windows Admin Center, we have also added a new Performance Monitor that can be run anywhere against any of the different versions of Windows Server. You can have multiple windows running, pause it, and much much more. We have also made it easier to look at. As with all things Windows Admin Center, this will work with current versions of Windows Server as well as future versions. It is also not limited to Azure Stack HCI or traditional failover clusters.

Windows Server: What's new and what's next

https://myignite.techcommunity.microsoft.com/sessions/81704

Badges

Badges? What are those? I don't need no stinkin' badges. Well let me tell you. Our partners have developed solutions optimized for running specific apps/workloads on Azure Stack HCI (both current and future). We are working with our partners to add these "badges" that will be in full view and searchable so you can ensure that the solution you are purchasing is the right solution for you.

Windows Server: What's new and what's next

https://myignite.techcommunity.microsoft.com/sessions/81704

Stretch Azure Stack HCI for Disaster Recovery

I mentioned was saving our biggest ask for last. In Windows Server vNext, we will now offer Stretch Azure Stack HCI for disaster recovery purposes. You will now be able to have automatic failovers in case of disaster. We are also making it easier as there is less for you to configure for this, we will do a lot of this for you. This is also one of those, "sorry, but this will be in Windows Server vNext".

Stretching Azure Stack HCI for disaster recovery: A glimpse into the future

https://myignite.techcommunity.microsoft.com/sessions/83962

What's next for software defined storage and networking for Windows Server

https://myignite.techcommunity.microsoft.com/sessions/81960

HCI is the name, futures are the game

https://myignite.techcommunity.microsoft.com/sessions/89330

As I have mentioned previously, we are not resting on our laurels. We are working on many many other improvements for Windows Server vNext, we just aren't ready to announce them just yet.

Thank you,

John Marlin

Senior Program Manager

Windows High Availability and Storage

Windows Server 2019 Failover Clustering New Features

John Marlin — Thu, 05 Dec 2019 00:59:48 GMT

Greeting Failover Cluster fans!! John Marlin here and I own the Failover Clustering feature within the Microsoft product team. In this blog, I will be giving an overview and demo a lot of the new features in Windows Server 2019 Failover Clustering. I have held off on this to let some things settle down with some of the announcements regarding Azure Stack HCI, upcoming Windows Server Summit, etc.

I have broken these all down into 7 videos so that you can view them in smaller chunks rather than one massive long video. With each video, I am including a quick description of what features that will be covered. Each of the videos are approximately 15 minutes long.

Part 1

In this video, we will take a brief look back at Windows Server 2016 Failover Clustering and preview what we did in regards to Windows Server 2019 to make things better.

Part 2

In Part 2 of the series, we take a look at Windows Admin Center and how it can make the user experience better, Cluster Performance History to get a history on how the cluster/nodes are performing, System Insights using predictive analytics (AI) and machine learning (ML), and Persistent Memory which is the latest in storage/memory technology.

Part 3

In Part 3 of the series, we take a look at Cluster Sets as our new scaling technology, actual in-place Windows Server Upgrades which have not been supported in the past, and Microsoft Distributed Transaction Coordinator (MSDTC).

Part 4

In this video, we take a look at Two-Node Hyperconverged and the new way of configuring resiliency, File Share Witness capabilities for achieving quorum at the edge, Split Brain detection and how we try to lessen chances of nodes running independent of each other, and what we did with Security in mind.

Part 5

This video talks about Scale-Out File Servers and some of the connectivity enhancements, Cluster Shared Volumes (CSV) with caching and a security enhancement, Marginal Disk support and the way we are detecting drives that are starting to go bad, and Cluster Aware Updating enhancements for when you are patching your Cluster nodes.

Part 6

This video will talk about enhancements we made with the Cluster Network Name, changes made for when running Failover Clusters in Azure as IaaS virtual machines, and how Domain Migrations are no longer a pain point moving between domains.

Part 7

As a wrap up, we will take a look at a couple announcements made and demonstrated at Microsoft Ignite 2018 regarding IOPs and Capacity.

My hope is that you enjoy these videos and get a good understanding our roadmap from Windows Server 2016 to Windows Server 2019 from a Failover Clustering standpoint. If there are any questions about any of these features, hit me up on Twitter (below).

Thanks

John Marlin

Senior Program Manager

High Availability and Storage

Twitter: @JohnMarlin_MSFT

List of Failover Cluster Events in Windows 2016/2019

John Marlin — Thu, 26 Sep 2019 23:51:30 GMT

Let me tell you of a story of myself and one of my asks while I was still in support.

We always thought the it would be nice to have a listing of all Failover Clustering events for references. Our customers ask for it, we ask for it. So logically, we approached the Product Group and asked for it. The response back wasn't what we really wanted to hear which was, "It's in the code that you can pull out but will take you some time to piece it all together". Again, not what we wanted to hear and had this ask on multiple occasions.

Then, little Johnny joined the big boys in the Product Group and became the PM owner of Failover Clustering infrastructure. Now, everyone keeps asking me for it. My response?

"It's in the code that you can pull out but will take you some time to piece it all together"

Awful hypocritical of myself, but when I did take a look at it, yes, it would take a while to do (not talking about a day or so). I could see why we had the response previously.

Here's a bit of trivia I bet you did not know.

Q: How many events are there for Failover Clustering?

A: Between the System Event and the Microsoft-Windows-FailoverClustering/Operational channel, there are a total of 388 events in Windows 2019

While in Support, I did not realize how many people were asking for it. I was getting hit up from all over the place both internally and externally. I was starting to look a lot like:

Finally, I decided that I was going to do it. So rolled up my sleeves and a couple weeks later, it was finally complete. FINALLY!!!! The spreadsheet with the events is attached to this blog. My hope is that you can make good use of it and is what you have been asking for.

To explain it a bit, this list is for Windows 2016 and 2019 Failover Clustering. Many of these same events are in previous versions. We have not removed any events, only added with each version. I have separated it into two tabs, one for Windows 2016 and the other for the Windows 2019 new events.

There are a few duplicate event IDs. That is by design. The description of the event is going to depend on the call made, so they may differ slightly.
I sorted it all by the severity levels. Feel free to sort however you wish.
You may notice '%1', '%2', etc values in the description. When there is an event, we collect the values such as resource, group, etc as a variable and substitute the variable in the actual description.
You may notice '%n' in some of the descriptions. I left those in the spreadsheet and are carriage returns. I didn't want to replace them as it would distort things if you wanted to sort differently.

Is there a moral of the story for little Johnny? Don't know. But now that it is done, he feels much better.

Happy Clustering !!!

John Marlin

Senior Program Manager

High Availability and Storage

Optimizing Hyper-V Live Migrations on an Hyperconverged Infrastructure

John Marlin — Thu, 05 Sep 2019 15:12:35 GMT

We have been doing some extensive testing in how to best configure Hyper-V live migration to achieve the best performance and highest level of availability.

Recommendations:

Configure Live Migration to use SMB. This must be performed on all nodes

    Set-VMHost –VirtualMachineMigrationPerformanceOption SMB

Use RDMA enabled NIC’s to offload the CPU and improve network performance

Configure SMB Bandwidth Limits to ensure live migrations do not saturate the network and throttle to 750 MB. This must be performed on all nodes

First install the SMB Bandwidth Limits feature:

     Add-WindowsFeature -Name FS-SMBBW

Throttle to 750 MB

     Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 750MB

Configure a maximum of 2 simultaneous Live migrations (which is default). This must be performed on all nodes

Leave at the default value of 2, no changes required

     Set-VMHost -MaximumVirtualMachineMigrations 2

Background:

For those that want to understand the ‘why’ on the above recommendations, read on!

What a live migration fundamentally does is take the memory allocated to a virtual machine and copies it over the network from one server to another. Let’s say you allocated 4 GB of memory to a virtual machine, when you invoke a live migration that 4 GB of memory is copied over the network between the source and destination server. Because the VM is running, that means that memory is changing while that 4 GB is copied. Those changes are tracked, and once the initial allocated memory is copied, then a second pass occurs and the changed memory is copied. In the second pass the amount of memory changed will be smaller and takes less time; all the while yet memory is changing while that happens. So a third pass happens, and so on with each pass getting faster and the delta of memory changed getting smaller. Eventually the set of memory gets small enough and the VM is paused and the final set of changes is copied over, then the VM is resumed on the new server. While the VM is paused and the final memory copy occurs the VM is not available, this is referred to as the blackout window. This is not unique to Hyper-V, all virtualization platforms have this. The magic of a live migration is that as long as the blackout window is within the TCP reconnect window, it is completely seamless to the applications. That’s how a live migration achieves zero downtime from an application perspective, even though there is a very small amount of downtime from an infrastructure perspective. Don’t get hung up on the blackout window (if it is within the TCP reconnect window), it’s all about the app!

Live migration supports TCPIP, Compression, and SMB to perform live migration. On nearly all hyperconverged infrastructure (HCI) systems have RDMA enabled network cards, and Server Message Block (SMB) has a feature called SMB Direct which can take advantage of RDMA. By using SMB as the protocol for the memory copy over the network, it will result in drastically reduced CPU overhead to conduct the data copy with the best network performance. This is important to minimize consuming CPU cycles from other running virtual machines and keeping the data copy windows small so that the number of passes to copy changed memory is minimized. Another feature of SMB is SMB Multi-channel, which will stream the live migration across multiple network interfaces to achieve even better performance.

An HCI system is a distributed system that is heavily dependent on reliable networking, as there is cluster communication and data replication also occurring over the network. From a network perspective, a live migration is a sudden burst of heavy network traffic. Using SMB bandwidth limits to achieve Network Quality of Service (QoS) is desired to keep this burst traffic from saturating the network and negatively impacting other aspects on the system. The testing conducted tested different bandwidth limits on a dual 10 Gpbs RDMA enabled NIC and measured failures under stress conditions and found that throttling live migration to 750 MB achieved the highest level of availability to the system. On a system with higher bandwidth, you may be able to throttle to a value higher than 750 MB.

When draining a node in an HCI system, multiple VMs can be live migrated at the same time. This parallelization can achieve faster overall times when moving large numbers of VMs off a host. As an example, instead of copying just 4 GB for a single machine, it will copy 4 GB for one VM and 4 GB for another VM. But there is a sweet spot, a single live migration at a time serializes and results in longer overall times and having too may simultaneous live migrations can end up taking much longer. Remember that if the network becomes saturated with many large copies, that each one takes longer… which means more memory is changing on each, which means more passes, and results in overall longer times for each. Two simultaneous live migrations were found to deliver the best balance in combination with a 750 MB throttle.

Lastly a live migration will not infinitely continue to make pass after pass copying changed memory on a very busy VM with a slow interconnect that is taking a long time, eventually live migration will give up and freeze the VM and make a final memory copy. This can result in longer blackout windows, and if that final copy exceeds the TCP reconnect window then it can impact the apps. This is why ensuring live migrations can complete in a timely manner is important.

In our internal testing we have found that these recommended settings will achieve the fastest times to drain multiple VMs off of server, will achieve the smallest blackout windows for application availability, and the least impact to production VMs, and will achieve greatest levels of availability to the infrastructure.

Elden Christensen

Principal Program Manager

High Availability and Storage Team

So what exactly is the CLIUSR account

John Marlin — Tue, 19 Nov 2019 22:44:50 GMT

From time to time, people stumble across the local user account called CLIUSR and wonder what it is, while you really don’t need to worry about it; we will cover it for the curious in this blog.

The CLIUSR account is a local user account created by the Failover Clustering feature when it is installed on Windows Server 2012 or later. Well, that’s easy enough, but why is this account here? Taking a step back, let’s take a look at why we are using this account.

In the Windows Server 2003 and previous versions of the Cluster Service, a domain user account was used to start the Cluster Service. This Cluster Service Account (CSA) was used for forming the Cluster, joining a node, registry replication, etc. Basically, any kind of authentication that was done between nodes used this user account as a common identity.

A number of support issues were encountered as domain administrators were pushing down group policies that stripped rights away from domain user accounts, not taking into consideration that some of those user accounts were used to run services. An example of this is the Logon as a Service right. If the Cluster Service account did not have this right, it was not going to be able to start the Cluster Service. If you were using the same account for multiple clusters, then you could incur production downtime across a number of critical systems. You also had to deal with password changes in Active Directory. If you changed the user accounts password in AD, you also needed to change passwords across all Clusters/nodes that use the account.

In Windows Server 2008, we learned and redesigned everything about the way we use start the service to make it more resilient, less error prone, and easier to manage. We started using the built-in Network Service to start the Cluster Service. Keep in mind that this is not the full blown account, just simply a reduced privileged set. Changing it to this reduced account was a solution for the group policy issues.

For authentication purposes, it was switched over to use the computer object associated with the Cluster Name known as the Cluster Name Object (CNO)for a common identity. Because this CNO is a machine account in the domain, it will automatically rotate the password as defined by the domain’s policy for you (which is every 30 days by default).

Great!! No more domain user account and its password changes we have to account for. No more trying to remember which Cluster was using which account. Yes!! Ah, not so fast my friend. While this solved some major pain, it did have some side effects.

Starting in Windows Server 2008 R2, admins started virtualizing everything in their datacenters, including domain controllers. Cluster Shared Volumes (CSV) was also introduced and became the standard for private cloud storage. Some admin’s completely embraced virtualization and virtualized every server in their datacenter, including to add domain controllers as a virtual machine to a Cluster and utilize the CSV drive to hold the VHD/VHDX of the VM.

This created a “chicken or the egg” scenario that many companies ended up in. In order to mount the CSV drive to get to the VMs, you had to contact a domain controller to get the CNO. However, you couldn’t start the domain controller because it was running on the CSV.

Having slow or unreliable connectivity to domain controllers also had effect on I/O to CSV drives. CSV does intra-cluster communication via SMB much like connecting to file shares. To connect with SMB, it needs to authenticate and in Windows Server 2008 R2, that involved authenticating the CNO with a remote domain controller.

For Windows Server 2012, we had to think about how we could take the best of both worlds and get around some of the issues we were seeing. We are still using the reduced Network Service privilege to start the Cluster Service, but now to remove all external dependencies we have a local (non-domain) user account for authentication between the nodes.

This local “user” account is not an administrative account or domain account. This account is automatically created for you on each of the nodes when you create a cluster or on a new node being added to the existing Cluster. This account is completely self-managed by the Cluster Service and handles automatically rotating the password for the account and synchronizing all the nodes for you. The CLIUSR password is rotated at the same frequency as the CNO, as defined by your domain policy (which is every 30 days by default). With it being a local account, it can authenticate and mount CSV so the virtualized domain controllers can start successfully. You can now virtualize all your domain controllers without fear. So we are increasing the resiliency and availability of the Cluster by reducing external dependencies.

This account is the CLIUSR account and is identified by its description.

One question that we get asked is if the CLIUSR account can be deleted. From a security standpoint, additional local accounts (not default) may get flagged during audits. If the network administrator isn’t sure what this account is for (i.e. they don’t read the description of “Failover Cluster Local Identity”), they may delete it without understanding the ramifications. For Failover Clustering to function properly, this account is necessary for authentication.

Joining node starts the Cluster Service and passes the CLIUSR credentials across.
All passes, so the node is allowed to join.

There is one extra safe guard we did to ensure continued success. If you accidentally delete the CLIUSR account, it will be recreated automatically when a node tries to join the Cluster.

Short story… the CLIUSR account is an internal component of the Cluster Service. It is completely self-managing and there is nothing you need to worry about regarding configuring and managing it. So leave it alone and let it do its job.

In Windows Server 2016, we will be taking this even a step further by leveraging certificates to allow Clusters to operate without any external dependencies of any kind. This allows you to create Clusters out of servers that reside in different domains or no domains at all. But that’s a blog for another day.

Also, please be aware that there are security guides/blogs out there that can block local non-administrative accounts from doing certain things. For example, this blog explains:

Blocking Remote Use of Local Accounts
https://blogs.technet.microsoft.com/secguide/2014/09/02/blocking-remote-use-of-local-accounts

In this blog, it talks about "Deny access to this computer from the network" with the default generic security identifier of S-1-5-113 (NT AUTHORITY\Local account). This will cause the Cluster Service not be able to do what it needs to do (joins, management, etc). CLIUSR is a local account without administrative rights, so it really cannot do anything to the system in a disruptive manner. It goes on to explain what you should put in which is the generic identifier S-1-5-114 (NT AUTHORITY\Local account and member of Administrators group). This way, Cluster can still perform as Cluster should.

As an FYI, this is not the only group policy. Others could include, but not limited to:

Access this computer over the network
Deny log on locally
Deny log on locally as a service

Hopefully, this answers any questions you have regarding the CLIUSR account and its use.

Happy Clustering !!!

John Marlin

Senior Program Manager

High Availability and Storage Team

Twitter: @Johnmarlin@MSFT

No such thing as a Heartbeat Network

John Marlin — Mon, 25 Mar 2019 22:49:09 GMT

This is a blog that has been a long time coming. From time to time, we get a request about how to configure networking in Failover Clusters. One of the questions we get is how should the heartbeat network be configured and that is what the focus is on this blog. I am here to say, there is no such thing, and never was, a heartbeat network.

Please allow me to give a little background and explain.

In Windows 2003 and below Failover Clustering, you could define which network was used for Cluster Communication. Below is a picture for reference.

In the picture above, we would want to select Private for our Cluster Communication to as to not use the Public which has all WAN traffic. All Cluster Communication between nodes (joins, registry updates/changes, etc) would go only over this network if it is up. As the picture shows, the networks are called Public and Private. As years went by, some started calling the Private network a Heartbeat network.

Heartbeats are small packets (134 bytes) that travel over UDP Port 3343 on all networks configured for Cluster use between all nodes. They serve multiple purposes.

Establishes if a Cluster Network is up or down
Establishes routes between nodes
Ensure the health of a node is good or bad

So let's say I have Private set as my priority network for Cluster Communications. If it is up, we are sending our communication through it. But what happens if that network wasn't reliable. If a node tries to join and packets are dropping, then the join could fail. If this is the case, you either determine where the problem is and fix it, or go back into the Cluster properties and set the Public as priority.

Starting in Windows 2008 Failover Clusters, the concept of Public and Private networks went out the window. We will now send Cluster Communication over any of our networks. One of the reasons for this was reliability. With that change, we also gave the heartbeats an additional purpose.

Determine the fastest and reliable routes between nodes

Since we are now determining the fastest and reliable routes, we could use different networks between nodes for our communication. Take the below as an example.

We have three individual networks between our nodes:

Blue – 10 gbps used for backups and administration only
Green – 40 gbps used for communicating out on the WAN to clients
Red – 40 gbps used for communicating out on the WAN to clients

As a refresher, here is what the heartbeats are doing:

Establishes if a Cluster Network is up or down
Establishes routes between nodes
Ensure the health of a node is good or bad
Determine the fastest and reliable routes between nodes

What the heartbeats are going to tell the Cluster is to use one of the faster networks for its communication. With that as the case, it is going to use either Red or Green network. If the heartbeats start detecting that neither of these is as reliable (i.e. dropping a packet, network congested, etc), it will automatically switch and use the Blue network. That's it, nothing for you to configure extra.

So to wrap things up, remember these things about Failover Clusters and Heartbeats.

There is no such thing as a heartbeat network or a network dedicated to heartbeats
Heartbeat packets are lightweight (134 bytes in size)
Heartbeats are sensitive to latency
Bandwidth is not an important factor, quality of service is. If your network is all teamed, ensure you have set up Network QOS policies for our UDP 3343 traffic.

For more information regarding configuring networks in a Cluster, please see Microsoft Ignite session:

Failover Clustering Networking Essentials

Happy Clustering !!!!

John Marlin

Senior Program Manager

Microsoft Corporation

Twitter: @Johnmarlin_MSFT

Windows Server 2016/2019 Cluster Resource / Resource Types

John Marlin — Tue, 03 Dec 2019 21:16:08 GMT

First published on MSDN on Jan 16, 2019
Over the years, we have been asked about what some of the Failover Cluster resources/resource types are and what they do. There are several resources that have been asked about on multiple occasions and we haven't really had a good definition to point you to. Well, not anymore.

What I want to do with this blog is define what they are, what they do, and when they were added (or removed). I am only going to cover the in-box resource types that come with Failover Clustering. But first, I wanted to explain what a cluster "resource" and "resource types" are.

Cluster resources are physical or logical entities, such as a file share, disk, or IP Address managed by the Cluster Service. The operating system does not distinguish between cluster and local resources. Resources may provide a service to clients or be an integral part of the cluster. Examples of resources would be physical hardware devices such as disk drives, or logical items such as IP addresses, network names, applications, and services. They are the basic and smallest configurable unit managed by the Cluster Service. A resource can only run on a single node in a cluster at a time.

Cluster resource types are dynamic library plug-ins. These Resource DLLs are responsible for carrying out most operations on cluster resources. A resource DLL is characterized as follows:

It contains the resource-specific code necessary to provide high availability for instances of one or more resource types.

It exposes this code through a standard interface consisting of a set of entry point functions .

It is registered with the Cluster service to associate one or more resource type names with the name of the DLL.

It is always loaded into a Resource Monitor's process when in use.

When the Cluster service needs to perform an operation on a resource, it sends the request to the Resource Monitor assigned to the resource. If the Resource Monitor does not have a DLL in its process that can handle that type of resource, it uses the registration information to load the DLL associated with the resource type. It then passes the Cluster service's request to one of the DLL's entry point functions. The resource DLL handles the details of the operation so as to meet the specific needs of the resource.

You can define your own resource types to provide customized support for cluster-unaware applications, enhanced support for cluster-aware applications, or specialized support for new kinds of devices. For more information, see Creating Resource Types .

All resource types that are available in a Failover Cluster can be seen by right-mouse clicking on the name of the Cluster, choosing Properties, and selecting the Resource Types tab ( shown below ).

You can also get a list from running the PowerShell command Get-ClusterResourceType . Please keep in mind that all resource types may not show up or have access to. For example, if the Hyper-V role is not installed, the virtual machine resource types will not be available.

So enough about this, let's get to the resource types, when they were available and, for some, when they were last seen.

Since there are multiple versions of Windows Clustering, this blog will only focus on the two latest versions (Windows Server 2016 and 2019).

Windows Server 2016 / 2019

Cloud Witness (clusres.dll): Cloud Witness is a quorum witness that leverages Microsoft Azure as the arbitration point. It uses Azure Blob Storage to read/write a blob file which is then used as an arbitration point in case of split-brain resolution.

DFS Replicated Folder (dfsrclus.dll): Manages a Distributed File System (DFS) replicated folder. When creating a DFS, this resource type is configured to ensure proper replication occurs. For more information regarding this, please refer to the 3-part blog series on the topic.

DHCP Service (clnetres.dll): The DHCP Service resource type supports the Dynamic Host Configuration Protocol (DHCP) Service as a cluster resource. There can be only one instance of a resource of this type in the cluster (that is, a cluster can support only one DHCP Service). Dynamic Host Configuration Protocol (DHCP) is a client/server protocol that automatically provides an Internet Protocol (IP) host with its IP address and other related configuration information such as the subnet mask and default gateway. RFCs 2131 and 2132 define DHCP as an Internet Engineering Task Force (IETF) standard based on Bootstrap Protocol (BOOTP), a protocol with which DHCP shares many implementation details. DHCP allows hosts to obtain required TCP/IP configuration information from a DHCP server.

Disjoint IPv4 Address (clusres.dll): IPv4 Resource type that can be used if setting up a site to site VPN Gateway. It can only be configured by PowerShell, not by the Failover Cluster Manager, the GUI tool on Windows Server. We added two IP addresses of this resource type, one for the internal network and one for the external network.

The internal address is plumbed down for the cluster network that is identified by Routing Domain ID and VLAN number. Remember, we mapped them to the internal network adapters on the Hyper-V hosts earlier. It should be noted that this address is the default gateway address for all machines on the internal network that need to connect to Azure.

The external address is plumbed down for the cluster network that is identified by the network adapter name. Remember, we renamed the external network adapter to “Internet” on both virtual machines.

Disjoint IPv6 Address (clusres.dll): IPv6 Resource type that can be used if setting up a site to site VPN Gateway. It can only be configured by PowerShell, not by the Failover Cluster Manager, the GUI tool on Windows Server. We added two IP addresses of this resource type, one for the internal network and one for the external network.

The internal address is plumbed down for the cluster network that is identified by Routing Domain ID and VLAN number. Remember, we mapped them to the internal network adapters on the Hyper-V hosts earlier. It should be noted that this address is the default gateway address for all machines on the internal network that need to connect to Azure.

The external address is plumbed down for the cluster network that is identified by the network adapter name. Remember, we renamed the external network adapter to “Internet” on both virtual machines.

Ras Cluster Resource (rasclusterres.dll): This resource object specifies where the site-to-site VPN configuration is stored. The file share can be anywhere the two virtual machines have read / write access to. It can only be configured by PowerShell, not by the Failover Cluster Manager, the GUI tool on Windows Server. This resource type is only available after installing the VPN Roles in Windows Server.

Distributed File System (clusres2.dll): Manages a Distributed File System (DFS) as a cluster resource. When creating a DFS, this resource type is configured to ensure proper replication occurs. For more information regarding this, please refer to the 3-part blog series on the topic.

Distributed Transaction Coordinator (mtxclu.dll): The Distributed Transaction Coordinator (DTC) resource type supports the Microsoft Distributed Transaction Coordinator (MSDTC). MSDTC is a Windows service providing transaction infrastructure for distributed systems, such as SQL Server. In this case, a transaction means a general way of structuring the interactions between autonomous agents in a distributed system.

File Server (clusres2.dll): Manages the shares that are created as highly available. A file share is a location on the network where clients connect to access data, including documents, programs, images, etc.

File Share Witness (clusres.dll): A File Share Witness is a witness (quorum) resource and is simply a file share created on a completely separate server from the cluster for tie-breaker scenarios when quorum needs to be established. A File Share Witness does not store cluster configuration data like a disk. It does, however, contain information about which version of the cluster configuration database is most recent.

Generic Application (clusres2.dll): The Generic Application resource type manages cluster-unaware applications as cluster resources, as well as cluster-aware applications that are not associated with custom resource types. The Generic Application resource DLL provides only very basic application control. For example, it checks for application failure by determining whether the application's process still exists and takes the application offline by terminating the process.

Generic Script (clusres2.dll): The Generic Script resource type works in conjunction with a script that you must provide to manage an application or service as a highly available cluster resource. In effect, the Generic Script resource type allows you to script your own resource DLL. For more information on how to use the Generic Script resource type, see Using the Generic Script Resource Type .

Generic Service (clusres2.dll): The Generic Service resource type manages services as cluster resources. Similar to the Generic Application resource type, the Generic Service resource type provides only the most basic functionality. For example, the failure of a Generic Service resource is determined by a query of the Service Control Manager (SCM). If the service is running, it is presumed to be online. To provide greater functionality, you can define a custom resource type (for information, see Creating Resource Types ).

A generic service resource type is usually used to manage a stateless service as a cluster resource, which can be failed over. However, generic services don't provide much state information other than their online state, so if they have an issue that doesn't cause the resource to go offline, it is more difficult to detect a service failure.

Generic services should only be used when all of the following conditions are true; otherwise, you should create a resource DLL.

The resource is not a device. The generic resource types are not designed to manage hardware.

The resource is stateless.

The resource is not dependent on other resources.

The resource does not have unique attributes that should be managed with private properties.

The resource does not have special functionality that should be exposed through control codes.

The resource can be started and stopped easily without using special procedures.

Health Service (healthres.dll): The Health Service constantly monitors your Storage Spaces Direct cluster to detect problems and generate "faults". Through either Windows Admin Center or PowerShell, it displays any current faults, allowing you to easily verify the health of your deployment without looking at every entity or feature in turn. Faults are designed to be precise, easy to understand, and actionable.

Each fault contains five important fields:

Severity

Description of the problem

Recommended next step(s) to address the problem

Identifying information for the faulting entity

Its physical location (if applicable)

IP Address (clusres.dll): The IP Address resource type is used to manage Internet Protocol (IP) network addresses. When an IP Address resource is included in a group with a Network Name resource, the group can be accessed by network clients as a failover cluster instance (formerly known as a virtual server).

IPv6 Address (clusres.dll): The IPv6 Address resource type is used to manage Internet Protocol version 6 (IPv6) network addresses. When an IPv6 Address resource is included in a group with a Network Name resource, the group can be accessed by network clients as a failover cluster instance (formerly known as a virtual server).

IPv6 Tunnel Address (clusres2.dll): The IPv6 Tunnel Address resource type is used to manage Internet Protocol version 6 (IPv6) network tunnel addresses. When an IPv6 Tunnel Address resource is included in a group with a Network Name resource, the group can be accessed by network clients as a failover cluster instance (formerly known as a virtual server).

iSCSI Target Server (wtclusres.dll): Creates a highly available ISCSI Target server for machines to connect to for drives.

Microsoft iSNS (isnsclusres.dll): Manages an Internet Storage Name Service (iSNS) server. iSNS provides discovery services for Internet Small Computer System Interface (ISCSI) storage area networks. iSNS processes registration requests, deregistration requests, and queries from iSNS clients. We would recommend not using this resource type moving forward as it is being removed from the product.

MSMQ (mqclus.dll): Message Queuing (MSMQ) technology enables applications running at different times to communicate across heterogeneous networks and systems that may be temporarily offline. Applications send messages to queues and read messages from queues.

MSMQTriggers (mqtgclus.dll): Message Queuing triggers allow you to associate the arrival of incoming messages at a destination queue with the functionality of one or more COM components or stand-alone executable programs. These triggers can be used to define business rules that can be invoked when a message arrives at the queue without doing any additional programming. Application developers no longer must write any infrastructure code to provide this kind of message-handling functionality.

Network File System (nfsres.dll): NFS cluster resource has dependency on one Network Name resource and can also depend on one or more disk resources in a resource group. For a give network name resource there can be only one NFS resource in a resource group. The dependent disk resource hosts one or more of NFS shared paths. The shares hosted on a NFS resource are scoped to the dependent network name resources. Shares scoped to one network name are not visible to clients that mount using other network names or node names residing on the same cluster.

Network Name (clusres.dll): The Network Name resource type is used to provide an alternate computer name for an entity that exists on a network. When included in a group with an IP Address resource, a Network Name resource provides an identity to the role, allowing the role to be accessed by network clients as a Failover Cluster instance.

Distributed Network Name (clusres.dll): A Distributed Network Name is a name in the Cluster that does not use a clustered IP Address. It is a name that is published in DNS using the IP Addresses of all the nodes in the Cluster. Client connectivity to this type name is reliant on DNS round robin. In Azure, this type name can be used in leiu of having the need for an Internal Load Balancer (ILB) address. The predominant usage of a Distributed Network Name is with a Scale-Out File Server (discussed next). In Windows Server 2019, we added the ability for the Cluster Name Object (CNO) to use a DNN. For more information on the CNO usage as a Distinguished Network Name, please refer to the Windows Server 2019 Failover Clustering New Features blog.

Scale Out File Server (clusres.dll): A Scale Out File Server (SOFS) is a clustered file share that can be accessed by any of the nodes. It uses the Distributed Network Name as the client access point and does not use a clustered IP Address. The Distributed Network Name is discussed previously.

Physical Disk (clusres.dll): The Physical Disk resource type manages a disk on a shared bus connected to two or more cluster nodes. Some groups may contain one or more Physical Disk resources as dependencies for other resources in the group. On a Storage Spaces Direct cluster, the disks are local to each of the nodes.

Hyper-V Network Virtualization Provider Address (provideraddressresource.dll): The IP address assigned by the hosting provider or the datacenter administrators based on their physical network infrastructure. The PA appears in the packets on the network that are exchanged with the server running Hyper-V that is hosting network virtualized virtual machine(s). The PA is visible on the physical network, but not to the virtual machines.

Storage Pool (clusres.dll): Manages a storage pool resource. It allows for the creation and deletion of storage spaces virtual disks.

Storage QoS Policy Manager (clusres.dll): A resource type for the Policy Manger that collects the performance of storage resources allocated to the individual highly available virtual machines. It monitors the activity to help ensure storage is used fairly within I/O performance established through any policies that may be configured.

Storage Replica (wvrres.dll): Storage Replica is Windows Server technology that enables replication of volumes between servers or clusters for disaster recovery. This resource type enables you to create stretch failover clusters that span two sites, with all nodes staying in sync. A Stretch Cluster allows configuration of computers and storage in a single cluster, where some nodes share one set of asymmetric storage and some nodes share another, then synchronously or asynchronously replicate with site awareness. By stretching clusters, workloads can be run in multiple datacenters for quicker data access by local proximity users and applications, as well as better load distribution and use of compute resources.

Task Scheduler (clusres.dll): Task Scheduler is a resource that is tied to tasks you wish to run against the Cluster. Clustered tasks are not created or shown in Failover Cluster Manager. To create or view a Clustered Scheduled Task, you would need to use PowerShell.

Set-ClusteredScheduledTask

Get-ClusteredScheduledTask

Unregister-ClusteredScheduledTask

Virtual Machine (vmclusres.dll): The Virtual Machine resource type is used to control the state of a virtual machine (VM). The following table shows the mapping between the state of the VM (indicated by the EnabledState property of the Msvm_ComputerSystem instance representing the VM) and the state of the Virtual Machine resource (indicated by the State property of the MSCluster_Resource class or the return of GetClusterResourceState function).

VM State	Virtual Machine resource state
Disabled	3
Offline	3
Suspended	32769
Starting	32770
Online Pending	129
Online	2
Stopping	32774
Offline Pending	130
Saving	32773
Enabled	2
Paused	32768
Pausing	32776
Resuming	32777

Virtual Machine Cluster WMI (vmclusres.dll): The Virtual Machine Cluster WMI resource type is one used when virtual machine grouping (also known as virtual machine sets) has been configured. By grouping virtual machines together, managing the "group" is much easier than all of the virtual machines individually. VM Groups enable checkpoints, backup and replication of VMs that form a guest-cluster and that use a Shared VHDX.

Virtual Machine Configuration (vmclusres.dll): The Virtual Machine Configuration resource type is used to control the state of a virtual machine configuration.

Virtual Machine Replication Broker (vmclusres.dll): Replication broker is a prerequisite if you are replicating clusters using Hyper-V replica. It acts the point of contact for any replication requests, and can query into the associated cluster database to decide which node is the correct one to redirect VM specific events such as Live Migration requests etc. The broker also handles authentication requests on behalf of the VMs. A new node can be added or removed from a cluster at any point, without the need to reconfigure the replication as the communication between the primary and recovery clusters is directed to the respective brokers.

Virtual Machine Replication Coordinator (vmclusres.dll): Coordinator comes into picture when we use the concept of “collection” in Hyper-V replica . This was introduced in Windows Server 2016 and is a prerequisite if you are using few of the latest features for eg: shared virtual hard disks. When VMs are replicated as part of a collection, the replication broker coordinates actions/ events that affect VM group – for eg:, to take a point in time snapshot which is app consistent, applying the replication settings, modifying the interval for replication etc. and propagating the change across all the VMs in the collection.

WINS Service (clnetres.dll): The WINS Service resource type supports the Windows Internet Name Service (WINS) as a cluster resource. There can be only one instance of a resource of this type in the cluster; in other words, a cluster can support only one WINS Service. Windows Internet Name Service (WINS) is a legacy computer name registration and resolution service that maps computer NetBIOS names to IP addresses.

Windows Server 2016 only

Cross Cluster Dependency Orchestrator (clusres.dll): This is a resource type that you can ignore and does not do anything. This was to be a new feature to be introduced. However, it never came to fruition, but the resource type was not removed. It is removed in Windows Server 2019.

Windows Server 2019 only

SDDC Management (sddcres.dll): SDDC Management is installed when the cluster is enabled for Storage Spaces Direct. It is the management API that Windows Admin Center uses to connect/manage your Storage Spaces Direct. It is an in-box resource type with Windows Server 2019 and is a download and manual addition to Windows Server 2016. For information regarding this, please refer to the Manage Hyper-Converged Infrastructure with Windows Admin Center document.

Scaleout Worker (scaleout.dll): This is used for Cluster Sets. In a Cluster Set deployment, the CS-Master interacts with a new cluster resource on the member Clusters called “Cluster Set Worker” (CS-Worker). CS-Worker acts as the only liaison on the cluster to orchestrate the local cluster interactions as requested by the CS-Master. Examples of such interactions include VM placement and cluster-local resource inventorying. There is only one CS-Worker instance for each of the member clusters in a Cluster Set.

Scaleout Master (scaleout.dll): This is used when In a Cluster Set, the communication between the member clusters is loosely coupled, and is coordinated by a new cluster resource called “Cluster Set Master” (CS-Master). Like any other cluster resource, CS-Master is highly available and resilient to individual member cluster failures and/or the management cluster node failures. Through a new Cluster Set WMI provider, CS-Master provides the management endpoint for all Cluster Set manageability interactions.

Infrastructure File Server (clusres.dll): In hyper-converged configurations, an Infrastructure SOFS allows an SMB client (Hyper-V host) to communicate with guaranteed Continuous Availability (CA) to the Infrastructure SOFS SMB server. This hyper-converged SMB loopback CA is achieved via VMs accessing their virtual disk (VHDx) files where the owning VM identity is forwarded between the client and server. This identity forwarding allows ACL-ing VHDx files just as in standard hyper-converged cluster configurations as before. There can be at most only one Infrastructure SOFS cluster role on a Failover Cluster. Each CSV volume created in the failover automatically triggers the creation of an SMB Share with an auto-generated name based on the CSV volume name. An administrator cannot directly create or modify SMB shares under an Infra SOFS role, other than via CSV volume create/modify operations. This role is commonly used with Cluster Sets.

Thanks
John Marlin
Senior Program Manager
High Availability and Storage

Follow me on Twitter @JohnMarlin_MSFT

Microsoft Ignite 2018 Clustering Sessions available

John Marlin — Fri, 15 Mar 2019 22:17:08 GMT

First published on MSDN on Nov 01, 2018
For those who attended Microsoft Ignite 2018 in Orlando, Florida, we thank you for making it another huge success.

So much fun was had by all. We had the privilege of showing you what is new and coming in Windows Server 2019 with 700+ deep dive sessions and over 100+ workshops.

You got the latest insights and skills from technology leaders and practitioners shaping the future of cloud, data, business intelligence, teamwork, and productivity. As well as immersed yourselves with the latest tools, tech, and experiences that matter, and heard the latest updates and ideas directly from the experts. There were demos galore throughout all the sessions.

Who can forget the demo showing these unheard-of numbers before now running Windows Server 2019 Storage Spaces Direct and Intel's Optane DC Persistent Memory:

Or, the storage limit increase to 4 petabytes. We are not just saying it because it’s a big number, we showed it with the help from our friends at Quanta Cloud Technology , Seagate , and Samsung .

In case you missed Ignite, attended but missed a session, or you wish to view the sessions again, here is the link to all the sessions available for your viewing pleasure both from the Migrosoft Ignite pages as well as YouTube.

To kick it all off, here is Satya Nadella's keynotes to kick off Microsoft Ignite 2018.

Vision Keynote
Ignite , YouTube
Satya Nadella - Chief Executive Officer of Microsoft

Since this is the Failover Clustering blog, I wanted to call out these sessions specifically to what we are doing in the hyper-converged infrastructure (HCI) space.

BRK2035 - Windows Server 2019: What’s new and what's next
Ignite , YouTube
Erin Chapple, Vijay Kumar
Windows Server is a key component in Microsoft's hybrid and on-premises strategy and in this session, hear what's new in Windows Server 2019. Join us as we discuss the product roadmap, Semi-Annual Channel, and demo some exciting new features.

BRK2241 - Windows Server 2019 deep dive
Ignite , YouTube
Jeff Woolsey
Hybrid at its core. Secure by design. With cloud application innovation and hyper-converged infrastructure built into the platform, backed by the world’s most trusted cloud, Azure, Microsoft presents Windows Server 2019. In this session Jeff Woolsey - Principal Program Manager - dives into the details of what makes Windows Server 2019 an exciting platform for IT pros and developers looking into modernizing their infrastructure and applications.

BRK2232 - Jumpstart your hyper-converged infrastructure deployment with Windows Server
Ignite , YouTube
Elden Christensen, Steven Ekren
The time is now to adopt hyper-converged infrastructure and Storage Spaces Direct. Where to start? This session covers design considerations and best practices, how to choose and procure the best hardware, sizing and planning, deployment, and how to validate your cluster is ready for showtime. Get tips and tricks directly from the experts! Applies to Windows Server 2016 and Windows Server 2019.

BRK2036 - From Hyper-V to hyper-converged infrastructure with Windows Admin Center
Ignite , YouTube
Cosmos Darwin, Daniel Lee
Discover how Windows Admin Center (Formerly Project "Honolulu") makes it easier than ever to manage and monitor Hyper-V. It’s quick to deploy, there’s no additional license, and it’s built from years of feedback – this is YOUR new dashboard! Ready to go hyper-converged? New features like Storage Spaces Direct and Software-Defined Networking (SDN) are built right in, so you get an integrated, seamless experience ready for the future of the software-defined datacenter.

BRK2231 - Be an IT hero with Storage Spaces Direct in Windows Server 2019
Ignite , YouTube
Cosmos Darwin, Adi Agashe
The virtualization wave of datacenter modernization, consolidation, and savings made you an IT hero. Now, the next big wave is here: Hyper-Converged Infrastructure, powered by software-defined storage! Storage Spaces Direct is purpose-built software-defined storage for Hyper-V. Save money, accelerate IO performance, and simplify your infrastructure, from the datacenter to the edge. This packed technical session covers everything that’s new for Storage Spaces Direct in Windows Server 2019.

BRK2233 - Get ready for Windows Server 2008 and 2008 R2 end of support
Ignite , YouTube
Ned Pyle, Jeff Woolsey, Sue Hartford
Windows Server 2008 and 2008 R2 were great operating systems at the time, but times have changed. Cyberattacks are commonplace, and you don’t want to get caught running unsupported software. End of support for Windows Server 2008 and 2008 R2 means no more security updates starting on January 14, 2020. Join us for a demo-intensive session to learn about your options for upgrading to the latest OS. Or consider migrating 2008 to Microsoft Azure where you can get three more years of extended security updates at no additional charge.

We even had a few of our Microsoft MVP's jump in and deliver some theater sessions.

THR3127 - Cluster Sets in Windows Server 2019: What is it and why should I use it?
Ignite , YouTube
Carsten Rachfahl, Microsoft MVP
Would you like to have an Azure-like availability set and fault domain across multiple clusters in your private cloud? Do you need to have more than 16 nodes in an hyper-converged infrastructure cluster or want multiple 4-node HCI clusters to behave like one? Then you definitely want to attend this session and learn about Cluster Sets - a new, amazing feature in Windows Server 2019 to solve these problems.

THR2233 - What is the Windows Server Software Defined (WSSD) program and why does it matter?
Ignite , YouTube
Carsten Rachfahl, Microsoft MVP
The Window Server Software Defined (WSSD) program allows vendors to build and offer a tested end-to-end hyper-converged infrastructure solution. After implementing more than 100 Storage Spaces Direct projects, Carsten think this is more important than ever. Why? In this session, learn the reasons, and get help choosing the right solution for you!

THR3137 - The case of the shrinking data: Data Deduplication in Windows Server 2019
Ignite , YouTube
Dave Kawula, Microsoft MVP
One of the most requested features for Storage Spaces Direct was ReFS with Data Deduplication. This feature was released over a year ago, but it was only in the Semi-Annual Release which did not include support for Storage Spaces Direct. The IT community has waited patiently, and the time has finally come with Windows Server 2019. This release has added full support for ReFS Data Deduplication into Storage Spaces Direct. What does this mean for you? How about more than 80% space savings on your VMs, Backups, ISO repositories, all running on Cluster Shared Volumes with Storage Spaces Direct. In this session, learn how to set up, configure, and test Data Deduplication with ReFS based on his years of knowledge working with Microsoft storage.

These are just the tip of the iceberg with the amount of sessions available to you. We hope you enjoy these sessions and had a great time at Ignite 2018 as we sure did.

I leave you now with two other huge announcements.

First, Ignite will be back in Orlando, Florida for Microsoft Ignite 2019 . The dates are set for November 4-8, 2019 at the Orange County Convention Center . You can pre-register today!!

Second, Ignite 2018 is hitting the road and going global with " Microsoft Ignite | The Tour ". Join us at the place where developers and tech professionals continue learning alongside experts. Explore the latest developer tools and cloud technologies and learn how to put your skills to work in new areas. Connect with our community to gain practical insights and best practices on the future of cloud development, data, IT, and business intelligence. Join us for two days of community-building and hands-on learning.

We will be heading to places such as:

Toronto, Canada

Sydney, Australia

Berlin, Germany

Amsterdam, The Netherlands

And these are just a few of the places we are going. Head to the "Microsoft Ignite | The Tour" page and find the city near you. Oh, and did I mention it is free!!!

Thanks
John Marlin
Senior Program Manager
High Availability and Storage

Follow me on Twitter @JohnMarlin_MSFT

Cluster Sets in Windows Server 2019 - Hyperscale for Hyperconverged !!

John Marlin — Fri, 12 Apr 2019 17:54:20 GMT

Cluster Sets is a new feature in Windows Server 2019 that was first introduced at Ignite 2017 . Cluster Sets is the new cloud scale-out technology that increases cluster node count in a single Software Defined Data Center (SDDC) cloud by orders of magnitude. A Cluster Set is a loosely-coupled federated grouping of multiple Failover Clusters: compute, storage or hyper-converged. The Cluster Sets technology enables virtual machine fluidity across member clusters within a cluster set and a unified storage namespace across the set in support of virtual machine fluidity.

Cluster Sets gives you the benefit of hyperscale while continuing to maintain great resiliency. So in more clearer words, you are pseudo clustering clusters together while not putting all your eggs in one basket. You can now have multiple baskets to maintain greater flexibility without sacrificing resiliency.

While preserving existing Failover Cluster management experiences on member clusters, a Cluster Set instance additionally offers key use cases, such as lifecycle management. The Windows Server Preview Scenario Evaluation Guide for Cluster Sets provides you the necessary background information along with step-by-step instructions to evaluate cluster sets technology using PowerShell.

Here is a video providing a brief overview of what Cluster Sets is and can do.

The evaluation guide to read more about Cluster Sets along with information on how to set it up is listed on the Microsoft Docs page where this, and numerous other Microsoft products are covered. The quick link to the Cluster Sets page is https://aka.ms/Cluster_Sets .

Finally, there is a GitHub lab scenario where you can set this up on your own and try it out that gives you additional instructions.

We hope that you try it out and provide feedback. Feedback can be done in two ways:

The Feedback Hub on Windows 10
Email Cluster Sets Feedback . This alias has been set up to provide feedback only.

Thanks,
John Marlin
Senior Program Manager
High Availability and Storage

Follow me on Twitter @JohnMarlin_MSFT

Scale-Out File Server Improvements in Windows Server 2019

John Marlin — Tue, 10 Sep 2019 23:39:08 GMT

SMB Connections move on connect

Scale-Out File Server (SOFS) relies on DNS round robin for inbound connections sent to cluster nodes. When using Storage Spaces on Windows Server 2016 and older, this behavior can be inefficient: if the connection is routed to a cluster node that is not the owner of the Cluster Shared Volume (aka the coordinator node), all data redirects over the network to another node before returning to the client. The SMB Witness service detects this lack of direct I/O and moves the connection to a coordinator. This can lead to delays.

In Windows Server 2019, we are much more efficient. The SMB Server service determines if direct I/O on the volume is possible. If direct I/O is possible, it passes the connection on. If it is redirected I/O, it will move the connection to the coordinator before I/O starts. Synchronous client redirection required changes in the SMB client, so only Windows Server 2019 and Windows 10 Fall 2017 clients can use this new functionality when talking to a Windows 2019 Failover Cluster. SMB clients from older OS versions will continue relying upon the SMB Witness to move to a more optimal server.

As a note here, I wanted to point out when a move would and would not occur in a stretch scenario and it will depend on the storage you are using. So for my example, my Scale-Out File Server is running on NodeA in SiteA. All node's IP Addresses are registered in DNS and it is round robin on where a client connects.

If you have a stretch Failover Cluster and the storage presents itself as symmetric; meaning, all nodes have access to the drives, the client connection will be moved to SiteA as described above.

But let's say the SAN storage and is asymmetric; meaning, each site has it's own SAN storage and there is replication between them. This is the process that will occur.

1. A client connection is sent to a node in SiteB

2. The node in SiteB will retain that connection.

3. All data requests will be redirected over the CSV network to SiteA.

4. Data is retrieved and sent back over the CSV network to the node in SiteB.

5. The node in SiteB then sends the data to the client.

6. Rinse, repeat for all other data requests.

Infrastructure Scale-Out File Server

There is a new Scale-Out File Server role in Windows Server 2019 called Infrastructure File Server. When you create an Infrastructure File Server, it will create a single namespace share automatically for the CSV drive (i.e. \\InfraSOFSName\Volume1, etc.). In hyper-converged configurations, an Infrastructure SOFS allows an SMB client (Hyper-V host) to communicate with guaranteed Continuous Availability (CA) to the Infrastructure SOFS SMB server. There can be at most only one infrastructure SOFS cluster role on a Failover Cluster.

To create the Infrastructure SOFS, you would need to use PowerShell. For example:
Add-ClusterScaleOutFileServerRole -Cluster MyCluster -Infrastructure -Name InfraSOFSName

SMB Loopback

There is an enhancement made with Server Message Block (SMB) to work properly with SMB local loopback to itself which was previously not supported. This hyper-converged SMB loopback CA is achieved via Virtual Machines accessing their virtual disk (VHDx) files where the owning VM identity is forwarded between the client and server.

This is a role that Cluster Sets takes advantage of where the path to the VHD/VHDX is placed as \\InfraSOFSName\Volume1. This \\InfraSOFSName\Volume1 path can then be utilized by the virtual machine if it is local or remote.

Identity Tunneling

In Server 2016, if Hyper-V virtual machines are hosted on a SOFS share, you must grant the machine accounts of the Hyper-V compute nodes permission to access the VHD/VHDX files. If the virtual machines and VHD/VHDX is running on the same cluster, then the user must have rights. This can make management difficult as two sets of permissions are needed.

In Windows Server 2019 when using SOFS, we now have “identity tunneling” on Infrastructure shares. When you access Infrastructure Share from the same cluster or Cluster Set, the application token is serialized and tunneled to the server, and VM disk access is done using that token. This works even if your identity is Local System, a service, or virtual machine account.

Thanks,
John Marlin
Senior Program Manager
High Availability and Storage

Follow me on Twitter @JohnMarlin_MSFT