Possible MAC address assignment strategies for tenant VMs running on Stack-HCI environment

Azure Stack HCI is a hyperconverged infrastructure (HCI) cluster solution consists of windows servers (Hyper-V), Storage Spaces Direct, and Azure-inspired SDN. All clustered servers share common configurations and resources by leveraging the Windows Server Failover Clustering feature. A Windows Failover Cluster consists of multiple windows servers running in a cluster to provide high availability i.e. If one server node goes down, then another node takes over. We can create multiple VMs on the failover cluster. VMs can be connected to different tenant networks. In this case we need to make sure VM connected on network 'A' with static ip should persist same network configuration even if it gets migrated from one node to another within a cluster. This is possible by assigning static MAC/static IP for the VM. But how to get free and unique MAC to assign to VM, there are different ways to solve this problem. In this article we will discuss some of them with pros and cons. (Note: solutions discussed in this article are just to suggestions, can not be considered optimal solutions)

Before this, lets understand things required to create a tenant VM. To create tenant VM connected to tenant network in stack-hci following steps are performed. 

1. VM network adapters created with static MAC addresses for the lifetime of the VM. (static MAC is required because If the MAC address changes during the VM lifetime, Network Controller cannot configure the necessary policy for the network adapter. Not configuring the policy for the network prevents the network adapter from processing network traffic, and all communication with the network fails )

2. Create a new network interface on network controller service(running on SDN) having same MAC and static IP assigned.

3. Associate created network adapter with network interface controller, to make vm gets tenant network connection.

 

Please refer the official document for more details why these steps are required. 

 

Following are powershell commands to perform above steps logically.

 

 

 

 

 

 

# 1. vm creation with static MAC
New-VM -Generation 2 -Name "MyVM" -Path "C:\VMs\MyVM" -MemoryStartupBytes 4GB -VHDPath "C:\VMs\MyVM\Virtual Hard Disks\WindowsServer2016.vhdx" -SwitchName "SDNvSwitch"

Set-VM -Name "MyVM" -ProcessorCount 4

# Setting static MAC
Set-VMNetworkAdapter -VMName "MyVM" -StaticMacAddress "00-11-22-33-44-55"

# Creating network interface on networkcontroller with same mac/static IP
$vnet = Get-NetworkControllerVirtualNetwork -ConnectionUri $uri -ResourceId "Contoso_WebTier"
$vmnicproperties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceProperties
$vmnicproperties.PrivateMacAddress = "001122334455"
$vmnicproperties.PrivateMacAllocationMethod = "Static"
$vmnicproperties.IsPrimary = $true

$vmnicproperties.DnsSettings = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceDnsSettings
$vmnicproperties.DnsSettings.DnsServers = @("24.30.1.11", "24.30.1.12")

$ipconfiguration = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfiguration
$ipconfiguration.resourceid = "MyVM_IP1"
$ipconfiguration.properties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfigurationProperties
$ipconfiguration.properties.PrivateIPAddress = "24.30.1.101"
$ipconfiguration.properties.PrivateIPAllocationMethod = "Static"

$ipconfiguration.properties.Subnet = New-Object Microsoft.Windows.NetworkController.Subnet
$ipconfiguration.properties.subnet.ResourceRef = $vnet.Properties.Subnets[0].ResourceRef

$vmnicproperties.IpConfigurations = @($ipconfiguration)
New-NetworkControllerNetworkInterface –ResourceID "MyVM_Ethernet1" –Properties $vmnicproperties –ConnectionUri $uri

$nic = Get-NetworkControllerNetworkInterface -ConnectionUri $uri -ResourceId "MyVM_Ethernet1"

#Do not change the hardcoded IDs in this section, because they are fixed values and must not change.
# 3. Finally make association of vm network adapter with nic created in last step
$FeatureId = "9940cd46-8b06-43bb-b9d5-93d50381fd56"

$vmNics = Get-VMNetworkAdapter -VMName "MyVM"

$CurrentFeature = Get-VMSwitchExtensionPortFeature -FeatureId $FeatureId -VMNetworkAdapter $vmNics

if ($CurrentFeature -eq $null) {
    $Feature = Get-VMSystemSwitchExtensionPortFeature -FeatureId $FeatureId

    $Feature.SettingData.ProfileId = "{$($nic.InstanceId)}"
    $Feature.SettingData.NetCfgInstanceId = "{56785678-a0e5-4a26-bc9b-c0cba27311a3}"
    $Feature.SettingData.CdnLabelString = "TestCdn"
    $Feature.SettingData.CdnLabelId = 1111
    $Feature.SettingData.ProfileName = "Testprofile"
    $Feature.SettingData.VendorId = "{1FA41B39-B444-4E43-B35A-E1F7985FD548}"
    $Feature.SettingData.VendorName = "NetworkController"
    $Feature.SettingData.ProfileData = 1

    Add-VMSwitchExtensionPortFeature -VMSwitchExtensionFeature  $Feature -VMNetworkAdapter $vmNics
} else {
    $CurrentFeature.SettingData.ProfileId = "{$($nic.InstanceId)}"
    $CurrentFeature.SettingData.ProfileData = 1

    Set-VMSwitchExtensionPortFeature -VMSwitchExtensionFeature $CurrentFeature  -VMNetworkAdapter $vmNics
}
Get-VM -Name "MyVM" | Start-VM

 

 

 

 

 

 

   

To make sure VMs works fine even after migration in windows failover cluster, we need to assign static MAC and static IP. To get available and unique MAC is challenge here. We will see following ways to solve this problem with pros and cons.

 

1. Generating random MAC on node
2. Assign dedicated unique MAC address pool range for every Node in cluster and Generating free MAC from Node's MAC address range pool.
3. Get free MAC from Network controller service and use same on VM

Lets discuss each approach in detailed-

 

1. Generating random MAC on node

This is very simple approach, where we will generate random MAC and same will be used. It includes below steps:

• Generate valid MAC on node, where we need to create VM(or new network interface).
• Set same MAC on vm network adapter as static
• Set same MAC on network interface along with static IP

Following is powershell script performs same above logical steps:

 

 

 

 

 

 

New-VM -Name $vm_name -MemoryStartupBytes $vm_memory -BootDevice VHD -VHDPath $image_path -Path $vm_data_path -Generation $vm_generation -SwitchName $switch_name
    
Add-ClusterVirtualMachineRole -vmname $vm_name -Name $vm_name
Start-Sleep -Seconds 3

    ############## Random MAC address generation and assignment
    # script block for mac address assingment kept in retry block
    $mac_address = Retry-Command -ScriptBlock {
        # do something
        # found used in az-auto-setup
        #$mac_address=("{0:D12}" -f ( Get-Random -Minimum 0 -Maximum 99999 ))
        $mac_address = (0..5 | ForEach-Object { '{0:x}{1:x}' -f (Get-Random -Minimum 0 -Maximum 15), (Get-Random -Minimum 0 -Maximum 15) }) -join '-'
        write-host "Tring to set : $mac_address"
        Set-VMNetworkAdapter -VMName $vm_name -StaticMacAddress "$mac_address"
        write-host "Mac set succssfully: $mac_address"
        Start-Sleep -Milliseconds 500
        Write-host $( Get-VMNetworkAdapter -VMName $vm_name ).MacAddress
        return $mac_address
    }

    write-host "Mac set succssfully: $mac_address"

    Write-host 'VM created..'

    
    ##########################################################################

    # network configuration starts here..
    # you can refer for more info: https://learn.microsoft.com/en-us/windows-server/networking/sdn/manage/create-a-tenant-vm#prerequisites
    $vnet = Get-NetworkControllerVirtualNetwork -ConnectionUri $uri -ResourceId $vnet_name



    $vmnicproperties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceProperties
    # give same mac address below as created before..
    $mac = -join($mac_address.split("-")).toupper()
    $vmnicproperties.PrivateMacAddress = $mac
    Write-host $mac
    $vmnicproperties.PrivateMacAllocationMethod = "Static"
    $vmnicproperties.IsPrimary = $true

    $vmnicproperties.DnsSettings = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceDnsSettings
    #$vmnicproperties.DnsSettings.DnsServers = @("192.168.1.254", "8.8.8.8")
    $vmnicproperties.DnsSettings.DnsServers = $dns_server
    $ipconfiguration = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfiguration
    $ipconfiguration.resourceid = $vm_name + "_IP1"
    $ipconfiguration.properties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfigurationProperties
    $ipconfiguration.properties.PrivateIPAddress = $ip_address
    $ipconfiguration.properties.PrivateIPAllocationMethod = "Static"

    $ipconfiguration.properties.Subnet = New-Object Microsoft.Windows.NetworkController.Subnet

    # do: programatically decide subnet full ref, or form path directly
    # $ipconfiguration.properties.subnet.ResourceRef = $vnet.Properties.Subnets[0].ResourceRef
    $ipconfiguration.properties.subnet.ResourceRef = "/virtualNetworks/" + $vnet_name + "/subnets/" + $subnet_name
    $vmnicproperties.IpConfigurations = @($ipconfiguration)
    $NIC_name = $vm_name + "_Eth1"
    New-NetworkControllerNetworkInterface -ResourceID $NIC_name -Properties $vmnicproperties -ConnectionUri $uri -Confirm:$false -force
    Write-host 'NIC config created..'
    Start-Sleep -Seconds 8

    $nic = Get-NetworkControllerNetworkInterface -ConnectionUri $uri -ResourceId $NIC_name

    #########################################################################

    #Do not change the hardcoded IDs in this section, because they are fixed values and must not change.

    $FeatureId = "9940cd46-8b06-43bb-b9d5-93d50381fd56"

    $vmNics = Get-VMNetworkAdapter -VMName $vm_name

    $CurrentFeature = Get-VMSwitchExtensionPortFeature -FeatureId $FeatureId -VMNetworkAdapter $vmNics

    if ($CurrentFeature -eq $null)
    {
        $Feature = Get-VMSystemSwitchExtensionPortFeature -FeatureId $FeatureId

        $Feature.SettingData.ProfileId = "{$( $nic.InstanceId )}"
        $Feature.SettingData.NetCfgInstanceId = "{56785678-a0e5-4a26-bc9b-c0cba27311a3}"
        $Feature.SettingData.CdnLabelString = "TestCdn"
        $Feature.SettingData.CdnLabelId = 1111
        $Feature.SettingData.ProfileName = "Testprofile"
        $Feature.SettingData.VendorId = "{1FA41B39-B444-4E43-B35A-E1F7985FD548}"
        $Feature.SettingData.VendorName = "NetworkController"
        $Feature.SettingData.ProfileData = 1

        Add-VMSwitchExtensionPortFeature -VMSwitchExtensionFeature  $Feature -VMNetworkAdapter $vmNics
    }
    else
    {
        $CurrentFeature.SettingData.ProfileId = "{$( $nic.InstanceId )}"
        $CurrentFeature.SettingData.ProfileData = 1

        Set-VMSwitchExtensionPortFeature -VMSwitchExtensionFeature $CurrentFeature  -VMNetworkAdapter $vmNics
    }
    Write-host 'finally applying setting..'
    Start-Sleep -Seconds 5



    Get-VM -Name $vm_name | Start-VM

 

 

 

 

 

 

But this approach has following problems :

• There is possibility that two vms running on same cluster may got same MAC and it may create MAC conflict issue after VM migration.
• Some specific MAC range is not allowed to be assigned, in this case we need to keep retrying random generation, which is not good.
• Very difficult to keep track of used MAC addresses.

So this method is not found reliable.

 

2. Assign dedicated unique MAC address pool range for every Node in cluster and Generating free MAC from Node's MAC address range pool.

We can preassign unique dedicated MAC address pool range to every node(hyper-v) in cluster, so whenever we need to create new network interface we will first get free MAC from the pool of node where vm will be running. Same MAC will be made static on vm network adapter and network interface followed by static ip. Following are logical steps need to perform:

1. Get free MAC from Node's MAC pool range.

2. Assign same static MAC on VM network adapter 

3. Set same MAC on network interface along with static IP

 

 

 

 

 

Following is an example of MAC pool range distribution plan for 255 clusters, 16 hosts each:

00-15-5D-[c1][c2]-[h1][v1]-[v2][v3]

field c1, c2: will indicate a cluster. 00 to FF : Max 255 clusters can be created. 

field h1 - will be for host/node identification in a cluster.    0 to F = max 16 hosts in a cluster.

field v1,v2,v3 will be used for vms. 000 to FFF = 16*16*16 = total 4096 possible vms/network interfaces per node can be created.

 

 

 

 

 

 

But main challenge here is there is no api or command available to get free MAC from node pool. There is a workaround that to get MAC assigned from the pool, we need to create dummy network adapter with dynamic configuration and start vm for a moment to get free dynamic MAC assigned from node pool range. Then we need to stop vm and revert the change and using same MAC we need to create static one and same steps need to be followed. This is not good approach since this is just a workaround, where dummy resource need to create.

 

 

 

# creating new network adapter with dynamic MAC
Add-VMNetworkAdapter -VMName $vm_name -SwitchName $switch_name -Name $adapter_name 

# starting and stopping vm to get free MAC from node pool
Start-VM -VMName $vm_name; Stop-VM -VMName $vm_name -Force 

# reading assigned dynamic MAC
$mac_address = (Get-VMNetworkAdapter -VMName $vm_name -Name $adapter_name).MacAddress 

# make same MAC as static
Set-VMNetworkAdapter -VMName $vm_name -Name $adapter_name -StaticMacAddress $mac_address 

# rest steps are same for network interface creation

 

 

 

 

3. Get free MAC from Network controller service and use same on VM network adapter.

There is a dedicated centralized network controller service on every Stack-hci cluster, where we can setup global MAC address range pool at network controller service. When we create new network interface on network controller service with dynamic configuration, then it assign free MAC from the global free pool. It is very reliable solution since it is a centralized service. In this, we will follow below steps

1. Create a Network interface in Network controller service with dynamic MAC assignment config, to get assigned free MAC from global MAC range.
2. Read assigned MAC on network interface
3. Assign same MAC address to vm network adapter as static. 

The following powershell commands perform above steps (Note: please use appropriate values in place of parameters):

 

 

 

# Creating network interface with dynamic MAC config on network controller
$vmnicproperties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceProperties
$vmnicproperties.PrivateMacAllocationMethod = "Dynamic"
$vmnicproperties.IsPrimary = $true

$vmnicproperties.DnsSettings = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceDnsSettings
$vmnicproperties.DnsSettings.DnsServers = $dns_server
$ipconfiguration = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfiguration
$ipconfiguration.resourceid = $vm_name + "_IP1"
$ipconfiguration.properties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfigurationProperties
$ipconfiguration.properties.PrivateIPAddress = $ip_address
$ipconfiguration.properties.PrivateIPAllocationMethod = "Static"

$ipconfiguration.properties.Subnet = New-Object Microsoft.Windows.NetworkController.Subnet

$ipconfiguration.properties.subnet.ResourceRef = "/virtualNetworks/" + $vnet_name + "/subnets/" + $subnet_name
$vmnicproperties.IpConfigurations = @($ipconfiguration)
$NIC_name = $vm_name + "_Eth1"
New-NetworkControllerNetworkInterface -ResourceID $NIC_name -Properties $vmnicproperties -ConnectionUri $uri -Confirm:$false -force
Write-host 'NIC config created..'
Start-Sleep -Seconds 8

$nic = Get-NetworkControllerNetworkInterface -ConnectionUri $uri -ResourceId $NIC_name
# Read obtained free MAC from global pool
$mac_address = $nic.Properties.PrivateMacAddress -replace '..(?!$)', '$&-'

###### Vm creation flow starts from here. We will set static MAC here
New-VM -Name $vm_name -MemoryStartupBytes $vm_memory -BootDevice VHD -VHDPath $image_path -Path $vm_data_path -Generation $vm_generation -SwitchName $switch_name
Add-ClusterVirtualMachineRole -vmname $vm_name -Name $vm_name
Set-VMNetworkAdapter -VMName $vm_name -StaticMacAddress "$mac_address"
write-host "Mac set succssfully: $mac_address"

 

 

 

 

Conclusion:

The third/last approach discussed in this article seems to be easy and more reliable, since we are consuming MAC from centralized network controller service. We will never face problem of MAC conflicts incase VMs get migrated from one node to another.

 

Note:

• Approaches discussed in this article are to show possible MAC address assignment strategies, with some advantages and limitations. You can use any strategy as per your requirement. 
• However if you find some better way, you can experiment that as well.