Forum Discussion
Guidance on Network Configurations for Azure Stack HCI Cluster
Hi HasanHasib,
I would recommend reaching out to a partner that can help you setup the cluster. If you bought a new solution the vendor can definitely help you. Also Windows Admin Center helps out when new to Azure Stack HCI and configures the right settings for you such as SET switch.(1) How to assign the IP addresses? since it's my first time with the black screen.
You can assign IP adresses using the 'sconfig' screen. When you login, you will end up on that screen automatically and then you type '8' to go to the networking config screen.(2) How to configure SET for the NICs to be used for virtualization/compute step-by-step?
You can specify creating a SET switch when executing the 'New-VMSwitch' cmdlet, see New-VMSwitch (Hyper-V) | Microsoft Learn(3) I know that I don't need to configure SET (Switch Embedded Teaming) for the Storage NICs, but what should I do for the Mgmt NICs? Should I configure SET for them OR should 1 NIC be enough for the Mgmt?
Based on your requirements (bandwidth etc) you can go either way. For production I prefer to use two.(4) How to configure SET step-by-step?
Same answer as (2).(5) What is the recommendation for the IP address scheming for all the NICs? Let's say I'm assigning 10.4.8.x to the Mgmt NICs, should I use the same IP address scheme for Storage and Compute NICs as well, or should I assign some different IP schemes like 172.x.x.x
You can set it up as you wish depending on your network architecture. Can you stay in the same 'scheme' so that everything is in 10.4.x.x range. So for example:
Management: 10.4.8.x
Storage1: 10.4.9.x
Storage2: 10.4.10.x
Or you can use smaller subnets to save IP addresses.(6) There are 2 NICs for Storage and we are not configuring SET for them... So, just to confirm, will I need to assign 2 IP addresses for Storage on each server?
Yes.(7) How to enable the iWARP RDMA for the Storage NICs? I'm thinking to use iWARP RDMA as I read MSFT engineers recommend/prefer iWARP over RoCE. Please confirm my understanding.
This is a setting you enable on the physical network card. Depending on the brand you can set it using Set-NetAdapterAdvancedProperty(8) For the Mgmt NIC, first I will configure the SET then I will assign the IP address to the SET NIC, just like we used to with the conventional NIC Teaming in Windows Server 2012 R2.
Correct.--
Darryl van der Peijl
Splitbrain
Run Get-netadapter and keep notes of NIC.
use Sconfig to assign IP address
(2) How to configure SET for the NICs to be used for virtualization/compute step-by-step?
If you are using WAC to configure so it will provide you option to configure NIC.
My suggestion create SET (MGMT + Compute) Set2 (only for SMB)
(3) I know that I don't need to configure SET (Switch Embedded Teaming) for the Storage NICs, but what should I do for the Mgmt NICs? Should I configure SET for them OR should 1 NIC be enough for the Mgmt?
i gave answer above.
(4) How to configure SET step-by-step?
You can use WAC or use Powershell Command to configure SET
(5) What is the recommendation for the IP address scheming for all the NICs? Let's say I'm assigning 10.4.8.x to the Mgmt NICs, should I use the same IP address scheme for Storage and Compute NICs as well, or should I assign some different IP schemes like 172.x.x.x
MGMT = 192.168.0.10/26 VLAN 10
Compute = 192.168.1.11/26 VLAN 11
SMB= 192.168.2/11/26 VLAN 12
(6) There are 2 NICs for Storage and we are not configuring SET for them... So, just to confirm, will I need to assign 2 IP addresses for Storage on each server?
yes
(7) How to enable the iWARP RDMA for the Storage NICs? I'm thinking to use iWARP RDMA as I read MSFT engineers recommend/prefer iWARP over RoCE. Please confirm my understanding.
check the documenantion, i personally suggest to use ROCV2 over iWARP
(8) For the Mgmt NIC, first I will configure the SET then I will assign the IP address to the SET NIC, just like we used to with the conventional NIC Teaming in Windows Server 2012 R2.
First assign IP Address and it will create NIC after that
HasanHasib
1. assign IP, Gateway on your first management NIC (lowest MACadress) on each nodes
2. for management NICs (for non-converged), and storage NICs (fully converged)
make sure that SR-IOV is in your UEFI and VF are set according to your vendors documentation. {127 instead 8 (default value)} these settings in UEFI are per NIC, not per NIC Port
reboot
3. use NetworkATC to finalize your basic configuration
install-module networkATC
#assume that mgmt traffic is VLAN 0 or untagged, and you don't want to use RDMA for compute / management, otherwise you need tagged VLAN for RDMA
#your pair of management NIC Ports, they should not be both on the same NIC for true redundancy, e.g. NIC FW updates
$Compute_Adapters = @('Slot 2 Port 1','Slot 7 Port 1')
$compute_vlan = @(0)
Add-NetIntent -Name Compute -Management -AdapterName $Compute_Adapters -Verbose -StorageVlans $compute_vlan
Strictly avoid to create manual SET switches this and also WAC uses different options that may not be optimal, especially if you care for SR-IOV.
after NetworkATC has created SET Switch "convergedSwitch(Compute)"
and the OS NIC "vEthernet(Compute)"
Check if everything is alright with SR-IOV
Get-VMSwitch *compute* | select *iov*
creating the SET Switch it will always use the first NIC in the pair above to take over the IP etc.
example
Name : ConvergedSwitch(compute)
IovEnabled : True
IovSupport : True
IovSupportReasons :
IovQueuePairCount : 130558
IovQueuePairsInUse : 64
IovVirtualFunctionCount : 254
IovVirtualFunctionsInUse : 0
There should be functions available. if it is not true, then there is something wrong in the UEFI settings. In this case you have to remove the intent and the vSwitch and start over.
If you do not need SR-IOV then it is easier. you do not have to alter UEFI NIC settings and can WAC can be used, too.