Storage Spaces Direct - Live Migration Uses Wrong Network

%3CLINGO-SUB%20id%3D%22lingo-sub-1343303%22%20slang%3D%22en-US%22%3EStorage%20Spaces%20Direct%20-%20Live%20Migration%20Uses%20Wrong%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1343303%22%20slang%3D%22en-US%22%3E%3CP%3EGood%20Morning%20all%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSTRONG%3E%3CFONT%20color%3D%22%23800080%22%3EThe%20formatting%20in%20this%20text%20is%20the%20best%20I%20can%20do...even%20tabs%2Fbullets%20do%20not%20seem%20to%20work%3C%2FFONT%3E%3C%2FSTRONG%3E%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EI%20have%20read%20a%20LOT%20on%20this%20topic%20and%20here%20is%20a%20summary%20of%20the%20problem%20I%20am%20facing%2C%20plus%20where%20I%20am%20currently.%3CBR%20%2F%3E%3CBR%20%2F%3EThe%20Problem%20%E2%80%93%20Live%20Migration%20is%20using%20the%20wrong%20network.%3CBR%20%2F%3EWhen%20I%20%E2%80%9CLive%20Migrate%E2%80%9D%20a%20VM%2C%20it%20is%20only%20the%20%E2%80%9CHost%20Mgmt%E2%80%9D%20network%20that%20shows%20any%20throughput%20in%20Task%20Manager%3CBR%20%2F%3ELive%20Migration%20does%20work%2C%20but%20I%20cannot%20seem%20to%20find%20a%20way%20to%20force%20the%20Live%20Migration%20traffic%20over%20the%2010GbE%20networks%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMain%20Question%3A%3CBR%20%2F%3E%E2%80%A2%20How%20do%20I%20configure%20the%2010GbE%20networks%20to%20be%20used%20the%20Live%20Migrations%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFollow%20up%20questions%3A%3CBR%20%2F%3E%E2%80%A2%20How%20do%20I%20test%20the%2010GbE%20networks%3F%3CBR%20%2F%3E%E2%80%A2%20How%20do%20I%20test%20the%20Cluster%20configuration%20for%20this%20problem%3F%3CBR%20%2F%3E%E2%80%A2%20How%20do%20I%20perform%20a%20file%20copy%20over%20a%20specific%20interface%20to%20prove%2Fdisprove%20the%20basic%2010GbE%20connectivity%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20seen%20articles%20on%20SMB%20and%20SMB%20Direct%20%E2%80%93%20but%20not%20sure%20where%20to%20go%20now%20as%20I%20have%20tried%20a%20lot%20of%20things%20to%20try%20and%20%E2%80%9Cfix%E2%80%9D%20this.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20would%20be%20grateful%20any%20help%20in%20this%20frustrating%20matter.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESo%2C%20how%20did%20I%20get%20here%E2%80%A6%E2%80%A6%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20Scenario%20%E2%80%93%202%20x%20node%20Storage%20Spaces%20Direct%20cluster%3A%3CBR%20%2F%3E%E2%80%A2%20Windows%202019%20Server%20on%20Dell%20R640%20servers%20with%20QLogic%20QL41162HMRJ%20%E2%80%93%202x1GbE%20%26amp%3B%202x10GbE%20card%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20hardware%20configuration%20includes%3A%3CBR%20%2F%3E%E2%80%A2%20Appropriate%20disks%20for%20caching%20and%20capacity%20-%20all%20working%20satisfactorily%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E2x1GbE%20%E2%80%93%20these%20are%20switched%20and%20connected%20to%20the%20local%20network%20infrastructure.%20I%20have%20created%20a%20SET%20VMSwitch%20and%20converged%20the%20networks%20with%20VMNetAdapters%20on%20top%20-%20all%20working%20satisfactorily%20with%20ping%20and%20routing%20etc%E2%80%A6%3C%2FP%3E%3CP%3E%3CBR%20%2F%3E2x10GbE%20%E2%80%93%20switchless%20%E2%80%93%20standard%20cable%20from%20server%20to%20server%20with%20no%20switch%20in%20between%3CBR%20%2F%3E%E2%80%A2%20These%20will%20be%20used%20for%20S2D%2C%20Live%20Migration%2C%20Storage%20Migration%20etc..%3CBR%20%2F%3E%E2%80%A2%20Set%20to%20iWARP%20in%20BIOS%3CBR%20%2F%3E%E2%80%A2%20RDMA%20enabled%20%26amp%3B%20Mode%20%3D%20iWARP%3CBR%20%2F%3E%E2%80%A2%20Jumbo%20Frames%20enabled%20to%209014%20%E2%80%93%20successfully%20checked%20with%20ping%20to%20ensure%20enabled%3CBR%20%2F%3E%E2%80%A2%20VMQ%20enabled%3CBR%20%2F%3E%E2%80%A2%20RSSProfile%20%3D%20NUMAScalingStatic%3CBR%20%2F%3E%E2%80%A2%20Network%20Direct%20Functionality%20%3D%20Enabled%3CBR%20%2F%3E%E2%80%A2%20Data%20Centre%20Bridging%20not%20configured%20%E2%80%93%20apparently%20not%20needed%20with%20switchless%20cluster%3CBR%20%2F%3E%E2%80%A2%20Each%20interface%20in%20own%20vLAN%3A%3CBR%20%2F%3E%E2%80%A2%20Server1_Port%231%26lt%3B-%26gt%3BServer2_Port%231%20on%20vLAN%20xxx%3CBR%20%2F%3E%E2%80%A2%20Server1_Port%232%26lt%3B-%26gt%3BServer2_Port%232%20on%20vLAN%20yyy%3CBR%20%2F%3E%E2%80%A2%20Each%20NIC%20can%20ping%20its%20corresponding%20partner%20successfully%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20Cluster%20configuration%20includes%3A%3CBR%20%2F%3E%E2%80%A2%20%E2%80%9CTest-Cluster%E2%80%9D%20output%20with%20100%25%20Success%20before%20creation%20of%20Cluster%3CBR%20%2F%3E%E2%80%A2%20Host%20Mgmt%20%3D%20Role%203%20%E2%80%9CCluster%20and%20Client%E2%80%9D%3CBR%20%2F%3E%E2%80%A2%20Other%20Converged%20vNICs%20%3D%20Role%201%20%E2%80%9CCluster%E2%80%9D%3CBR%20%2F%3E%E2%80%A2%20Storage%20%5B10GbE%5D%20%3D%20Role%201%20%E2%80%9CCluster%E2%80%9D%3CBR%20%2F%3E%E2%80%A2%20All%20Resources%20are%20%E2%80%9CUp%E2%80%9D%20in%20Cluster%20GUI%20%E2%80%93%20no%20errors%20reported%3CBR%20%2F%3E%E2%80%A2%20Live%20Migration%20networks%20selected%20are%20ONLY%20the%2010GbE%20ones%20%E2%80%93%20and%20top%20of%20the%20list%20too%3CBR%20%2F%3E%E2%80%A2%20%E2%80%9CValidate%20Cluster%E2%80%9D%20output%20with%20100%25%20Success%20after%20creation%20of%20Cluster%3CBR%20%2F%3E%E2%80%A2%20Cluster%20Events%20show%20%E2%80%9CNO%20Errors%E2%80%9D%20or%20indeed%20any%20records%20at%20all%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHyper-V%20Configuration%20includes%3A%3CBR%20%2F%3E%26nbsp%3B%20%E2%80%A2%20Live%20Migration%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%20Enabled%20%3D%20True%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%204%20Simultaneous%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%20%E2%80%9CUse%20these%20IP%20Addresses%20for%20Live%20Migration%20%E2%80%93%20is%20enabled%20and%20show%20the%20correct%20addresses%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%20Advanced%20Features%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%20%E2%80%9CUse%20Kerberos%E2%80%9D%20%3D%20true%3CBR%20%2F%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%E2%80%A2%20%E2%80%9CSMB%E2%80%9D%20%3D%20true%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20listening.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EStephen.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1343303%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EClustering%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ENetworking%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EWindows%20Server%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1743658%22%20slang%3D%22en-US%22%3ERe%3A%20Storage%20Spaces%20Direct%20-%20Live%20Migration%20Uses%20Wrong%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1743658%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F639472%22%20target%3D%22_blank%22%3E%40Stephen666%3C%2FA%3E%26nbsp%3B%20%26nbsp%3B%20Try%20check%20Jumbo%20Frame%20config.%3C%2FP%3E%3CP%3EOn%20my%20HP%20servers%20sometime%20after%20restart%20it%20switch%20from%20Ethernet%20to%20InfiniBand%20and%20after%20revert%20to%20Ethernet%20all%20settings%20keep%20same%20except%20Jumbo%20which%20became%201514%20instead%20of%209014.%20After%20correct%20jumbo%20it%20starting%20use%20a%20right%20network.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1763227%22%20slang%3D%22en-US%22%3ERe%3A%20Storage%20Spaces%20Direct%20-%20Live%20Migration%20Uses%20Wrong%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1763227%22%20slang%3D%22en-US%22%3EHi%2C%3CBR%20%2F%3E%3CBR%20%2F%3EI%20will%20check%20it%20out%20the%20server%20NIC%20settings%20as%20this%20a%20two-node%20switchless%20cluster.%3CBR%20%2F%3E%3CBR%20%2F%3EThanks%20for%20the%20tip%20though.%3CBR%20%2F%3E%3CBR%20%2F%3EStephen%3C%2FLINGO-BODY%3E
Highlighted
New Contributor

Good Morning all,

 

The formatting in this text is the best I can do...even tabs/bullets do not seem to work


I have read a LOT on this topic and here is a summary of the problem I am facing, plus where I am currently.

The Problem – Live Migration is using the wrong network.
When I “Live Migrate” a VM, it is only the “Host Mgmt” network that shows any throughput in Task Manager
Live Migration does work, but I cannot seem to find a way to force the Live Migration traffic over the 10GbE networks

 

Main Question:
• How do I configure the 10GbE networks to be used the Live Migrations?

 

Follow up questions:
• How do I test the 10GbE networks?
• How do I test the Cluster configuration for this problem?
• How do I perform a file copy over a specific interface to prove/disprove the basic 10GbE connectivity?

 

I have seen articles on SMB and SMB Direct – but not sure where to go now as I have tried a lot of things to try and “fix” this.

 

I would be grateful any help in this frustrating matter.

 

So, how did I get here……

 

The Scenario – 2 x node Storage Spaces Direct cluster:
• Windows 2019 Server on Dell R640 servers with QLogic QL41162HMRJ – 2x1GbE & 2x10GbE card

 

The hardware configuration includes:
• Appropriate disks for caching and capacity - all working satisfactorily

 

2x1GbE – these are switched and connected to the local network infrastructure. I have created a SET VMSwitch and converged the networks with VMNetAdapters on top - all working satisfactorily with ping and routing etc…


2x10GbE – switchless – standard cable from server to server with no switch in between
• These will be used for S2D, Live Migration, Storage Migration etc..
• Set to iWARP in BIOS
• RDMA enabled & Mode = iWARP
• Jumbo Frames enabled to 9014 – successfully checked with ping to ensure enabled
• VMQ enabled
• RSSProfile = NUMAScalingStatic
• Network Direct Functionality = Enabled
• Data Centre Bridging not configured – apparently not needed with switchless cluster
• Each interface in own vLAN:
• Server1_Port#1<->Server2_Port#1 on vLAN xxx
• Server1_Port#2<->Server2_Port#2 on vLAN yyy
• Each NIC can ping its corresponding partner successfully

 

The Cluster configuration includes:
• “Test-Cluster” output with 100% Success before creation of Cluster
• Host Mgmt = Role 3 “Cluster and Client”
• Other Converged vNICs = Role 1 “Cluster”
• Storage [10GbE] = Role 1 “Cluster”
• All Resources are “Up” in Cluster GUI – no errors reported
• Live Migration networks selected are ONLY the 10GbE ones – and top of the list too
• “Validate Cluster” output with 100% Success after creation of Cluster
• Cluster Events show “NO Errors” or indeed any records at all

 

Hyper-V Configuration includes:
  • Live Migration
    • Enabled = True
    • 4 Simultaneous
    • “Use these IP Addresses for Live Migration – is enabled and show the correct addresses
    • Advanced Features
      • “Use Kerberos” = true
      • “SMB” = true

 

Thanks for listening.

 

Stephen.

 

2 Replies
Highlighted

@Stephen666    Try check Jumbo Frame config.

On my HP servers sometime after restart it switch from Ethernet to InfiniBand and after revert to Ethernet all settings keep same except Jumbo which became 1514 instead of 9014. After correct jumbo it starting use a right network.

 

Highlighted
Hi,

I will check it out the server NIC settings as this a two-node switchless cluster.

Thanks for the tip though.

Stephen