Home
%3CLINGO-SUB%20id%3D%22lingo-sub-388121%22%20slang%3D%22en-US%22%3ENo%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-388121%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20a%20blog%20that%20has%20been%20a%20long%20time%20coming.%26nbsp%3B%20From%20time%20to%20time%2C%20we%20get%20a%20request%20about%20how%20to%20configure%20networking%20in%20Failover%20Clusters.%26nbsp%3B%20One%20of%20the%20questions%20we%20get%20is%20how%20should%20the%20heartbeat%20network%20be%20configured%20and%20that%20is%20what%20the%20focus%20is%20on%20this%20blog.%26nbsp%3B%20I%20am%20here%20to%20say%2C%20there%20is%20no%20such%20thing%2C%20and%20never%20was%2C%20a%20heartbeat%20network.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EPlease%20allow%20me%20to%20give%20a%20little%20background%20and%20explain.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20Windows%202003%20and%20below%20Failover%20Clustering%2C%20you%20could%20define%20which%20network%20was%20used%20for%20Cluster%20Communication.%26nbsp%3B%20Below%20is%20a%20picture%20for%20reference.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22text-align%3A%20center%3B%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20372px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F99808i7A1BFFE52D37672F%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22Heartbeat1.jpg%22%20title%3D%22Heartbeat1.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20style%3D%22text-align%3A%20center%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20the%20picture%20above%2C%20we%20would%20want%20to%20select%20Private%20for%20our%20Cluster%20Communication%20to%20as%20to%20not%20use%20the%20Public%20which%20has%20all%20WAN%20traffic.%26nbsp%3B%20All%20Cluster%20Communication%20between%20nodes%20(joins%2C%20registry%20updates%2Fchanges%2C%20etc)%20would%20go%20only%20over%20this%20network%20if%20it%20is%20up.%26nbsp%3B%20As%20the%20picture%20shows%2C%20the%20networks%20are%20called%20Public%20and%20Private.%26nbsp%3B%20As%20years%20went%20by%2C%20some%20started%20calling%20the%20Private%20network%20a%20Heartbeat%20network.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EHeartbeats%20are%20small%20packets%20(134%20bytes)%20that%20travel%20over%20UDP%20Port%203343%20on%20all%20networks%20configured%20for%20Cluster%20use%20between%20all%20nodes.%26nbsp%3B%20They%20serve%20multiple%20purposes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EEstablishes%20if%20a%20Cluster%20Network%20is%20up%20or%20down%3C%2FLI%3E%0A%3CLI%3EEstablishes%20routes%20between%20nodes%3C%2FLI%3E%0A%3CLI%3EEnsure%20the%20health%20of%20a%20node%20is%20good%20or%20bad%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3ESo%20let's%20say%20I%20have%20Private%20set%20as%20my%20priority%20network%20for%20Cluster%20Communications.%26nbsp%3B%20If%20it%20is%20up%2C%20we%20are%20sending%20our%20communication%20through%20it.%26nbsp%3B%20But%20what%20happens%20if%20that%20network%20wasn't%20reliable.%26nbsp%3B%20If%20a%20node%20tries%20to%20join%20and%20packets%20are%20dropping%2C%20then%20the%20join%20could%20fail.%26nbsp%3B%20If%20this%20is%20the%20case%2C%20you%20either%20determine%20where%20the%20problem%20is%20and%20fix%20it%2C%20or%20go%20back%20into%20the%20Cluster%20properties%20and%20set%20the%20Public%20as%20priority.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EStarting%20in%20Windows%202008%20Failover%20Clusters%2C%20the%20concept%20of%20Public%20and%20Private%20networks%20went%20out%20the%20window.%26nbsp%3B%20We%20will%20now%20send%20Cluster%20Communication%20over%20any%20of%20our%20networks.%26nbsp%3B%20One%20of%20the%20reasons%20for%20this%20was%20reliability.%26nbsp%3B%20With%20that%20change%2C%20we%20also%20gave%20the%20heartbeats%20an%20additional%20purpose.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EDetermine%20the%20fastest%20and%20reliable%20routes%20between%20nodes%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3ESince%20we%20are%20now%20determining%20the%20fastest%20and%20reliable%20routes%2C%20we%20could%20use%20different%20networks%20between%20nodes%20for%20our%20communication.%26nbsp%3B%20Take%20the%20below%20as%20an%20example.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22text-align%3A%20center%3B%22%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20style%3D%22width%3A%20384px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F99831i2DACE0C54E9095EF%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20alt%3D%22Heartbeat2.jpg%22%20title%3D%22Heartbeat2.jpg%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%20style%3D%22text-align%3A%20center%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20have%20three%20individual%20networks%20between%20our%20nodes%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%20style%3D%22text-align%3A%20left%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3E%3CFONT%20color%3D%22%230000FF%22%3E%3CSPAN%3EBlue%3C%2FSPAN%3E%3C%2FFONT%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%E2%80%93%2010%20gbps%20used%20for%20backups%20and%20administration%20only%3C%2FLI%3E%0A%3CLI%3E%3CFONT%20color%3D%22%23008000%22%3EGreen%3C%2FFONT%3E%3CSPAN%20style%3D%22font-family%3A%20inherit%3B%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20style%3D%22font-family%3A%20inherit%3B%22%3E%E2%80%93%2040%20gbps%20used%20for%20communicating%20out%20on%20the%20WAN%20to%20clients%3C%2FSPAN%3E%3C%2FLI%3E%0A%3CLI%3E%3CFONT%20color%3D%22%23FF0000%22%3E%3CSPAN%3ERed%3C%2FSPAN%3E%3C%2FFONT%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%E2%80%93%2040%20gbps%20used%20for%20communicating%20out%20on%20the%20WAN%20to%20clients%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%20style%3D%22text-align%3A%20left%3B%22%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CLI-WRAPPER%3E%3C%2FLI-WRAPPER%3E%3C%2FP%3E%0A%3CP%3EAs%20a%20refresher%2C%20here%20is%20what%20the%20heartbeats%20are%20doing%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%3EEstablishes%20if%20a%20Cluster%20Network%20is%20up%20or%20down%3C%2FLI%3E%0A%3CLI%3EEstablishes%20routes%20between%20nodes%3C%2FLI%3E%0A%3CLI%3EEnsure%20the%20health%20of%20a%20node%20is%20good%20or%20bad%3C%2FLI%3E%0A%3CLI%3EDetermine%20the%20fastest%20and%20reliable%20routes%20between%20nodes%3C%2FLI%3E%0A%3C%2FUL%3E%0A%3CP%3EWhat%20the%20heartbeats%20are%20going%20to%20tell%20the%20Cluster%20is%20to%20use%20one%20of%20the%20faster%20networks%20for%20its%20communication.%26nbsp%3B%20With%20that%20as%20the%20case%2C%20it%20is%20going%20to%20use%20either%20Red%20or%20Green%20network.%26nbsp%3B%20If%20the%20heartbeats%20start%20detecting%20that%20neither%20of%20these%20is%20as%20reliable%20(i.e.%20dropping%20a%20packet%2C%20network%20congested%2C%20etc)%2C%20it%20will%20automatically%20switch%20and%20use%20the%20Blue%20network.%26nbsp%3B%20That's%20it%2C%20nothing%20for%20you%20to%20configure%20extra.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ESo%20to%20wrap%20things%20up%2C%20remember%20these%20things%20about%20Failover%20Clusters%20and%20Heartbeats.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EThere%20is%20no%20such%20thing%20as%20a%20heartbeat%20network%20or%20a%20network%20dedicated%20to%20heartbeats%3C%2FLI%3E%0A%3CLI%3EHeartbeat%20packets%20are%20lightweight%20(134%20bytes%20in%20size)%3C%2FLI%3E%0A%3CLI%3EHeartbeats%20are%20sensitive%20to%20latency%3C%2FLI%3E%0A%3CLI%3EBandwidth%20is%20not%20an%20important%20factor%2C%20quality%20of%20service%20is.%26nbsp%3B%20If%20your%20network%20is%20all%20teamed%2C%20ensure%20you%20have%20set%20up%20Network%20QOS%20policies%20for%20our%20UDP%203343%20traffic.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3EFor%20more%20information%20regarding%20configuring%20networks%20in%20a%20Cluster%2C%20please%20see%20Microsoft%20Ignite%20session%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fchannel9.msdn.com%2FEvents%2FTechEd%2FNorthAmerica%2F2013%2FMDC-B337%23fbid%3DZpvM0cLRvyX%22%20target%3D%22_self%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3EFailover%20Clustering%20Networking%20Essentials%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EHappy%20Clustering%20!!!!%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EJohn%20Marlin%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ESenior%20Program%20Manager%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3EMicrosoft%20Corporation%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%3ETwitter%3A%26nbsp%3B%40Johnmarlin_MSFT%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-388121%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20continue%20to%20here%20about%20configuring%20a%20%22heartbeat%20network%22.%26nbsp%3B%20This%20blogs%20will%20explain%20that%20ther%20eis%20no%20such%20thing.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-SUB%20id%3D%22lingo-sub-394054%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-394054%22%20slang%3D%22en-US%22%3E%3CP%3EGreat%2C%20thanks%20for%20your%20help%20John!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-394036%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-394036%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20Cluster%20networking%20component%20NetFt%20is%20what%20decides%20which%20path%20it%20will%20take%20for%20Cluster%20communications.%26nbsp%3B%20PlumbAllCrossSubnetRoutes%20simply%20just%20tries%20to%20establish%20if%20a%20new%20route%20can%20be%20find%20that%20NetFt%20can%20use%20if%20it%20needs%20it.%26nbsp%3B%20For%20the%20communications%20itself%2C%20it%20is%20not%20round%20robin.%26nbsp%3B%20NetFt%20will%20establish%20a%20route%20for%20the%20communications%20and%20will%20generally%20tend%20to%20stay%20on%20it%20unless%20a%20better%20one%20is%20found.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-394028%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-394028%22%20slang%3D%22en-US%22%3E%3CP%3ESo%20with%26nbsp%3B%3CSPAN%3E(Get-Cluster).%20PlumbAllCrossSubnetRoutes%20%3D%201%20set%20will%20the%20cluster%20still%20prefer%20the%20faster%20routes%20or%20spread%20cluster%20comms%20across%20all%20known%20network%20in%20a%20round%20robin%20or%20load%26nbsp%3Bbalancing%20fashion%3F%20Or%20do%20all%20comms%20stick%20with%20one%20network%20path%20until%20it%20fails%20or%20slows%20down%3F%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EThanks!%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-394024%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-394024%22%20slang%3D%22en-US%22%3E%3CP%3EPlumbAllCrossSubnetRoutes%20is%20different.%26nbsp%3B%20This%20property%20is%26nbsp%3Bused%20to%20detect%20if%20there%20are%20other%20routes%20we%20can%20find.%26nbsp%3B%20For%20example%2C%20Node1%20has%201.1.1.1%20and%20192.1.1.1%20networks.%26nbsp%3B%20Node2%20has%201.1.1.2%20and%20192.1.1.2%20networks.%26nbsp%3B%20Cluster%20Network%201%20would%20comprise%20the%201.x%20network.%26nbsp%3B%20Cluster%20Network%202%20would%20comprise%20the%20192.x%20network.%26nbsp%3B%20What%20this%20property%20setting%20would%20do%20is%20periodically%20see%20if%20it%20can%20make%20a%20connection%20between%201.1.1.1%20and%20192.1.1.2%20as%20well%20as%201.1.1.2%20and%20192.1.1.1.%26nbsp%3B%20If%20it%20can%2C%20NetFt%20will%20add%20it%20to%20it's%20internal%20route%20table.%26nbsp%3B%20If%20it%20cannot%2C%20it%20will%20check%20again%20at%20a%20later%20time.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EMore%20Info%3A%3C%2FP%3E%0A%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FFailover-Clustering%2FWhat-the-heck-is-PlumbAllCrossSubnetRoutes%2Fba-p%2F371348%22%20target%3D%22_blank%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FFailover-Clustering%2FWhat-the-heck-is-PlumbAllCrossSubnetRoutes%2Fba-p%2F371348%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-394017%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-394017%22%20slang%3D%22en-US%22%3E%3CP%3EThanks%20for%20the%20write%20up%20John!%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESo%20should%20we%20still%20use%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3E(Get-Cluster).%20PlumbAllCrossSubnetRoutes%20%3D%201%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eto%20use%20all%20routes%20for%20cluster%20comms%3F%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-389112%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-389112%22%20slang%3D%22en-US%22%3E%3CP%3EWell%20I%20understand%20your%20theory%20behind%20this%20in%20practice%20there%20are%20other%20products%20besides%20Windows%20clustering%20were%20a%20heartbeat%20network%20is%20still%20a%20thing.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ENow%20with%20that%20said%20I%20will%20use%20the%20term%20Network%20loosely...%20As%20I%20don't%20really%20consider%20a%20crossover%20cable%20between%20two%20physical%20servers%20a%20network%20per%20se.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThere%20are%20systems%20out%20there%20such%20as%20drbd%20and%20other%20technologies%2C%20as%20I'm%20just%20using%20this%20as%20an%20example%2C%20we're%20a%20heartbeat%20is%20only%20sent%20out%20over%20one%20network%20and%20configured%20as%20such.%20Other%20configurations%20may%20exist%20to%20approximate%20where%20Windows%20clustering%20has%20progressed%20to%2C%20but%20not%20everything%20is%20there...%20Yet.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-500730%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-500730%22%20slang%3D%22en-US%22%3E%3CP%3EJohn%2C%3C%2FP%3E%3CP%3E%22Heartbeats%20are%20sensitive%20to%20latency%22%3C%2FP%3E%3CP%3EQuestion%3A%20Is%20this%20heartbeat%20information%20the%20the%20keep-alive%20in%20reference%20to%20a%20Converged%20Network%20topology%2C%20Non-Converged%20Network%20topology%20or%20there%20is%20no%20difference%20from%20your%20point%20of%20view%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-500756%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-500756%22%20slang%3D%22en-US%22%3E%3CP%3Ethere%20is%20no%20difference.%26nbsp%3B%20can%20we%20see%20the%20packet%20and%20the%20sequence%20number%20in%20the%20packet%20and%20can%20we%20return.%26nbsp%3B%20It%20is%20simply%20a%20matter%20of%20can%20we%20connect%20through%20the%20network.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-503259%22%20slang%3D%22en-US%22%3ERe%3A%20No%20such%20thing%20as%20a%20Heartbeat%20Network%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-503259%22%20slang%3D%22en-US%22%3E%3CP%3Ethank%20you%20John.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EFrom%20our%20view%20point%20we%20see%20the%20difference%20as%20actual%20bandwidth%20per%20network.%26nbsp%3B%20Items%20that%20can%20compete%20for%20cycles%20on%20an%20interface%2C%20like%20multiple%20networks%2C%20are%20not%20a%20factor%20in%20a%20dedicated%20topology%20(non-converged).%26nbsp%3B%20The%20Keepalive%20(heartbeat)%20using%20all%20networks%20available%2C%20smart%20move!%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EJP%3C%2FP%3E%3C%2FLINGO-BODY%3E
Microsoft

This is a blog that has been a long time coming.  From time to time, we get a request about how to configure networking in Failover Clusters.  One of the questions we get is how should the heartbeat network be configured and that is what the focus is on this blog.  I am here to say, there is no such thing, and never was, a heartbeat network.

 

Please allow me to give a little background and explain.

 

In Windows 2003 and below Failover Clustering, you could define which network was used for Cluster Communication.  Below is a picture for reference.

 

Heartbeat1.jpg

 

In the picture above, we would want to select Private for our Cluster Communication to as to not use the Public which has all WAN traffic.  All Cluster Communication between nodes (joins, registry updates/changes, etc) would go only over this network if it is up.  As the picture shows, the networks are called Public and Private.  As years went by, some started calling the Private network a Heartbeat network.

 

Heartbeats are small packets (134 bytes) that travel over UDP Port 3343 on all networks configured for Cluster use between all nodes.  They serve multiple purposes.

 

  • Establishes if a Cluster Network is up or down
  • Establishes routes between nodes
  • Ensure the health of a node is good or bad

So let's say I have Private set as my priority network for Cluster Communications.  If it is up, we are sending our communication through it.  But what happens if that network wasn't reliable.  If a node tries to join and packets are dropping, then the join could fail.  If this is the case, you either determine where the problem is and fix it, or go back into the Cluster properties and set the Public as priority.

 

Starting in Windows 2008 Failover Clusters, the concept of Public and Private networks went out the window.  We will now send Cluster Communication over any of our networks.  One of the reasons for this was reliability.  With that change, we also gave the heartbeats an additional purpose.

 

  • Determine the fastest and reliable routes between nodes

Since we are now determining the fastest and reliable routes, we could use different networks between nodes for our communication.  Take the below as an example.

 

Heartbeat2.jpg

 

We have three individual networks between our nodes:

 

 

  • Blue – 10 gbps used for backups and administration only
  • Green – 40 gbps used for communicating out on the WAN to clients
  • Red – 40 gbps used for communicating out on the WAN to clients

 

 

As a refresher, here is what the heartbeats are doing:

 

  • Establishes if a Cluster Network is up or down
  • Establishes routes between nodes
  • Ensure the health of a node is good or bad
  • Determine the fastest and reliable routes between nodes

What the heartbeats are going to tell the Cluster is to use one of the faster networks for its communication.  With that as the case, it is going to use either Red or Green network.  If the heartbeats start detecting that neither of these is as reliable (i.e. dropping a packet, network congested, etc), it will automatically switch and use the Blue network.  That's it, nothing for you to configure extra.

 

So to wrap things up, remember these things about Failover Clusters and Heartbeats.

 

  1. There is no such thing as a heartbeat network or a network dedicated to heartbeats
  2. Heartbeat packets are lightweight (134 bytes in size)
  3. Heartbeats are sensitive to latency
  4. Bandwidth is not an important factor, quality of service is.  If your network is all teamed, ensure you have set up Network QOS policies for our UDP 3343 traffic.

For more information regarding configuring networks in a Cluster, please see Microsoft Ignite session:

 

Failover Clustering Networking Essentials

 

Happy Clustering !!!!

 

John Marlin

Senior Program Manager

Microsoft Corporation

Twitter: @Johnmarlin_MSFT

9 Comments
Regular Visitor

Well I understand your theory behind this in practice there are other products besides Windows clustering were a heartbeat network is still a thing. 

 

Now with that said I will use the term Network loosely... As I don't really consider a crossover cable between two physical servers a network per se. 

 

There are systems out there such as drbd and other technologies, as I'm just using this as an example, we're a heartbeat is only sent out over one network and configured as such. Other configurations may exist to approximate where Windows clustering has progressed to, but not everything is there... Yet.

Visitor

Thanks for the write up John!

 

So should we still use

 

(Get-Cluster). PlumbAllCrossSubnetRoutes = 1

 

to use all routes for cluster comms? 

Microsoft

PlumbAllCrossSubnetRoutes is different.  This property is used to detect if there are other routes we can find.  For example, Node1 has 1.1.1.1 and 192.1.1.1 networks.  Node2 has 1.1.1.2 and 192.1.1.2 networks.  Cluster Network 1 would comprise the 1.x network.  Cluster Network 2 would comprise the 192.x network.  What this property setting would do is periodically see if it can make a connection between 1.1.1.1 and 192.1.1.2 as well as 1.1.1.2 and 192.1.1.1.  If it can, NetFt will add it to it's internal route table.  If it cannot, it will check again at a later time.

 

More Info:

https://techcommunity.microsoft.com/t5/Failover-Clustering/What-the-heck-is-PlumbAllCrossSubnetRoute...

 

Visitor

So with (Get-Cluster). PlumbAllCrossSubnetRoutes = 1 set will the cluster still prefer the faster routes or spread cluster comms across all known network in a round robin or load balancing fashion? Or do all comms stick with one network path until it fails or slows down? 

 

Thanks!

Microsoft

The Cluster networking component NetFt is what decides which path it will take for Cluster communications.  PlumbAllCrossSubnetRoutes simply just tries to establish if a new route can be find that NetFt can use if it needs it.  For the communications itself, it is not round robin.  NetFt will establish a route for the communications and will generally tend to stay on it unless a better one is found.

Visitor

Great, thanks for your help John!

Occasional Visitor

John,

"Heartbeats are sensitive to latency"

Question: Is this heartbeat information the the keep-alive in reference to a Converged Network topology, Non-Converged Network topology or there is no difference from your point of view?

 

 

Microsoft

there is no difference.  can we see the packet and the sequence number in the packet and can we return.  It is simply a matter of can we connect through the network.

Occasional Visitor

thank you John.  

 

From our view point we see the difference as actual bandwidth per network.  Items that can compete for cycles on an interface, like multiple networks, are not a factor in a dedicated topology (non-converged).  The Keepalive (heartbeat) using all networks available, smart move! 

 

JP