SOLVED
Home

Consistent Thin Client Disconnection from WVD Pool

%3CLINGO-SUB%20id%3D%22lingo-sub-1063353%22%20slang%3D%22en-US%22%3EConsistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1063353%22%20slang%3D%22en-US%22%3E%3CP%3EHello%20we%20have%20been%20experiencing%20some%20random%20but%20consistent%20disconnects%20from%20our%20WVD%20Pool.%20We%20have%20roughly%2010%20users%20and%20have%20been%20getting%20different%20event%20viewer%20logs%20for%20when%20they%20disconnect.%20We%20have%20Thin%20Clients%20on%20Windows%2010%20version%201607.%20When%20the%20users%20disconnect%20it%20will%20happen%20multiple%20times%20per%20day%2C%20however%20some%20days%20they%20do%20not%20disconnect.%20Attached%20are%20the%20event%20viewer%20logs%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1074116%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1074116%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20have%20the%20same%20random%20disconnects%20from%20a%20WVD%20Pool%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1074362%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1074362%22%20slang%3D%22en-US%22%3EWhich%20Remote%20Desktop%20app%20version%20are%20you%20currently%20using%3F%20Had%20the%20same%20issue%20before%20with%20previous%20version.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1076769%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1076769%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F486989%22%20target%3D%22_blank%22%3E%40Sbarnet%3C%2FA%3E%3A%20Thank%20you%20for%20sharing%20the%20logs.%20Unfortunately%20there%20isn't%20enough%20information%20to%20debug%20the%20disconnects.%20I%20recommend%20to%20open%20a%20support%20case%20and%20provide%20activity%20IDs%20for%20the%20connections%20you%20see%20issues%20with.%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1218496%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1218496%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F486989%22%20target%3D%22_blank%22%3E%40Sbarnet%3C%2FA%3E%26nbsp%3BHi%2C%20we%20are%20also%20experiencing%20these%20issues%20for%20two%20weeks%20now%20(50%20user%20environment).%3C%2FP%3E%3CP%3E-%20Happens%20from%20multiple%20locations%20(so%20independent%20of%20network%20or%20ISP).%3C%2FP%3E%3CP%3E-%20Happens%20on%20both%20Windows%2010%20and%20HTML5%20clients.%3C%2FP%3E%3CP%3E-%20Seems%20to%20happen%20completely%20random.%3C%2FP%3E%3CP%3E-%20Problem%20so%20far%20not%20related%20to%20a%20particular%20WVD%20host%20in%20the%20pool.%3C%2FP%3E%3CP%3E-%20Some%20days%20no%20problems%20at%20all%2C%20the%20next%20day%20it%20could%20me%2080%2B%20disconnects%20on%20one%20day.%3C%2FP%3E%3CP%3E-%20It%20seems%20to%20be%20related%20to%20a%20session.%20When%20user%20A%20gets%20is%20in%20the%20morning%2C%20chances%20are%20high%20he%20will%20get%20it%20multiple%20time%20a%20day%2C%20until%20he%20completely%20logs%20off%20and%20on%20again%2C%20so%20a%20completely%20new%20sessions%20is%20build%20up.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAlready%20have%20a%20Microsoft%20case%20running%20for%20this%2C%20but%20so%20far%20no%20solution%2C%20or%20not%20even%20a%20cause.%20I%20suspect%20Azure's%20WVD%20gateway%2C%20but%20can't%20prove%20this%20yet.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDoes%20anyone%20else%20experience%20this%20in%20the%20West%20Europe%20region%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGreets%2C%20Marco%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1218840%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1218840%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F140582%22%20target%3D%22_blank%22%3E%40Marco%20Brouwer%3C%2FA%3E%26nbsp%3BWe've%20been%20having%20the%20same%20issue%20for%20about%20a%20week%20and%20a%20half.%20We%20have%20about%20100%20users%20on%20WVD.%20Last%20week%20everyone%20got%20kicked%20off%20and%20unable%20to%20connect.%20I%20log%20them%20off%20via%20the%20WVDAdmin%20tool%20and%20they%20can%20get%20back%20in.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EToday%20we%20had%20a%20similar%20issue%20but%20only%20a%20had%20full%20of%20users%20got%20the%20sessions%20kicked%20off.%20I%20believe%20also%20this%20has%20to%20do%20with%20the%20WVD%20Session%20Gateway.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1218859%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1218859%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F318058%22%20target%3D%22_blank%22%3E%40mmarti1223%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20for%20the%20heads%20up!%20We%20do%20not%20experience%20the%20same.%20When%20we%20are%20disconnected%2C%20users%20auto-reconnect%2C%20or%20they%20connect%20again%20to%20their%20session%20manually.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E-%20Sometimes%20it%20happens%20just%20for%20one%20user.%3C%2FP%3E%3CP%3E-%20Sometimes%20it%20happens%20for%20multiple%20users%20simultaneously%20(but%20not%20all).%20Those%20users%20are%20not%20(always)%20on%20same%20WVD%20host%2C%20and%20not%20on%20same%20client%20network%20%2F%20ISP%20(we%20have%20a%20few%20clients%20connecting%20from%20a%204G%20LTE%20router%20to%20rule%20out%20the%20customers%20network%20or%20ISP).%3C%2FP%3E%3CP%3E-%20Today%20it%20happened%20for%20all%20logged%20users%20at%20once%2C%20for%20the%20first%20time.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWhat%20really%20sticks%20out%2C%20is%20that%20is%20seems%20connected%20to%20particular%20sessions.%20User%20A%20get%20kicks%20out%20today%204%20times%2C%20but%20tomorrow%20he's%20fine.%20Instead%2C%20user%20B%20gets%20kicked%20out%20tomorrow%20multiple%20times.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20asked%20users%20to%20completely%20log%20off%20and%20and%20log%20back%20in%20again%20after%20experiencing%20a%20disconnect%2C%20that%20%22seems%22%20to%20help.%20They%20don't%20get%20kicked%20out%20again%20that%20day.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EYou%20almost%20should%20say%20that%20a%20%22new%22%20session%20is%20being%20brokered%20%2F%20load%20balancing%20through%20another%20server%20in%20the%20Azure%20gateway%20cluster%2C%20which%20does%20not%20have%20the%20problem%20of%20disconnecting%20users%20at%20random.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDo%20you%20have%20contact%20with%20Microsoft%20about%20this%3F%20Can%20you%20tell%20me%20what%20troubleshooting%20steps%20have%20been%20taken%20so%20far%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1220909%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1220909%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F140582%22%20target%3D%22_blank%22%3E%40Marco%20Brouwer%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%20Marco%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20are%20experiencing%20the%20same%20issue%2C%20sometimes%20users%20disconnect%20randomly%20during%20the%20day%2C%20others%20can%20work%20without%20any%20issue.%20It%20looks%20like%20an%20issue%20with%20one%20or%20a%20couple%20of%20RD%20Gateway%20servers%20serving%20multiple%20customers%20in%20West-EU%20WVD%20infrastructure%20.%20I%20cannot%20find%20any%20information%20however%20in%20the%20logs%20of%20the%20WVD%20hosts%20themselves.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHas%20anyone%20had%20luck%20with%20Microsoft%20Support%20going%20through%20this%20issue%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks%20in%20advance%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBest%20regards%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221872%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221872%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F468579%22%20target%3D%22_blank%22%3E%40knowlite%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%2C%20thanks%20for%20the%20response!%20I%20really%20wonder%20if%20we%20experience%20the%20problems%20at%20the%20same%20time.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EOur%20experience%3A%3C%2FP%3E%3CP%3E-%20Friday%20March%206th%3A%20No%20problems%3C%2FP%3E%3CP%3E-%20Monday%209th%3A%20Over%2075%20disconnects%20(16%20concurrent%20users)%3C%2FP%3E%3CP%3E-%20Yesterday%3A%20No%20problems%3C%2FP%3E%3CP%3E-%20Today%3A%20No%20problems%20(yet).%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ECan%20you%20run%20this%3A%3C%2FP%3E%3CP%3EImport-Module%20-Name%20AzureAD%3CBR%20%2F%3EImport-Module%20-Name%20Microsoft.RDInfra.RDPowerShell%3CBR%20%2F%3EAdd-RdsAccount%20-DeploymentUrl%20%3CA%20href%3D%22https%3A%2F%2Frdbroker.wvd.microsoft.com%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Frdbroker.wvd.microsoft.com%3C%2FA%3E%3C%2FP%3E%3CP%3EGet-RdsDiagnosticActivities%20-TenantName%20%26lt%3B%3CTENANTNAME%3E%26gt%3B%20-Outcome%20Failure%20-Detailed%20-Type%20Connection%20-starttime%2007-03-2020%20%7C%20Sort-Object%20endtime%20%7C%20ft%3C%2FTENANTNAME%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThis%20shows%20you%20when%20the%20disconnects%20took%20place.%20Maybe%20it%20happens%20exactly%20the%20same%20time%2C%20which%20proves%20something%20was%20wrong%20in%20Azure%20network%20%2F%20WVD%20gateway%20%2F%20WVD%20load%20balancer%20or%20broker.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EPlease%20let%20me%20know!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221905%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221905%22%20slang%3D%22en-US%22%3E%3CP%3EWe%20had%20monday%20also%20problems%2C%20between%209%3A30%20-%2011%3A30%20(Dutch%20time).%20Last%202%20days%20till%20this%20moment%20no%20problems.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221916%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221916%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F303257%22%20target%3D%22_blank%22%3E%40swalra%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%2C%20let's%20get%20a%20bit%20more%20precise.%20Last%20monday%2C%20the%20disconnects%20happened%20in%20batches.%20Like%20multiple%20%2F%20all%20users%20disconnecting%20at%20once.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EJust%20got%20into%20the%20logs%2C%20we%20see%20this%20disconnect%20waves%20at%20these%20times%20(according%20to%20Powershell%20command%20result)%3A%3C%2FP%3E%3CP%3E-%2011%3A03%3A19%20AM%3C%2FP%3E%3CP%3E-%2011%3A03%3A22%20AM%3C%2FP%3E%3CP%3E-%2011%3A12%3A20%20AM%3C%2FP%3E%3CP%3E-%2011%3A24%3A29%20AM%3C%2FP%3E%3CP%3E-%2011%3A37%3A52%20AM%3C%2FP%3E%3CP%3E-%2013%3A45%3A18%20PM%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBetween%20these%20times%2C%20we%20also%20has%20multiple%20disconnects%20for%20just%20one%20user%20at%20a%20time.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20really%20wonder%20if%20you%20see%20the%20exact%20same%20times%20of%20disconnect%20waves.%20That%20would%20defenitely%20prove%20it's%20something%20in%20the%20WVD%20plane.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221919%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221919%22%20slang%3D%22en-US%22%3EHi%2C%3CBR%20%2F%3E%3CBR%20%2F%3EI've%20ran%20the%20query%20and%20indeed%2099%25%20of%20the%20Failures%20are%20on%209%2F6%2C%20customer%20gave%20feedback%20that%20yesterdag%20and%20today%20go%20without%20any%20issue.%3CBR%20%2F%3EOf%20course%20not%20all%20errors%20may%20be%20related%20to%20the%20connection%20issue.%20If%20I%20do%20the%20query%20on%20Outcome%20Failure%20from%206-3%20until%20today%3A%3CBR%20%2F%3E6-3%3A%203%20events%3CBR%20%2F%3E9-3%3A%20107%20events%3CBR%20%2F%3E10-3%3A%2020%20event%3CBR%20%2F%3E11-3%3A%201%20event%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221935%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221935%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F468579%22%20target%3D%22_blank%22%3E%40knowlite%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThat%20seems%20to%20be%20a%20pattern%20%3A).%20These%20are%20our%20results.%3C%2FP%3E%3CP%3ENo%20problems%20today%20and%20yesterday.%3C%2FP%3E%3CP%3EWas%2026th%20of%20February%20also%20a%20%22bad%20day%22%20for%20you%3F%3C%2FP%3E%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%222020-03-11_11-39-14.png%22%20style%3D%22width%3A%20215px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Fgxcuf89792.i.lithium.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F176337i4D8BB26FEBC8374A%2Fimage-size%2Flarge%3Fv%3D1.0%26amp%3Bpx%3D999%22%20title%3D%222020-03-11_11-39-14.png%22%20alt%3D%222020-03-11_11-39-14.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%3CDIV%20class%3D%22mceNonEditable%20lia-copypaste-placeholder%22%3E%26nbsp%3B%3C%2FDIV%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1221990%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1221990%22%20slang%3D%22en-US%22%3EIf%20i%20use%20the%20following%20query%3A%3CBR%20%2F%3EGet-RdsDiagnosticActivities%20-TenantName%20TenantName%20-ActivityId%20ActivityId%20-Detailed%20%7C%20select-object%20*%20-ExpandProperty%20Errors%3CBR%20%2F%3E%3CBR%20%2F%3EI%20get%20the%20following%20errors%3A%3CBR%20%2F%3EErrorSource%20%3A%20RDGateway%3CBR%20%2F%3EErrorOperation%20%3A%20GatewayConnectionActive%3CBR%20%2F%3EErrorCode%20%3A%20-2146233088%3CBR%20%2F%3EErrorCodeSymbolic%20%3A%20ConnectionFailedServerDisconnect%3CBR%20%2F%3EErrorMessage%20%3A%20One%20or%20more%20errors%20occurred.%20(StartServerConnectionAsync)%3CBR%20%2F%3EErrorInternal%20%3A%20False%3CBR%20%2F%3EReportedBy%20%3A%20RDGateway%3CBR%20%2F%3E%3CBR%20%2F%3EErrorSource%20%3A%20RDStack%3CBR%20%2F%3EErrorOperation%20%3A%20ConnectionEstablished%3CBR%20%2F%3EErrorCode%20%3A%2014%3CBR%20%2F%3EErrorCodeSymbolic%20%3A%20UnexpectedNetworkDisconnect%3CBR%20%2F%3EErrorMessage%20%3A%20Unexpected%20network%20disconnect%3CBR%20%2F%3EErrorInternal%20%3A%20False%3CBR%20%2F%3EReportedBy%20%3A%20RDStack%3CBR%20%2F%3E%3CBR%20%2F%3EErrorSource%20%3A%20Client%3CBR%20%2F%3EErrorOperation%20%3A%20ClientGatewayNetworkConnection%3CBR%20%2F%3EErrorCode%20%3A%2050331694%3CBR%20%2F%3EErrorCodeSymbolic%20%3A%20PROXY_ERR_NETWORKDOWN%3CBR%20%2F%3EErrorMessage%20%3A%20AutoReconnect%20due%20to%20Network%20Error%3CBR%20%2F%3EErrorInternal%20%3A%20False%3CBR%20%2F%3EReportedBy%20%3A%20Client%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1222112%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1222112%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F140582%22%20target%3D%22_blank%22%3E%40Marco%20Brouwer%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ESame%20result%20here.%20Multiple%20WVD%20deployments%20all%20reported%20connection%20loss%20at%20the%20same%20time%20of%20yours.%20Like%20yourself%20they're%20also%20WVD%20Deployments%20in%20the%20Europa%20West%20region%20so%20I%20figure%20it%20was%20a%20local%20and%20temporary%20issue.%20That's%20what%20you%20get%20for%20outsourcing%20the%20Gateway%20I%20guess%2C%20but%20some%20sort%20of%20health%20monitoring%20would%20be%20a%20good%20thing.%20At%20the%20very%20least%20we%20know%20the%20issue%20is%20out%20of%20our%20hands.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1222153%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1222153%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F449941%22%20target%3D%22_blank%22%3E%40XxArkayxX%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDid%20anyone%20also%20experience%20that%20much%20disconnects%20on%2026%20February%20like%20we%20did%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EGreets%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1222156%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1222156%22%20slang%3D%22en-US%22%3EAround%2040-50%20errors%20that%20day%20here.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1222173%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1222173%22%20slang%3D%22en-US%22%3EDoes%20anyone%20already%20have%20a%20case%20running%20at%20Microsoft%20for%20this%20issue%3F%20Maybe%20we%20can%20contact%20them%2C%20and%20point%20out%20this%20%22multi-customer%20issue%22%2C%20which%20might%20give%20it%20more%20priority%20%2F%20urgency.%3CBR%20%2F%3E%3CBR%20%2F%3EAll%20we%20get%20from%20MS%20is%20send%20logs%20from%20clients%20and%20from%20WVD%20hosts%2C%20but%20for%20me%20it's%20clear%20that%20the%20gateway%20or%20something%20else%20in%20Azure%20is%20te%20problem.%20Maybe%2011-12AM%20are%20peak%20hours%20in%20some%20timezone%3F%3CBR%20%2F%3E%3CBR%20%2F%3EAlthough%20we%20run%20in%20West%20Europe%2C%20part%20of%20the%20Azure%20gateway%20services%20are%20located%20in%20East%20US%2C%20handling%20request%20for%20the%20whole%20world.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1222200%22%20slang%3D%22en-US%22%3ERe%3A%20Consistent%20Thin%20Client%20Disconnection%20from%20WVD%20Pool%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1222200%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F140582%22%20target%3D%22_blank%22%3E%40Marco%20Brouwer%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%20Marco%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMy%20experience%20with%20MS%20Support%20isn't%20that%20great%20to%20be%20honest%20regarding%20issues%20like%20these.%20I%20don't%20even%20bother%20opening%20tickets%20about%20performance%20issues%20like%20these.%26nbsp%3B%20They%20had%20similar%20latency%20problems%20with%20Exchange%20Online%20during%20the%20same%20period.%20Most%20people%20wouldn't%20have%20noticed%20them%20very%20much%20with%20Outlook%20running%20in%20Cached%20Mode%20but%20we%20also%20run%20a%20200%2B%20user%20RDS%20deployment%20without%20Cached%20Mode%20enabled%20and%20they%20got%20hit%20hard%20during%20the%20same%20period.%20Given%20the%20reports%20of%20you%20two%20I'm%20writing%20it%20off%20as%20local%20issues%20in%20the%20EU%20West%20region.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EExperience%20seems%20to%20have%20improved%20a%20tad%20over%20the%20past%20day%20or%20two%20but%20that's%20not%20based%20on%20any%20actual%20data%20I've%20collected.%20I%20don't%20handle%20the%20frontline%20helpdesk%20myself%20but%20this%20is%20what%20they%20tell%20me.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Occasional Visitor

Hello we have been experiencing some random but consistent disconnects from our WVD Pool. We have roughly 10 users and have been getting different event viewer logs for when they disconnect. We have Thin Clients on Windows 10 version 1607. When the users disconnect it will happen multiple times per day, however some days they do not disconnect. Attached are the event viewer logs

74 Replies
Highlighted

We have the same random disconnects from a WVD Pool

Highlighted
Which Remote Desktop app version are you currently using? Had the same issue before with previous version.
Highlighted
Solution

@Sbarnet: Thank you for sharing the logs. Unfortunately there isn't enough information to debug the disconnects. I recommend to open a support case and provide activity IDs for the connections you see issues with. 

Highlighted

@Sbarnet Hi, we are also experiencing these issues for two weeks now (50 user environment).

- Happens from multiple locations (so independent of network or ISP).

- Happens on both Windows 10 and HTML5 clients.

- Seems to happen completely random.

- Problem so far not related to a particular WVD host in the pool.

- Some days no problems at all, the next day it could me 80+ disconnects on one day.

- It seems to be related to a session. When user A gets is in the morning, chances are high he will get it multiple time a day, until he completely logs off and on again, so a completely new sessions is build up.

 

Already have a Microsoft case running for this, but so far no solution, or not even a cause. I suspect Azure's WVD gateway, but can't prove this yet.

 

Does anyone else experience this in the West Europe region?

 

Greets, Marco

Highlighted

@Marco Brouwer We've been having the same issue for about a week and a half. We have about 100 users on WVD. Last week everyone got kicked off and unable to connect. I log them off via the WVDAdmin tool and they can get back in. 

 

Today we had a similar issue but only a had full of users got the sessions kicked off. I believe also this has to do with the WVD Session Gateway.

Highlighted

@mmarti1223 

 

Thanks for the heads up! We do not experience the same. When we are disconnected, users auto-reconnect, or they connect again to their session manually.

 

- Sometimes it happens just for one user.

- Sometimes it happens for multiple users simultaneously (but not all). Those users are not (always) on same WVD host, and not on same client network / ISP (we have a few clients connecting from a 4G LTE router to rule out the customers network or ISP).

- Today it happened for all logged users at once, for the first time.

 

What really sticks out, is that is seems connected to particular sessions. User A get kicks out today 4 times, but tomorrow he's fine. Instead, user B gets kicked out tomorrow multiple times.

 

We asked users to completely log off and and log back in again after experiencing a disconnect, that "seems" to help. They don't get kicked out again that day.

 

You almost should say that a "new" session is being brokered / load balancing through another server in the Azure gateway cluster, which does not have the problem of disconnecting users at random.

 

Do you have contact with Microsoft about this? Can you tell me what troubleshooting steps have been taken so far?

Highlighted

@Marco Brouwer 

 

Hi Marco,

 

We are experiencing the same issue, sometimes users disconnect randomly during the day, others can work without any issue. It looks like an issue with one or a couple of RD Gateway servers serving multiple customers in West-EU WVD infrastructure . I cannot find any information however in the logs of the WVD hosts themselves.

 

Has anyone had luck with Microsoft Support going through this issue?

 

Thanks in advance

 

Best regards,

 

Highlighted

@knowlite 

 

Hi, thanks for the response! I really wonder if we experience the problems at the same time.

 

Our experience:

- Friday March 6th: No problems

- Monday 9th: Over 75 disconnects (16 concurrent users)

- Yesterday: No problems

- Today: No problems (yet).

 

Can you run this:

Import-Module -Name AzureAD
Import-Module -Name Microsoft.RDInfra.RDPowerShell
Add-RdsAccount -DeploymentUrl https://rdbroker.wvd.microsoft.com

Get-RdsDiagnosticActivities -TenantName <<TENANTNAME>> -Outcome Failure -Detailed -Type Connection -starttime 07-03-2020 | Sort-Object endtime | ft

 

This shows you when the disconnects took place. Maybe it happens exactly the same time, which proves something was wrong in Azure network / WVD gateway / WVD load balancer or broker.

 

Please let me know!

Highlighted

We had monday also problems, between 9:30 - 11:30 (Dutch time). Last 2 days till this moment no problems.

Highlighted

@swalra 

 

Hi, let's get a bit more precise. Last monday, the disconnects happened in batches. Like multiple / all users disconnecting at once.

 

Just got into the logs, we see this disconnect waves at these times (according to Powershell command result):

- 11:03:19 AM

- 11:03:22 AM

- 11:12:20 AM

- 11:24:29 AM

- 11:37:52 AM

- 13:45:18 PM

 

Between these times, we also has multiple disconnects for just one user at a time.

 

I really wonder if you see the exact same times of disconnect waves. That would defenitely prove it's something in the WVD plane.

Highlighted
Hi,

I've ran the query and indeed 99% of the Failures are on 9/6, customer gave feedback that yesterdag and today go without any issue.
Of course not all errors may be related to the connection issue. If I do the query on Outcome Failure from 6-3 until today:
6-3: 3 events
9-3: 107 events
10-3: 20 event
11-3: 1 event
Highlighted

@knowlite 

 

That seems to be a pattern :). These are our results.

No problems today and yesterday.

Was 26th of February also a "bad day" for you?

2020-03-11_11-39-14.png

 
Highlighted
If i use the following query:
Get-RdsDiagnosticActivities -TenantName TenantName -ActivityId ActivityId -Detailed | select-object * -ExpandProperty Errors

I get the following errors:
ErrorSource : RDGateway
ErrorOperation : GatewayConnectionActive
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedServerDisconnect
ErrorMessage : One or more errors occurred. (StartServerConnectionAsync)
ErrorInternal : False
ReportedBy : RDGateway

ErrorSource : RDStack
ErrorOperation : ConnectionEstablished
ErrorCode : 14
ErrorCodeSymbolic : UnexpectedNetworkDisconnect
ErrorMessage : Unexpected network disconnect
ErrorInternal : False
ReportedBy : RDStack

ErrorSource : Client
ErrorOperation : ClientGatewayNetworkConnection
ErrorCode : 50331694
ErrorCodeSymbolic : PROXY_ERR_NETWORKDOWN
ErrorMessage : AutoReconnect due to Network Error
ErrorInternal : False
ReportedBy : Client
Highlighted

@Marco Brouwer 

 

Same result here. Multiple WVD deployments all reported connection loss at the same time of yours. Like yourself they're also WVD Deployments in the Europa West region so I figure it was a local and temporary issue. That's what you get for outsourcing the Gateway I guess, but some sort of health monitoring would be a good thing. At the very least we know the issue is out of our hands.

Highlighted

@XxArkayxX 

 

Hi,

 

Did anyone also experience that much disconnects on 26 February like we did?

 

Greets

Highlighted
Around 40-50 errors that day here.
Highlighted
Does anyone already have a case running at Microsoft for this issue? Maybe we can contact them, and point out this "multi-customer issue", which might give it more priority / urgency.

All we get from MS is send logs from clients and from WVD hosts, but for me it's clear that the gateway or something else in Azure is te problem. Maybe 11-12AM are peak hours in some timezone?

Although we run in West Europe, part of the Azure gateway services are located in East US, handling request for the whole world.
Highlighted

@Marco Brouwer 

 

Hi Marco,

 

My experience with MS Support isn't that great to be honest regarding issues like these. I don't even bother opening tickets about performance issues like these.  They had similar latency problems with Exchange Online during the same period. Most people wouldn't have noticed them very much with Outlook running in Cached Mode but we also run a 200+ user RDS deployment without Cached Mode enabled and they got hit hard during the same period. Given the reports of you two I'm writing it off as local issues in the EU West region.

 

Experience seems to have improved a tad over the past day or two but that's not based on any actual data I've collected. I don't handle the frontline helpdesk myself but this is what they tell me.

Highlighted

Hallo,

 

Several users (internal, external) had some performance issue between 11:00 and now (11:45). Anyone also expires performance issues at that moments?  There where no disconnects at that moment.

Highlighted

@swalra 

 

Hi, yes screen performance seems a bit slow last hours. Like dragging windows over the screen, it shocks. Seems a connection lag, not a CPU problem or something like that. So WVD Gateway again I guess.

 

Also over here no disconnects yet today!

 

A reaction from Microsoft we got today:
"we have some disconnection issues but the Product group didn’t provide any updates regard changes but all we know that they are investigating and should provide a fix."

 

Well, if they want to provide a fix, at least they acknowledge something is broken :).

Highlighted

@swalra 

 

Same here.

 

Altough the problem seems widespread in EU West it isn't the case for all WVD connections. Our own internal WVD works fine and in one client we're even seeing some users with 4 bars of WVD connectivity and decent (not great) performance while others in the same office are experiencing 1 bar and piss-poor performance. FYI, standard RDP connectivity remains normal at those times. It's 100% an issue with (some of?) their WVD Gateway Entrypoints in EU West

Highlighted

@XxArkayxX 

 

Did anyone maybe try to make a second WVD tenant within the same Azure (onmicrosoft.com) tenant? Would that make a difference?

 

I really wonder how MS configures this. It seems some customers are really assigned to certain gateways (clusters).

 

 

Highlighted

@Marco Brouwer 

 

I doubt that would make for a good testcase. We have different behaviour within a single tenant so my guess at the moment would be that the load balancing mechanism isn't based on tenancy.

 

They do need some sort of health monitoring mechanism though. It's bad enough that we have to tell our customers it's an Azure problem and out of our hands but not even having a clear monitoring mechanism other the this forum to determin if it is Azure-side is worrysome.

 

Highlighted

Apparently they are aware of issues in EU West with Recovery Vault. The problem is just more widespread as they know/are acknowledging. Or their attempts at fixing the issue is causing issues elsewhere in the same datacenter.

 

https://app.azure.com/h/9SX1-198/acdc45

Highlighted
Hi, we see the health issue to in our portal. At least something is happening :)
Highlighted

@Marco Brouwer 

 

How do you see the health issues? 

swalra_1-1584016773700.png

 

 

Highlighted
Hi,

They should be in your screenshot. Log in to Azure portal, go to https://portal.azure.com/#blade/Microsoft_Azure_Health/AzureHealthBrowseBlade/serviceIssues

Seems that you don't have the issue :) Are you using Azure Backup Vault?
Highlighted

@swalra 

 

Your filter is too narrow. On your screen you can click through to the only active Health Issue at the moment: the Recovery Vault one. 

 

I logged a case for the WVD performance degradation as well. I'd suggest we all report them. If enough people complain they will create a Health issue for it I guess. Given that the problems have been happening for more then a week now and they're still not "aware" of the issue it's the only way forward I see.

Highlighted

The Backup health case has been "resolved" but the latency issues with WVD remain.

 

I have an open ticket but awaiting feedback. If you run https://azure.microsoft.com/en-us/services/virtual-desktop/assessment/  it sometimes peaks over 100 ms but I'm not sure if that's a valid test. Would be nice to get some sort of historic graph on that. I assume it just pings an end-point so we could do it ourselves but haven't found what exact end-point to ping yet.

Highlighted

Hello,

 

this morning we had again performance issues, freezes and disconnects. 

Highlighted
Just got reply from MS:

PG said there were transient errors from Azure REDIS cache components, which caused users to be disconnected.

However they are working to improve robustness on external dependencies failures; so let’s monitor closely and see the status, if anything comes up let us know.
Highlighted

@swalra 

 

Today again performance issues in several enviroments :(

Highlighted

@oiab_nl 

 

Yep, today very bad performance.....

Highlighted

@swalra 

 

Likewise.

 

Just so we're all on the same page here. Everyone experiencing these issues are contacting infrastructure in the Europe West region?

 

Further breakdown. Users usually have 1-2 bars on their WVD connection while experiencing the issue. Although that's a poor metric ofcourse. Sometimes the latency is present with 4 bars as well. I'm 100% certain it's the gateway infrastructure though. If I RDP straght to the VM's I get 4 bars and no latency issues at all.

 

I have escalated the case (through Ingram) to severity A but still not even an acknowledgement from MS. Does anyone else have any open tickets with MS? Mine is: 120031223001948 if you want to link them.

 

Highlighted

@XxArkayxX 

 

I will try to open a ticket and link.

 

My biggest clients are logging in now from a site 2 site vpn, then its ok.

Highlighted

Just got complaints from our client aswel, working with 9 per VM and feedback is slow.
Thinking about migrating them back to our own datacenter environment where we can troubleshoot the whole infrastructure.

Highlighted

just got off the line with a MS Engineer. He did little more then run Psping:

psping -t rdweb.wvd.microsoft.com:443

 

You see a lot of spikes depending on which site you run it from it gets worse and worse but they all experience the spiking. Naturally you don't see such spikes to other internet infrastructure to exclude the possibility of a local issue.

 

After seeing that he is escalating once more internally and will provide feedback.

 

I suggest we all try running these continuously and keep comparing. This is a screenshot of the current "performance" although I do have to mention it's a whole lot better then this morning.

 

WVD.JPG

Highlighted
They are doing DNS LB so your rdweb endpoint isn't the same as mine,our results will differ.
Your resolved IP address is different than mine, however the spikes are also there.

Is there a way to resolve the RDGW that the user is using at the time of issues?
Highlighted
Hi,

Our case number at Microsoft is 120022923000282 (via Ingram Micro though). They have just asked to run another network trace tool on one of the customers fat clients, but I refused.

I referred to this thread, and stated the issue is NOT in the customers network, but for sure somewhere in Azure WVD.

When I get more information, I'll post it here. Today I did not see any disconnects, and since I am not at the customer on site with all Corona madness going on, I can't really check for "slow sessions".
Highlighted

@Marco Brouwer 

Hi Marco,

 

I just tested more on our environment.

  1. RDP to APP server running in same vnet
    bandwidth 20mbit, fast speed, amazing!
    Annotation 2020-03-16 151553.jpg
  2. Then I logged in through Remote desktop app, I ran the tool from Bram Wolfs.
    This immediately gave a warning, high latency.
    Annotation 2020-03-16 151447.jpg

The only difference is that I am now going through MS RD Gateway infrastructure.

This is exactly the issue that our users are experiencing (I even think this is also related to the disconnect issue)

Highlighted
Hi,

We have 27 thin clients, and changed 12 of them to each connect directly to one of our 4 WVD hosts (so classic RDP, through site-to-site VPN to the WVD host).

Not a single session was disconnected, and users report less lag in their sessions when using this "direct RDP" workaround.

THE PROBLEM IS IN AZURE WVD GATEWAY.
Highlighted

@Pieter Wigleven (WINDOWS)@Marco Brouwer 

 

Can someone at MSFT pick up this issue please? This is not related to a customer specific.

Thank you in advance!

Highlighted

@knowlite We've had multiple reports regarding high latency in the europe region and are investigating. Thanks for letting us know. I will circle back to this thread once I know more.

Highlighted

Status update for everyone:

 

MS Engineer told me that the PG Team have applied a policy change to mittigate the latency issues we are seeing. They did ask us to reboot the WVD hosts to apply this change so he advises everyone on this thread to do so.

 

Afterwards the WVD deployments I rebooted have decent performance but then we had decent performance before the reboot as well given the time of the day. I allowed him to downgrade the issue to B status and we will check tomorrow morning if the problem persists or not.

 

He did also acknowledge that given the Corona outbreak the WVD infrastructure has a peak in usage and the infrastructure team are looking at improvements to assist in that area.

 

So I'm not sure if their fix actually resolves the issue or it's just a case of increasing capacity for WVD after increased usage the last weeks. If it's the second case I assume the problems will persist for some time. I can take single server customers to VPN-Vanilla RDP without too much changes but customers who require loadbalancing will need a special temp solution then :-S

Highlighted

@XxArkayxX @Pieter Wigleven (WINDOWS) 

Thanks Pieter for the confirmation, I will reboot our WVD hosts tonight to see if anything has improved tomorrow according to @XxArkayxX advice.

 

Highlighted

Tonight, our sessionshosts will be rebooted also. Hopes this helps......

Highlighted

@knowlite 

While I know it is after business hours and after rebooting the host. Current connection status:

Old:

Annotation 2020-03-16 151447.jpg

New:
Annotation 2020-03-16 193720.jpg

Highlighted

We have the same issue in Canada. Different customers, somes in Canada Central and Canada East. Both getting random but consistent disconnects. This is not related to a specific VM, i can RDP to 3389 without any disconnect the whole day and my customer complaining for multiple disconnect (over 5 per user per day) that can take 3 minutes up to 30 minutes to come back. When a disconnect occurs, it affect some users with the error 14 "Unexpected network disconnect" or "50331694 AutoReconnect due to Network Error" at the same time on different servers and differents start session time.

 

Do you have any ETA about the fix for this problem? Customers need to work remotely more than ever due to the current situation. The Azure gateway is very unreliable. Do you support a way to connect a Windows Server 2019 Gateway to WVD?

 

Highlighted
I already thought about setting up a RDS GW myself, and let that server broker / load balance / gateway the WVD sessions. Too bad Server 2019 GW does not support Windows 10 multi-session as workers for the farm.

The last few weeks I was checking for disconnects for this command:
Get-RdsDiagnosticActivities -Outcome Failure -starttime 13-03-2020 -detailed -tenant <<TENANTNAME>> | sort-object endtime | ft

Now it suddenly gives me this error:
Get-RdsDiagnosticActivities : There were no activities found.

Does anyone else have this too?