Consistent Thin Client Disconnection from WVD Pool

[Sbarnet]

Hello we have been experiencing some random but consistent disconnects from our WVD Pool. We have roughly 10 users and have been getting different event viewer logs for when they disconnect. We have Thin Clients on Windows 10 version 1607. When the users disconnect it will happen multiple times per day, however some days they do not disconnect. Attached are the event viewer logs

[#2]

We have the same random disconnects from a WVD Pool

[#3]

Which Remote Desktop app version are you currently using? Had the same issue before with previous version.

[#4 · Accepted solution]

@Sbarnet: Thank you for sharing the logs. Unfortunately there isn't enough information to debug the disconnects. I recommend to open a support case and provide activity IDs for the connections you see issues with. 

[#5]

@Sbarnet Hi, we are also experiencing these issues for two weeks now (50 user environment).

- Happens from multiple locations (so independent of network or ISP).

- Happens on both Windows 10 and HTML5 clients.

- Seems to happen completely random.

- Problem so far not related to a particular WVD host in the pool.

- Some days no problems at all, the next day it could me 80+ disconnects on one day.

- It seems to be related to a session. When user A gets is in the morning, chances are high he will get it multiple time a day, until he completely logs off and on again, so a completely new sessions is build up.

Already have a Microsoft case running for this, but so far no solution, or not even a cause. I suspect Azure's WVD gateway, but can't prove this yet.

Does anyone else experience this in the West Europe region?

Greets, Marco

[#6]

@Marco Brouwer We've been having the same issue for about a week and a half. We have about 100 users on WVD. Last week everyone got kicked off and unable to connect. I log them off via the WVDAdmin tool and they can get back in. 

Today we had a similar issue but only a had full of users got the sessions kicked off. I believe also this has to do with the WVD Session Gateway.

[#7]

@mmarti1223 

Thanks for the heads up! We do not experience the same. When we are disconnected, users auto-reconnect, or they connect again to their session manually.

- Sometimes it happens just for one user.

- Sometimes it happens for multiple users simultaneously (but not all). Those users are not (always) on same WVD host, and not on same client network / ISP (we have a few clients connecting from a 4G LTE router to rule out the customers network or ISP).

- Today it happened for all logged users at once, for the first time.

What really sticks out, is that is seems connected to particular sessions. User A get kicks out today 4 times, but tomorrow he's fine. Instead, user B gets kicked out tomorrow multiple times.

We asked users to completely log off and and log back in again after experiencing a disconnect, that "seems" to help. They don't get kicked out again that day.

You almost should say that a "new" session is being brokered / load balancing through another server in the Azure gateway cluster, which does not have the problem of disconnecting users at random.

Do you have contact with Microsoft about this? Can you tell me what troubleshooting steps have been taken so far?

[#8]

@Marco Brouwer 

Hi Marco,

We are experiencing the same issue, sometimes users disconnect randomly during the day, others can work without any issue. It looks like an issue with one or a couple of RD Gateway servers serving multiple customers in West-EU WVD infrastructure . I cannot find any information however in the logs of the WVD hosts themselves.

Has anyone had luck with Microsoft Support going through this issue?

Thanks in advance

Best regards,

[#9]

@knowlite 

Hi, thanks for the response! I really wonder if we experience the problems at the same time.

Our experience:

- Friday March 6th: No problems

- Monday 9th: Over 75 disconnects (16 concurrent users)

- Yesterday: No problems

- Today: No problems (yet).

Can you run this:

Import-Module -Name AzureAD
Import-Module -Name Microsoft.RDInfra.RDPowerShell
Add-RdsAccount -DeploymentUrl https://rdbroker.wvd.microsoft.com

Get-RdsDiagnosticActivities -TenantName <<TENANTNAME>> -Outcome Failure -Detailed -Type Connection -starttime 07-03-2020 | Sort-Object endtime | ft

This shows you when the disconnects took place. Maybe it happens exactly the same time, which proves something was wrong in Azure network / WVD gateway / WVD load balancer or broker.

Please let me know!

[#10]

We had monday also problems, between 9:30 - 11:30 (Dutch time). Last 2 days till this moment no problems.

[#11]

@swalra 

Hi, let's get a bit more precise. Last monday, the disconnects happened in batches. Like multiple / all users disconnecting at once.

Just got into the logs, we see this disconnect waves at these times (according to Powershell command result):

- 11:03:19 AM

- 11:03:22 AM

- 11:12:20 AM

- 11:24:29 AM

- 11:37:52 AM

- 13:45:18 PM

Between these times, we also has multiple disconnects for just one user at a time.

I really wonder if you see the exact same times of disconnect waves. That would defenitely prove it's something in the WVD plane.

[#12]

Hi,

I've ran the query and indeed 99% of the Failures are on 9/6, customer gave feedback that yesterdag and today go without any issue.
Of course not all errors may be related to the connection issue. If I do the query on Outcome Failure from 6-3 until today:
6-3: 3 events
9-3: 107 events
10-3: 20 event
11-3: 1 event

[#13]

@knowlite 

That seems to be a pattern :). These are our results.

No problems today and yesterday.

Was 26th of February also a "bad day" for you?

[#14]

If i use the following query:
Get-RdsDiagnosticActivities -TenantName TenantName -ActivityId ActivityId -Detailed | select-object * -ExpandProperty Errors

I get the following errors:
ErrorSource : RDGateway
ErrorOperation : GatewayConnectionActive
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedServerDisconnect
ErrorMessage : One or more errors occurred. (StartServerConnectionAsync)
ErrorInternal : False
ReportedBy : RDGateway

ErrorSource : RDStack
ErrorOperation : ConnectionEstablished
ErrorCode : 14
ErrorCodeSymbolic : UnexpectedNetworkDisconnect
ErrorMessage : Unexpected network disconnect
ErrorInternal : False
ReportedBy : RDStack

ErrorSource : Client
ErrorOperation : ClientGatewayNetworkConnection
ErrorCode : 50331694
ErrorCodeSymbolic : PROXY_ERR_NETWORKDOWN
ErrorMessage : AutoReconnect due to Network Error
ErrorInternal : False
ReportedBy : Client

[#15]

@Marco Brouwer 

Same result here. Multiple WVD deployments all reported connection loss at the same time of yours. Like yourself they're also WVD Deployments in the Europa West region so I figure it was a local and temporary issue. That's what you get for outsourcing the Gateway I guess, but some sort of health monitoring would be a good thing. At the very least we know the issue is out of our hands.

[#16]

@XxArkayxX 

Hi,

Did anyone also experience that much disconnects on 26 February like we did?

Greets

[#17]

Around 40-50 errors that day here.

[#18]

Does anyone already have a case running at Microsoft for this issue? Maybe we can contact them, and point out this "multi-customer issue", which might give it more priority / urgency.

All we get from MS is send logs from clients and from WVD hosts, but for me it's clear that the gateway or something else in Azure is te problem. Maybe 11-12AM are peak hours in some timezone?

Although we run in West Europe, part of the Azure gateway services are located in East US, handling request for the whole world.

[#19]

@Marco Brouwer 

Hi Marco,

My experience with MS Support isn't that great to be honest regarding issues like these. I don't even bother opening tickets about performance issues like these.  They had similar latency problems with Exchange Online during the same period. Most people wouldn't have noticed them very much with Outlook running in Cached Mode but we also run a 200+ user RDS deployment without Cached Mode enabled and they got hit hard during the same period. Given the reports of you two I'm writing it off as local issues in the EU West region.

Experience seems to have improved a tad over the past day or two but that's not based on any actual data I've collected. I don't handle the frontline helpdesk myself but this is what they tell me.

[#20]

Hallo,

Several users (internal, external) had some performance issue between 11:00 and now (11:45). Anyone also expires performance issues at that moments?  There where no disconnects at that moment.