WVD outage / hosts unavailable with status "NoHeartbeat"

[Nicholas Semenkovich]

Last night, our WVD instance became unavailable, with clients unable to connect. The initial error is "ConnectionFailedNoHealthyRdshAvailableErrorMessage", and looks like the RD server / endpoint is now in status "NoHeartbeat"

However the endpoint is up & healthy when we connect directly in Azure / via tunneled RDP. 

The RdAgentBootloader & RDAgent services are running. We've tried restarting the machine & those services, with no change in the status.

SessionHostName    : ***
TenantName         : ***
TenantGroupName    : Default Tenant
GroupHostPoolName  : ***
AllowNewSession    : True
Sessions           : 1
LastHeartBeat      : 11/25/2019 3:46:07 AM
AgentVersion       : 1.0.1534.2000
AssignedUser       :
OsVersion          : 10.0.18363
SxSStackVersion    : rdp-sxs190927002
Status             : NoHeartbeat
UpdateState        : Succeeded
LastUpdateTime     : 11/19/2019 12:06:32 PM
UpdateErrorMessage :

Any idea how we can debug this further?

[#2]

FWIW we've followed the simple steps here without any luck: https://docs.microsoft.com/en-us/azure/virtual-desktop/troubleshoot-vm-configuration#error--windows-...

• RDAgentBootLoader is running; and nothing happens if we restart it or the machine
  • The Event Viewer shows some RD Agent logs; but they all seem to be healthy / info messages -- and are similar to those before we stopped getting heartbeats
• psping to test port 443 to the WVD selfhost url works fine (psping.exe rdbroker.wvdselfhost.microsoft.com:443)
• qwinsta seems to show everything working & awaiting a connection:

  • PS C:\> qwinsta
    SESSIONNAME USERNAME ID STATE TYPE DEVICE
    services 0 Disc
    console 1 Conn
    >rdp-tcp#1 nick 2 Active
    31c5ce94259d4... 65536 Listen
    rdp-tcp 65537 Listen
    rdp-sxs190927002 65538 Listen

[#3]

@Nicholas Semenkovich 

We experienced the same issue today. Came in this morning and 3 session hosts had a status of NoHeartbeat. Restarting the services/session host had no effect. Only way to fix it was to remove the session hosts from their pools and re-add them.

Happened again to a 4th session host this afternoon.

I've raised a ticket with Microsoft and sent them some logs.

[#4]

@DanRobb Thanks!

When you say re-add session hosts (haven't done that before), do you re-install the agent and use New-RdsRegistrationInfo & Export-RdsRegistrationInfo ?

[#5 · Accepted solution]

@Nicholas Semenkovich 

These are the steps I followed:

1. Remove the session host from the host pool:

Remove-RdsSessionHost -TenantName [TenantName] -HostPoolName [HostPoolName] -Name [SessionHostName] -Force

2. Get the host pool registration token (replace with New-RdsRegistrationInfo if the token has already expired)

Export-RdsRegistrationInfo -TenantName[TenantName] -HostPoolName [HostPoolName] | Select -Expand Token

3. Log into the session host and update the following registry keys:

Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RDInfraAgent

Name: RegistrationToken

Value: Token Goes Here

Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RDInfraAgent

Name: IsRegistered

Value: 0

4. Restart the RDAgentBootLoader service

[#6]

@DanRobb Worked perfectly -- many thanks! 

[#7]

@DanRobb 

Nice one, I had gone for the more manual method of re-installing the agent. This option is much better, thanks!

I've got a support case open with MS as well, hopefully they will come back with something soon.

Ben

[#8]

Perfect guys! Thanks for this smart solution.