RD Session Host stuck at Status: "Upgrading"

Brass Contributor
Hi, I am trying to add a Windows Server 2016 as a session host. I have followed the steps at: https://docs.microsoft.com/en-us/azure/virtual-desktop/create-host-pools-powershell and it shows up as a session host with PS with the Get-RdsSessionHost command, but the status in the output is stuck at Status : Upgrading. In the event logs in gives a bunch of errors, these are a couple of them: Events: Microsoft.RDInfra.Agent.Monitoring.MonitoringAgentInstallSettings Value of the 'AgentMonitoringAuthId' is empty Microsoft.RDInfra.Agent.Monitoring.MonitoringAgentInstallSettings RDInfraAgent key doesn't exist Unable to retrieve DefaultAgent from registry: System.NullReferenceException RDAgentBootLoader.BootLoaderSettings.get_DefaultAgentPath() i C:\agent\_work\1\s\src\RDAgent\src\RDAgentBootLoader\BootLoaderSettings.cs:rad 82
17 Replies

@andersidahl: Can you please re-run the template and create a new host pool? Delete the existing one it got into a bad state. If this happens again please let us know as we would like to get into the root cause of those issues.

@Eva Seydl Hi Eva, thanks for reaching out.

 

I am not deploying with a template, that one works fine. I am trying to deploy to a Windows Server 2016 previously deployed.

 

The steps done are from: https://docs.microsoft.com/en-us/azure/virtual-desktop/create-host-pools-powershell. Everything seems to go fine with all steps, except that we are unable to connect after doing such steps and also when running the Get-RdsSessionHost it shows the agent as still updating. On the server its also some events which shows some error, I could send that directly if you have a secure drop for that?

 

Best regards, Anders

@andersidahl 

 

I am in the same boat. Everything works fine with the template but I would like to be able to automate the creation of my own Session Hosts and can get everything moving to the point of where the RD Session Host is stuck with the Status: Upgrading. Status remained the same overnight.

 

Anyway we can check what could be failing. RDSDiagnosticActivities does not seem to have anything regarding the failing of that registration process.

Checking Event Viewer, I see that the Remote Desktop Services SxS Network Stack is stuck in some sort of loop where the Session Host keeps re-running the MSI installer and re-configures the installed RDS SxS Network Stack product every 5 minutes:

 

Log Name: Application
Source: Microsoft-Windows-RestartManager
Date: 3/28/2019 2:32:50 PM
Event ID: 10001
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: wvdhost.test.com
Description:
Ending session 0 started ‎2019‎-‎03‎-‎28T14:32:48.506934400Z.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-RestartManager" Guid="{}" />
<EventID>10001</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:50.599897100Z" />
<EventRecordID>552</EventRecordID>
<Correlation />
<Execution ProcessID="5600" ThreadID="2720" />
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<UserData>
<RmSessionEvent xmlns="http://www.microsoft.com/2005/08/Windows/Reliability/RestartManager/">
<RmSessionId>0</RmSessionId>
<UTCStartTime>2019-03-28T14:32:48.506934400Z</UTCStartTime>
</RmSessionEvent>
</UserData>
</Event>

Log Name: Application
Source: MsiInstaller
Date: 3/28/2019 2:32:50 PM
Event ID: 1042
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: wvdhost.test.com
Description:
Ending a Windows Installer transaction: C:\Program Files\Microsoft RDInfra\SxSStack-1.0.1902.17001.msi. Client Process Id: 4400.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MsiInstaller" />
<EventID Qualifiers="0">1042</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:50.622450000Z" />
<EventRecordID>551</EventRecordID>
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<EventData>
<Data>C:\Program Files\Microsoft RDInfra\SxSStack-1.0.1902.17001.msi</Data>
<Data>4400</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>
</Data>
</EventData>
</Event>

Log Name: Application
Source: MsiInstaller
Date: 3/28/2019 2:32:50 PM
Event ID: 1035
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: wvdhost.test.com
Description:
Windows Installer reconfigured the product. Product Name: Remote Desktop Services SxS Network Stack. Product Version: 1.0.1902.17001. Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 0.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MsiInstaller" />
<EventID Qualifiers="0">1035</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:50.591183400Z" />
<EventRecordID>550</EventRecordID>
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<EventData>
<Data>Remote Desktop Services SxS Network Stack</Data>
<Data>1.0.1902.17001</Data>
<Data>1033</Data>
<Data>0</Data>
<Data>Microsoft Corporation</Data>
<Data>(NULL)</Data>
<Data>
</Data>
<Binary></Binary>
</EventData>
</Event>

Log Name: Application
Source: MsiInstaller
Date: 3/28/2019 2:32:50 PM
Event ID: 11728
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: wvdhost.test.com
Description:
Product: Remote Desktop Services SxS Network Stack -- Configuration completed successfully.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MsiInstaller" />
<EventID Qualifiers="0">11728</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:50.575597200Z" />
<EventRecordID>549</EventRecordID>
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<EventData>
<Data>Product: Remote Desktop Services SxS Network Stack -- Configuration completed successfully.</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>
</Data>
<Binary></Binary>
</EventData>
</Event>

Log Name: Application
Source: Microsoft-Windows-RestartManager
Date: 3/28/2019 2:32:48 PM
Event ID: 10000
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: wvdhost.test.com
Description:
Starting session 0 - ‎2019‎-‎03‎-‎28T14:32:48.506934400Z.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-RestartManager" Guid="{}" />
<EventID>10000</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:48.518299200Z" />
<EventRecordID>548</EventRecordID>
<Correlation />
<Execution ProcessID="5600" ThreadID="6308" />
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<UserData>
<RmSessionEvent xmlns="http://www.microsoft.com/2005/08/Windows/Reliability/RestartManager/">
<RmSessionId>0</RmSessionId>
<UTCStartTime>2019-03-28T14:32:48.506934400Z</UTCStartTime>
</RmSessionEvent>
</UserData>
</Event>

Log Name: Application
Source: MsiInstaller
Date: 3/28/2019 2:32:48 PM
Event ID: 1040
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: wvdhost.test.com
Description:
Beginning a Windows Installer transaction: C:\Program Files\Microsoft RDInfra\SxSStack-1.0.1902.17001.msi. Client Process Id: 4400.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MsiInstaller" />
<EventID Qualifiers="0">1040</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2019-03-28T14:32:48.491308300Z" />
<EventRecordID>547</EventRecordID>
<Channel>Application</Channel>
<Computer>wvdhost.test.com</Computer>
<Security UserID="" />
</System>
<EventData>
<Data>C:\Program Files\Microsoft RDInfra\SxSStack-1.0.1902.17001.msi</Data>
<Data>4400</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>(NULL)</Data>
<Data>
</Data>
</EventData>
</Event>

 Any thoughts?

Same problem here. It seems like the agent is upgrading immediately after installing. I now have two versions of the agent installed.

I tried uninstalling the agent and re-installing using what looks like an upgraded installer (version 1.0.1.10) but now instead of Status: Upgrading, my RD host is in Status: Unavailable.

 

Also have these additional errors in the RemoteDesktopServices-MonitoringLogger and RemoteDesktopServices-MonitoringAgentLoader logs respectively

 

Microsoft.RDInfra.Monitoring.AgentMonitoringService.AgentMonitoringServiceWorker
Value of the 'AgentMonitoringAccount' in 'SOFTWARE\Microsoft\RDMonitoringAgent' is empty

-------

Microsoft.RDInfra.Monitoring.AgentMonitoringService.AgentMonitoringServiceWorker
Fail to set default dimensions Invalid Operation: Fail to set default dimensions, error code is 2147942487.. Error code is 80070057

@andersidahl 

Hi Anders,

 

The above exception, to me, indicates that the bootloader cannot find the agent because the required location is missing a registry key (Unable to retrieve DefaultAgent path from registry).

 

After installing the WVD Agent - it should have declared itself the default agent by creating this key in the AgentBootLoader regkey location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RDAgentBootLoader -> DefaultAgent

 

Can you verify that this key exists? Also please make sure you stop the AgentBootLoader service before trying to install the WVD Agent MSI.

 

thanks,

-al

 

 

 

@smithanc 

Hi,

 

Can you verify if the reverse connect listener is up and running? On the VM that is stuck upgrading open a cmd window and run 'Qwinsta'  you should see <SESSIONNAME> rdp-sxs190217001 <id> 655.. <STATE>Listen

 

if you do - typically rebooting the VM can help work around this.

 

Thanks,

-al

@Stephen Ferrero 

Hi Stephen,

 

The monitoring agent is reporting this because there was a failure onboarding - please stop the bootloader service and try re-installing the WVD Agent. Verify that the DefaultAgent key exists under

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RDAgentBootLoader -> DefaultAgent.

 

Then start the bootloader service.

 

thanks,

-al

 

@alhenriq_ms 

 

Hi, I can verify that this was already tested. I also tried pulling the newer versjon setup from another server-setup that worked and installing manually and changing the registerkey, but it kept reconfiguring back to 1.6/1.7 i believe the version was. Also a lot of reboots inbetween and waiting, though it kept going back to Upgrading.

@andersidahl 

 

Ok I misunderstood, I thought you had issues with the bootloader loading the agent. I can check your logs from my end, if you like. You can send me your VM name (FQDN helps)  via Private Message.

 

thanks,

-al

@alhenriq_ms 

@andersidahl 

 

It looks like the RDAgent on the doc for manually configuring is different than the version being pushed using the deployment for a host pool. The downloaded agent version appears to be 1.0.0.1462 and the version of the agent on the provisioned host pools is 1.0.1.10. It seems to upgrade it automatically but never complete for one reason or another.

 

The boot loader and sxs installers don't have version numbers in them that I can find to confirm they are the same.

 

Is there a download link for the correct agent version?

Hi @aspicola2605 

 

You can determine the versions via control panel "uninstall or change a program"

As of now the current versions pushed are:

The bootloader at this time should report version "1.0.0.0"

"Remote Desktop Services Sxs Network Stack" - "1.0.1902.17001"

"Remote Desktop Services Infrastructure Agent" - "1.0.1.10"

 

I am interested why the agent is failing the upgrade can you forward me your VM FQDN (private message)

 

thanks,

-al

 

@alhenriq_ms 

 

I know the versions. What I'm saying is the downloadable version in the MS docs is wrong which is probably why it keeps upgrading and getting stuck.

 

I'd rather not provide my FQDN because it's part of my companies network.

@alhenriq_ms 

 

Thanks for the response! I had a look at the "Qwinsta" (result below) and saw that the SXS was running, so I rebooted the server and waited 15 minutes with no change.

 

SESSIONNAME USERNAME ID STATE TYPE DEVICE
services 0 Disc
console 1 Conn
>rdp-tcp#1  2 Active
31c5ce94259d4... 65536 Listen
rdp-tcp 65537 Listen
rdp-sxs##### 65538 Listen

 

Still have the same behavior of SXS seeming like its trying to reinstall every 5 minutes with regards to entries in the Event Viewer.

Cheers

@alhenriq_ms 

Thanks,

 

I seem to be OK now after a complete removal and re-install. I see the agent version incremented to 1.0.1.18 so perhaps something has changed.

 

I do still see duplicated entries in appwiz for the RDS Infrastructure Agent. Looks like the original version I installed, and the updated agent are both still installed.

 

This garbage still happens even today. At least every other day to one of our hosts. We're running the newer VMs generations too - D16sv4's. Sometimes it'll upgrade with people connected and just create so many headaches, or it'll upgrade and get stuck and then nobody can log into it, but it registers as open sessions so my auto-scale logic doesn't open up more machines and again, creates issues. Figure it out...
Please reach out to support for debugging. THank you