An internal error occurred / can't connect anymore

[gdglee]

We've been running WVD succesfully for a few months, but since this morning my users cannot login anymore. Not through the desktop and not through the webclient. The desktop client returns 'An internal error occurred' after logging in. The webclient has a cannot connect message (in Dutch). I've traced the connection failure via Powershell to below two errors. It seems my SID has changed somehow, but I cannot understand how:

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : OrchestrateAsync: SID value in the database is different than the value returned in the orchestration reply from the agent for user ≤PRIVATE≥ with Id PRIVATE. This scenario is not supported - we will not be able to redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 28-6-2019 12:42:44

ErrorSource : Client
ErrorOperation : ClientRDPConnect
ErrorCode : 2147965400
ErrorCodeSymbolic :
ErrorMessage : Your computer can't connect to the Remote Desktop Gateway server. Contact your network administrator for assistance.
ErrorInternal : True
ReportedBy : Client
Time : 28-6-2019 12:42:44

I've tried removing an RdsAppGroupUser and re adding it, but the same error remains.

[#2]

@gdglee We've exactly this same issue, we tried to open a case, but it's not supported for preview services :(

[#3]

@P_haem good to know we're not the only ones.

[#4]

It's seem's to be cause by the agent upgrade (yesterday around 10PM for us) : New agent version : 1.0.833.4

[#5]

@P_haem So were you able to resolve?

[#6]

We are also experiencing this issue.@gdglee 

[#7]

@gdglee - Same thing on this end. Started happening around 2-3p EST on Friday 06/28. This host pool was perfectly fine prior to this. 

[#8]

We're having this issue as well. I can RDP into my WDV VMs, but trying to run published apps (browser or RDP shortcut) fail with a connection error.

[#9]

Same here. This seems to be a widespread problem. For us started last week on Thursday.

[#10]

@gdglee: thank you for reporting this.  We have introduced additional security checks when resolving user identities. In some environments this leads to restricted connectivity due to legacy set-ups. We are reviewing the issue and will update once a solution is available.

Please ensure that you follow as well the following best practice:

- Have a validation host pool set-up to escalate issues before they hit the majority of your users. 

- Set-up service alerts to receive health advisories and notification for your subscription.

[#11]

@gdglee 

Same here.

For us started last week on Friday.

[#12]

@Eva Seydl Thank you for your answer! Do you have any resolution date ? And is there any way to to go back to the previous version ?

[#13]

@Eva Seydl 

We are also having the same issue, Raised ticket to MS Support team, they said "We cannot not provide support for Windows Virtual Desktop because it’s in preview so unfortunately we cannot answer your questions, or assist with issues you are experiencing."

Also, i have received Service alert from Microsoft "Windows Virtual Desktop - East US 2 - Exploring Mitigation" But there is no ETA As of now.

[#14]

@Eva Seydl - so what do you suggest we do with host pools that are experiencing this issue in the meantime? We have a high visibility POC at a very large client that this is interrupting. Telling them their host pool is ruined  and needs to be redployed won't go over so well. 

Will redeploying even fix this, without intervention on your side? 

Additionally, can you provide some more detail on what these security checks are? Any details on what leads to this condition from a "legacy setup" perspective? 

[#15]

@Eva Seydl we're having the same issue as well on both existing host pools that were working properly and on a newly deployed host pool. 

[#16]

@Eva Seydl-  Would you do me a favor? We have redeployed 3 times to fix this, but didn't fix.....

[#17]

@Raja R 

Please set-up a validation pool as we have a fix deployed to the validation pools. Learn here how to set those up: https://docs.microsoft.com/en-us/azure/virtual-desktop/create-validation-host-pool

We recommend to make use of Azure Service Health Alerts where you will be notified when the fix is available for production: https://docs.microsoft.com/en-us/azure/virtual-desktop/set-up-service-alerts 

[#18 · Accepted solution]

Please set-up a validation pool as we have a fix deployed to the validation pools. Learn here how to set those up: https://docs.microsoft.com/en-us/azure/virtual-desktop/create-validation-host-pool We recommend to make use of Azure Service Health Alerts where you will be notified when the fix is available for production: https://docs.microsoft.com/en-us/azure/virtual-desktop/set-up-service-alerts

[#19]

Hi @Eva Seydl that is good to know. I changed my hostpool to a validationenvironment. The agent is upgraded: AgentVersion : 1.0.833.5

Unfortunately I see the same behaviour still. Not able to connect, same error.

[#20]

@Eva Seydl Unfortunately we cannot deploy to Validation ring because the machine fails to join the domain. We are using the same script as to deploy to prod ring (which has no issues to join the domain) with the only difference being the 

-ValidationEnv $True flag on the new-rdshostpool command.

15:10:25 - Resource Microsoft.Compute/virtualMachines/extensions 'rmrvwval4-0/joindomain' failed with message '{
"status": "Failed",
"error": {
"code": "ResourceDeploymentFailure",
"message": "The resource operation completed with terminal provisioning state 'Failed'.",
"details": [
{
"code": "VMExtensionProvisioningError",
"message": "VM has reported a failure when processing extension 'joindomain'. Error message: \"Exception(s) occured while joining Domain '{sanitized}.com'\"."
}
]
}
}'

[#21]

@Deleted Please review our troubleshooting section under https://aka.ms/wvdpreview. We have articles on that as well.

[#22]

@Eva Seydl I looked over the troubleshooting steps but none of it applies, we deploy to a new resource group in the same vnet, same region, same source image as our production ring. If I create the new host pool with -ValidationEnv $False flag then there is no problem with domain join, and the only issue is the SID error on connection.

[#23]

@Eva SeydlAwesome, it worked perfectly. 

Thanks for the quick fix.

[#24]

@Eva Seydl I created a new hostpool as validationenvironment, to see if that would make difference. The agent is 1.0.833.5 but the error remains the same. In an earlier reply you mentioned a legacy setup. Could you elaborate on that? What could we setup differently to make this work again?

Thanks, Gerrit

[#25]

We were able to set up the validation hostpool, but the internal error occurred message still pops up.@Eva Seydl 

[#26]

@Eva Seydl 

Maybe relevant to know from our setup. Users created in the Azure AD as cloud only are able to connect, users synchronized from our on-premise AD are not.

[#27]

@JanPijnacker @Eva Seydl  This is true for us as well.

[#28]

@Eva Seydl - anything new on this? All of my client's IDs are sourced from on-prem AD > AADC > AAD. The error remains on my existing pools, and based on the feedback here - suspect it will with a validation pool deployment. Moving to cloud only IDs isn't really an option 

Can you provide any more detail on what leads to this condition? 

[#29]

This started working for me yesterday without any intervention on our part. I did notice that the RDS Infrastructure Agent updated on 7/3/19 to 1.0.833.5

[#30]

@Eva Seydl This is still not working for us in either validation pool or production pool. These are the agent versions installed on my VM.

Remote Desktop Services Infrastructure Agent         Microsoft Corporation  1.0.833.5
Remote Desktop Services SxS Network Stack            Microsoft Corporation  1.0.1904.29002
Remote Desktop Agent Boot Loader                     Microsoft Corporation  1.0.0.0
Remote Desktop Services Infrastructure Geneva Agent  Microsoft Corporation  42.3.9
Remote Desktop Services Infrastructure Agent         Microsoft Corporation  1.0.0.1462

[#31]

@richiewrt I just had the same thing happen on this end. I'm super interested to see the RCA for this. 

[#32]

@Eva Seydl Any update?

[#33]

 @Eva Seydl 
For us its the same as for the others. Only some InCloud accounts are able to access Virtual Desktop. The validation pool is setup. InCloud as well as AAD synced Accounts get the following error:

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : OrchestrateAsync: SID value in the database is different than the value returned in the orchestration reply from the agent for user
≤Firstname.Surname@domain.com≥ with Id xxxxxxxx-yyyy-zzzz-xxxx-xxxxyyyyzzzz. This scenario is not supported - we will not be able to redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 11.07.2019 10:42:56 

[#34]

@Eva Seydl Is there any news on the fix? We're unfortunately dead in the water here with synced accounts.

[#35]

Please review our troubleshooting guide for domain join issues: https://docs.microsoft.com/en-us/azure/virtual-desktop/troubleshoot-set-up-issues

[#36]

Can I just move my pool to validation pool instead or cretaing new POOL , i found that I can switch the POOL to validation pool using command below Set-RdsHostPool -TenantName MVPPOC -Name POCHOST1 -ValidationEnv $true

[#37]

Do we know if there is a fix for this yet.  I have installed a new Validation Pool and still the same error.  My agent is 1.0.833.5

[#38]

@Eva Seydl It's been 2.5 weeks now. Can you get some ETA on the fix for this problem? Neither validation or production environment works!!

TenantGroupName : Default Tenant Group
HostPoolName : MyTest_HostPool
FriendlyName : My Test Host Pool
Description :
Persistent : False
CustomRdpProperty :
MaxSessionLimit : 999999
LoadBalancerType : BreadthFirst
ValidationEnv : True
Ring :

Still getting the same SID error:

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : OrchestrateAsync: SID value in the database is different than the value returned in the
orchestration reply from the agent for user ≤rhythmnewt@rhythmnewt.com≥ with Id
85a45a4c-413d-4074-2e41-08d6e4d9abe8. This scenario is not supported - we will not be able to
redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 7/16/2019 3:08:54 PM

User activity log

ActivityId : 10dd46a2-4836-49f1-8f89-face053b0000
ActivityType : Connection
StartTime : 7/16/2019 3:08:54 PM
EndTime : 7/16/2019 3:08:54 PM
UserName : rhythmnewt@rhythmnewt.com
RoleInstances : mrs-eus2r0c001-rdgateway-prod::RD2818785C114D;mrs-eus2r0c002-rdbroker-prod::RD2818788A0588;≤rmrvw-0
.rhythmnewt.com≥
Outcome : Failure
Status : Completed
Details : {[ClientOS, ], [ClientVersion, ], [ClientType, ], [PredecessorConnectionId, ]...}
LastHeartbeatTime : 7/16/2019 3:10:26 PM
Checkpoints : {LoadBalancedNewConnection, TransportConnecting, TransportConnected, RdpStackDisconnect...}
Errors : {Microsoft.RDInfra.Diagnostics.Common.DiagnosticsErrorInfo}

[#39]

@rhythmnewt We would like to understand more about your domain setup. We observe that SID which VM resolves the user to doesn't match the SID we are getting from his AAD token. Can you please give brief overview of your domain setup and how it is connected to AAD? Do you have multiple domains?

[#40]

@Joe Flynn

Do we know if there is a fix for this yet.  I have installed a new Validation Pool and still the same error.  My agent is 1.0.833.5

---

[#41]

@Roop_WVD I sent you identifiable details about my user account and domain in a PM.

My setup is as follows On-Prem AD -> Ad Sync -> AAD -> Azure ADDS

I do have password write-back enabled.

I do NOT have multiple On-Prem AD instances.

I do have multiple stand-alone AAD (cloud-only) instances.

VM in question is domain-joined to my Azure ADDS instance and I have no problem authenticating into it with my domain credentials.

Thank you for looking into this.

[#42]

@Roop_WVD  I hope you look into this in a more general way, as we are many with the same problem... Our setup is exactly the same as @rhythmnewt , so please express your findings here for us all to see.. :)

[#43]

@rhythmnewt Thanks for sharing this detail. Its very helpful to understand the setup. We have recently introduced a change where we need User SID's from VM and token to match before we allocate a session. There seems to be a case with AADDS where they may not always match. We are currently investigating how do we handle these scenarios. I will keep you posted on progress.

[#44]

@Mtollex70 : Thanks for letting us know that you have similar setup and yes we will look into it in a general way.

---

@Mtollex70 wrote:

@Roop_WVD  I hope you look into this in a more general way, as we are many with the same problem... Our setup is exactly the same as @rhythmnewt , so please express your findings here for us all to see.. :)

---

[#45]

We have identified a bug when are in this setup. This is now documented here : https://techcommunity.microsoft.com/t5/Windows-Virtual-Desktop/Announcement-Connectivity-issues-from...  . We are actively working to fix this. 

[#46]

Thanks for the update, I hope the fix comes soon!

[#47]

@Roop_WVDGlad to hear there is a fix underway. Can you provide us at least a rough ETA please. We are currently investigating moving some of our systems to WVD but right now our investigations and POCs have had to go on hold because of the broken resource templates and now this.

Can you also please comment as to when this service will be covered under full support and SLA if clients such as ourselves are looking to go to market,

Thanks in advance

Mark