[Announcement] Connectivity issues from synchronized users to VMs joined to AAD DS

[christianmontoya]

Hi everyone, thanks for the continued testing of WVD. We’ve seen multiple connection errors with UPN when connecting to VMs joined to Azure AD Domain Services. We’ve done some preliminary investigations and figured out which scenarios are currently affected and which scenarios should continue to work.

Works

Logging into VM joined to Azure AD DS instance with Azure AD user sourced from Azure Active Directory (aka, New user created just in Azure AD).

Does not work (and investigating fix)

Logging into VM connected to Azure AD DS with Azure AD user sourced from Windows Server AD (aka, synchronized to Azure AD through Azure AD Connect).

You will see an error in the Diagnostics similar to below:

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : OrchestrateAsync: SID value in the database is different than the value returned in the
orchestration reply from the agent for user ≤user1@contoso.com≥ with Id
54a45a4c-41ad-4374-5e41-08d6e4d9acde. This scenario is not supported - we will not be able to
redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 7/16/2019 3:17:24 PM

Workaround

If your setup matches the description but you would still like to test, we suggest creating cloud users in Azure Active Directory for the time being.

Resolution

No current ETA, but working towards a fix.

How to check where your user is sourced from

You can navigate to the Azure AD portal or the Azure Active Directory blade in the Azure portal, then go to users:

Locate where the Azure AD user is sourced.

[#2]

@christianmontoya 

I know before the post that Cloud ID only is working but that is not valid for our production POC

i been testing with cloud ID only and that works , further more the issue with synced account, it looks like recently (because this was working before) you doing  SID check between the azure synced account and the account in azure DS and that will not match. i'm wondering if the scenario without azure DS , i mean extending AD to the cloud and join the virtual desktop machines to the same domain will have the same issue or not for synced user account.

[#3]

@ashro2 : Thanks for the clarifying question, but no, the issue will not replicate if you have a hybrid setup and are joining your virtual machines to the domain that is syncing up the users with Azure AD Connect. The primary issue lies in the SID check, and that Azure AD DS creates a new SID (by design) for the users that it creates on the managed domain services instance.

[#4]

@christianmontoya 

Thanks   i came to the same conclusion when looking ate the object SID in AAD and Azure DS and the Mismatch. i have 2 comments 

1. this check was introduced recently because this scenario was working before , is it possible to trun off this check of the SID? I saw the feedback on the form suggested moving the pool to validation pool where you deployed a fix for the issue but looks like that is not working as well. so is there a way to trun off this check i can do in my side?

2. is there a way to modify the Azure DS object SID to match AAD ? we don't have much control over the object in Azure DS I realized ?

it will be great if we can manually turnoff this SID check manually at least for testing

[#5]

@ashro2 : Unfortunately, it's not quite as simple as turning off the check since this check was implemented to stabilize the reconnection scenarios so that users get redirected back to a previously existing session (as opposed to get a new session).

I'm not sure if there's a way to manipulate the SIDs, but we're investigating all possible options right now.

Thank you for the feedback and dialogue though. We want to unblock testing, but also do not want to leave users in a bad state.

[#6]

@christianmontoya So no workaround for this scenario since the SID check is active now and according to you no ETA too. that's a bit disappointing! 

[#7]

I know the service is currently in preview, but i find the fact that this bug took multiple weeks to identify and acknowledge is a bit worrying for the state/future of AAD DS (that we rarely deployed before WVD).

Are there so few orgs using AAD DS ? Should we drop it and extend on-prem ADs to Azure LAN for WVD instead ?

[#8]

@Bazam Chekrian Valappu : Yes, we solved one failing behavior but now it's hindering another, but definitely working to achieve both.

[#9]

@Arthur GERARD : I wouldn't say that no one is using Azure AD DS or that it's not a viable solution. Primarily, understanding this failing scenario is an intersection of where customers are today and how they are piloting Windows Virtual Desktop with just cloud users (before trying to extend this with a full site-to-site on-prem infrastructure).

Between using Azure AD DS or extending existing domain structure to Azure, it depends on your scenarios you're targeting. You have much more flexibility by extending, since you can use Federation, Passthrough Authentication, or password hash (whereas AAD DS only works with password hash). Not sure if you've already seen this comparison article.

[#10]

@christianmontoyaThat makes sense, thank you.

Is "Azure AD join" on the roadmap for WVD ? Or will AAD DS continue to be the lightest deployment for our SMB customers ?

[#11]

@Arthur GERARD : Azure AD Join is definitely a scenario we want to support and we're in the initial investigation stages, as it's a larger change from how VDI/RDS has worked in the past. Unfortunately, this feature is not something that will make it into our initial GA. We will continue to update these forums and our Docs site as we have more information on this scenario, and other new ones.

[#12]

Just checking if there's any ETA on the fix for the initial problem in this thread. Thanks.

[#13]

@christianmontoya 

Hi Christian,

I seem to be experiencing the exact same error in a test environment. However, the user is sourced from Azure Active Directory.

I would be happy to help troubleshoot since I have clients looking forward to WVD. below is some info that might be relevant and if you need identifying tenant info I'll be happy to send via PM:

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : OrchestrateAsync: SID value in the database is different than the value returned in the orchestration reply from the agent for user ≤username≥ with Id <id>. This scenario is
not supported - we will not be able to redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 7/28/2019 14:17:15

TenantName : IC-WVD2
TenantGroupName : Default Tenant Group
HostPoolName : Desktop
FriendlyName :
Description :
Persistent : False
CustomRdpProperty :
MaxSessionLimit : 999999
LoadBalancerType : BreadthFirst
ValidationEnv : True
Ring :

[#14]

Hi,

We have just noticed the same problem in our test environment.

But a strange thing is that it only affects one of the 17 pilot users.

The users were synced from a local AD to Azure AD.

Azure AD connect sync was removed 1 year ago.

Azure AD services was setup to support the WVD environment.

Users envolved in pilot had to reset their passwords and could then logon.

But now, one user gets the error message:

SID value in the database is different than the value returned in the orchestration reply from the agent for user...

The Hostpool is in "validation" 

<#

ErrorSource      : RDBroker
ErrorOperation    : OrchestrateSessionHost
ErrorCode        : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage      : OrchestrateAsync: SID value in the database is different than the value returned in the orchestration reply from the agent for user ≤a.b@domain.se≥ with Id b663bb3d-3f67-42e9-f891-08d6fb3eb712. This scenario is not supported - we will not be able to redirect the user session.
ErrorInternal    : False
ReportedBy        : RDGateway
Time              : 2019-07-18 09:36:42

ErrorSource      : Client
ErrorOperation    : ClientRDPConnect
ErrorCode        : 2147965400
ErrorCodeSymbolic :
ErrorMessage      : Your computer can't connect to the Remote Desktop Gateway server. Contact your network administrator for assistance.
ErrorInternal    : True
ReportedBy        : Client
Time              : 2019-07-18 09:36:42

#>

[#15]

@christianmontoya would be great to get an update on when this will be fixed - we were happily using this with this setup then in abruptly broke and we've been investigating on and off as time allowed ever since.

Now i stumbled across this issue (after finally figuring out how to debug what was going wrong). Do we have an ETA as this is now a total block on us using WVD.

I'm really disappointed as this is the 2nd major stumbling block - we've fully adopted Azure AD and the lack of support for Azure AD join is the other one.

This can be such a good solution it's just so frustrating.....

[#16]

Is there a way to identify the public IP range used by azure virtual desktops to communicate with external resources such as O365

Services . this is required to apply some azure  access control policy

Thanks

[#17]

@ashro2 See this thread on github https://github.com/MicrosoftDocs/azure-docs/issues/33988#issuecomment-509722530

[#18]

@christianmontoya  Same issue for one of test Azure-born user, the second one (Azure-born as well) still works fine.

[#19]

Any update on a resolution? This is a hard blocker for us.

The workaround only works with NEWLY CREATED users - meaning I cannot delete a Windows Server AD user, then recreate with the same username as an Azure AD sourced user. It seems like Windows Virtual Desktop permanently stores the upn and sid in its database....so deleting and recreating the user in Azure AD doesn’t help...

[#20]

@jeffb8  @Richard Harrison @Integral-Consulting @rhythmnewt @Alex Ignatenko : Hi everyone, apologies for this thread going a bit quiet. We've done investigations and started implementing the fix, but will take time to roll into production. Once I get a better date, I will reach out here so you all can continue your testing!

Since we roll out in phases, I'd highly recommend setting up a host pool for validation, as this will be the first place to test.

Again, thanks all and we're definitely trying to get this fix rolled out as soon as we can.

[#21]

@Torbjörn Granheden : As it stands now, the issue stems from the SID's being synchronized as part of the Azure AD token and then receiving a different one through Azure AD Domain Services. Are you aware of any difference of properties between this 1 user and the other 16?

[#22]

@Arthur GERARD : Azure AD Join is in our backlog. We've heard overwhelming interest for this, and we want to align with Azure AD Join/Intune as a means of deploying and managing Windows. We don't have any specific dates on this, but we definitely want to supporting this as a scenario down the road.

[#23]

Thank you for keeping us informed!

[#24]

Thanks. Is there an eta for when a fix will be available to host pools in validation mode?

If not, is there any way to submit a support request to get you to delete stale user accounts from your sql azure database or is this exposed in any way? It would at least allow us to proceed with testing if there was a way to recreate the user accounts with problems. As I mentioned above - today we can’t even delete/recreate the user. It has to be created as a cloud only user with a different upn...

[#25]

Hola @christianmontoya, thanks for the information.

In my case, the scenario and behavior are the following:

I have an Active Directory On-Premise synchronized to Azure Active Directory through ADConnect. In Azure I have implemented an Azure Active Directory Domanin Services (AADDS). Both directories are synchronized (ADDS and AADDS) through the AAD. I have password hashes replication set. I implemented a WVD HostPool.

To perform tests with my synchronized users, I have also created Cloud users (AAD only).

Both types of users allow me to connect the most virtual machines of the WVD HostPool through RDP. However, when I try to use the WebClient through the URL https://rdweb.wvd.microsoft.com/webclient/index.html both types of users can log in with their AADDS and AAD credentials. But by selecting applications to log in to them, only users created in the cloud (in AAD) can successfully start; synchronized users from ADDS get the error from the following image:

The log error for synced users is the follow:

ActivityId : e5eaa99a-0873-4e39-9063-d39e511c0000
ActivityType : Connection
StartTime : 12/08/2019 5:09:12 p. m.
EndTime : 12/08/2019 5:09:18 p. m.
UserName : F21212121@fvl.org.co
RoleInstances : rdwebclient;mrs-eus2r0c002-rdgateway-prod-staging::RD2818788A5384;mrs-eus2r0c001-rdbroker-prod-staging::RD2818782C7086;≤WVDSH-0.fvl.org.co≥
Outcome : Failure
Status : Completed
Details : {[ClientOS, Win32 Chrome 76.0.3809.100], [ClientVersion, 1.0.18.5], [ClientType, HTML], [PredecessorConnectionId, ]...}
LastHeartbeatTime : 12/08/2019 5:09:19 p. m.
Checkpoints : {LoadBalancedNewConnection, TransportConnected, TransportConnecting}
Errors : {Microsoft.RDInfra.Diagnostics.Common.DiagnosticsErrorInfo}

Is the same error that you are describing in this post?

Thanks a lot for your response.

Paul Pedroza

[#26]

@christianmontoya 

any updates? we are hard blocked in terms of using Windows Virtual Desktop

[#27]

@christianmontoyaI have checked with every powershell cmdlet i can think of, but the users are identical configured. I have compared with another user that was hired at the same time (2014). And also has been migrated from an onprem AD to an Azure AD only environment. The ad connect was removed a year ago ish. The Azure Domain Services was setup to support WVD preview in June.

My user is on vaccation and I cannot get an answer if it still is an issue or if it has been solved by agent update.

But, you should think of a rollback of the sid verification and do a rearchitect.
If it is so much trouble for preview users, how will this work for GA?

/Mr T-Bone

[#28]

@christianmontoya  Any update on this  ? As others have reported we are at a stand still. 
Synced from on-premise aren't working.  I have tried validation pools and still no luck with Sync accounts. 

[#29]

Seems like we’re in store for a repeat of Azure RemoteApp.

[#30]

@christianmontoyaAnother week without status update?

Any progress of getting the WVD working again for all of us with Azure DS?

I have only one user out of 30 pilots that get sid failure?

• Cannot see any different attributes on this specific user compared with another user created same week.
• Both accounts created 3 years ago in a local AD.
• Synced to Azure AD with AD connect.
• Local AD and Azure AD connect dismounted and retired 12 month ago.
• Azure DS started for WVD 3 months ago.

 /Mr-Tbone

/Torbjörn

[#31]

@Torbjörn Granheden  @cititechs  @jeffb8 : Thanks for being patient with us. As an update, we've identified the issue and have taken the first step to solving it, just that's a multi-phase fix/roll-out.

Also, to address some of the feedback, in order to login users and work between cloud/on-prem accounts, there are only so many interfaces and returned values that the system gives us for logon. And, unfortunately, it wasn't as easy as rolling back because then we would then have other sets of users be unable to reconnect to existing sessions.

Will hope to have another update soon regarding the full fix.

[#32]

@jeffb8 : Just to get more clarity, is it primarily this issue that you think will make it the next Azure RemoteApp? Is there other functionality that we're missing, should be focusing on, or should be fixing?

[#33]

Hi @christianmontoya ,

The validation pool seems like a good idea (https://docs.microsoft.com/en-gb/azure/virtual-desktop/create-validation-host-pool)

However to make that really viable we need a schedule of upcoming releases to know when we should be validating (and potentially what specific areas to check). Is that something that is also going to be published?

Some control of when updates are pushed would also be very useful - for example if we find an issue during validation can we prevent that being pushed to our environments or would if just get pushed anyway after some timeout period?

Cheers,

Rich

[#34]

@christianmontoya- one other thing to just mention - we recently had some other issues with AADDS and in conversations with the product group there they told us there is a new version of the sync process planned (quite soon I think) from AAD to AADDS - not sure if this helps you in any way with the issues you have - perhaps if you have any requirements for changes these could be included in what that team is doing?

[#35]

@Richard Harrison : Great questions! We definitely intend to push out notice of things coming out the validation pool so it can be tested. We have done this in limited capacity and to smaller groups of customers, but we intend to use this more. We have also not pushed a build all the way to the general population due to issues we've seen in validation, so we plan on using it exactly like you're expecting.

And thank you for the notification. Will bring this up with the Azure AD DS team.

[#36]

@christianmontoya I have deleted and re-created my WVD test environment several times, now I can't longer log in even with users created directly in the Azure cloud, with these accounts, the users before could login. I can no longer log in with synchronized users from my AD On-Premise (ADDS -> AAD -> AADDS) nor with the old ones created directly in Azure (AAD -> AADDS). I can only use the scenario if I create new users in Azure.

ErrorSource : RDBroker
ErrorOperation : OrchestrateSessionHost
ErrorCode : -2146233088
ErrorCodeSymbolic : ConnectionFailedUserSIDInformationMismatch
ErrorMessage : User wahtever@whatever: SID information in the database 'S-1-5-21-1201331163-3862359571-1670876360-8430' does not match SID information returned by agent
'S-1-5-21-1194805571-575163812-3500997978-1549' in the orchestration reply.. This scenario is not supported - we will not be able to redirect the user session.
ErrorInternal : False
ReportedBy : RDGateway
Time : 28/08/2019 3:24:57 p. m.

[#37]

@christianmontoya

This issue specifically is **extremely** concerning - because this isn’t an edge case; this is a fundamental architecture/database design problem in how you uniquely identify users.

You don’t need to get AAD Domain Services or any other complicated scenario in the mix to reproduce this problem. All you need to do is delete **any** user in **any** kind of environment and then create a new one with the same upn. And bam, that user is screwed...forever.

Deleting and recreating the tenant doesn’t help any, which tells me that user registration data is stored independently of tenant data. This will lead down an avenue of problems with no end. There are alternative architectural approaches that would likely be more reliable.

[#38]

@jeffb8 @christianmontoya 

I can attest this! It seems like a flaw in design.

What happend to me is the following:

I had a user in my old Azure AD tenant. lets call it tenant A, with a UPN of user@domain.com and used WVD succesfully there.

Now I moved my domain.com to another Azure AD tenant, tenant B. Setup Azure ADDS there and tried to login with a user with UPN user@domain.com, so the exact same UPN of the user that existed in tenant A. although offcourse it doesn't exists in tenant A anymore.

What I saw when I logged in, was the WVD tenant with the published desktops that I created in my Azure AD tenant A! AND I saw my new WVD tenant with the published desktops that I created in my Azure AD tenant B.

Now, when I tried to sign-in to the desktops from the new tenant B, I get the same error as everyone else, that the SID doesn't match with the one in the database. 

Yes, I can understand that it doesn't match if you still saved the SID from the user in tenant A. But this is a completely new user in tenant B.

Just like JeffN825 already concludes, this means that the user data is independent from the tenant data, which seems strange to me.

Also, I would like some way to delete my old user with its SID from this backend database.

[#39]

@christianmontoya 

With each day that passes with no meaningful reply on this issue, I become more skeptical that the right team (one with extensive experience in distributed, AzureAD based authentication and authorization) is working on this product. I also wonder if the lack of/delay in reply is indicative of the team taking a pause to re-evaluate if they can successfully develop this solution...?

Is there any information you can provide that might alleviate this concern? 

[#40]

I'm trying to add users from a 'invited user' source or from an 'external aad' source to a remoteapp group using powershell. I notice that only users created directly in AAD can be added, but externals or invited ones cannot. I keep getting the error  "The specified UserPrincipalName does not exist in the Azure AD associated with the RD tenant.".

Can anyone confirm if these external users or invited users should work with WVD? 

[#41]

@Marcel A' Campo - external users can't work with WVD because they are not replicated via Azure AD Domain Services to the managed AD domain.

[#42]

@christianmontoya

This issue a showstopper for us.  We're trying to roll out VDI for thousands of users globally.... but the inability to automate handling of multiple security identifiers is a bit deal.

Glad I encountered this now, instead of 2 months from now when a single user identity will exist in AADDS, AzureAD, and ADDS.  

[#43]

@DubC85 @jeffb8 @Marcel A' Campo @Roel Everink @pau_pedroza @Torbjörn Granheden @cititechs / All :

To provide an update, we're still working on this fix. As alluded to before, the fix is 2 steps:

1. To create and populate the fields

2. To adjust the logic to use these fields

We've completed step #1 completely and #2 is what we're implementing/validating now. We expect this to complete and rollout within the next month.

This may still seem like a long time out, but we do want to be more cautious on the rollout and ensure we don't break user connections like the changes we made that landed here. There is always an option to push straight to production, but that also doesn't help us if there's another case that we missed and if we did quick validation.

Ultimately, we realize that this has led to the inability to quickly test out the service using Azure AD DS. Once the fix is live, you should be quickly unblocked, re-start efforts to evaluate the product as you need, and hope to see continued feedback as you have been so far on TechCommunity so far.

We will post back here as we progress further in the fix and when we have this available in the validation pools to test.

[#44]

@christianmontoya 

Another month - alright.  Still paying for the actual resource (that doesn't work)...have been for several months now.  This has become very disappointing.  Standstill - Business Units waiting on the solution - complaining about chargebacks for the resource (that doesn't work) will be the next thing.

Please guys - get this fixed.

[#45]

Hi @christianmontoya , is there any news on this?

This is a hard block for us as well!

[#46]

@christianmontoya

Can you share some technical details on what these fields are and how they are used? Any details that would inspire confidence in the solution you’ve designed would be helpful.

[#47]

@christianmontoya thanks for the update. 

We are waiting to deploy WVD with Azure AD DS and if I understand correctly it will be possible once the second fix is rolled out? We have host pools set as "Validation" host pools, can we hope to get the fixes out sooner to these hostpools?

Cheers,

Joakim

[#48]

@Joakim Westin - I wouldn't count on this fixing the underlying issue. From the description of the fix, it sounds like it will just work around it for some situations.

[#49]

@jeffb8 : Essentially, there are three pieces of information we need for processing the new or reconnecting user connection:

1. UPN in Azure AD token

2. SID in Azure AD token

3. SID that the on-premises domain sends back when it matches up the user

Everything is based according to the UPN, as that is provided in all tokens. The fixes will:

1. Update the SID for the UPN (accounts for user migration on premises or new instantiations of Azure AD DS for users sourced in Windows Server AD)

2. Update the SID that the on-premises domain sends back when it matches up the user, which is needed for manual/auto-reconnect scenarios.

We definitely hear feedback on "why SID?", but unfortunately that is needed for current logon APIs if we want to provide a consistent re-connect experience that can get triggered even if you lose Internet connectivity for a brief second or switch wireless networks.

[#50]

I don’t understand how this will resolve the underlying issue, which is as simple to reproduce as deleting and recreating a user. Or deleting an AAD DS domain and recreating it. It seems that would still be broken after this fix.

Further, you say
“2. Update the SID that the on-premises domain sends back when it matches up the user, which is needed for manual/auto-reconnect scenarios.”

First - I assume you mean the RDSH agent? If so, how is the agent going to get the token for the current AAD user (which contains the SID you want)? If what you mean is that you’re going to try to silently acquire a token from the agent as the user...please don’t. The user could be subject to MFA policies which would muck you up even further...