Azure AD Connect Failed Automatic Upgrade 1.6.16.0 to 2.1.16.0

[CSU_Scott]

We had an automatic upgrade fail for our Azure AD Connect from version 1.6.16.0 to 2.1.16.0.

logs show the following:

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::SetStringValue(HKEY_LOCAL_MACHINE, SYSTEM\CurrentControlSet\services\ADSync\Parameters, SqlAoaAsyncMode, False)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::SetStringValue

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::SetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, ServiceAccountType, VirtualServiceAccount)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::SetStringValue

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupAdmins, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncAdmins

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupOperators, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncOperators

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupBrowse, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncBrowse

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupPasswordSet, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncPasswordSet

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SYSTEM\CurrentControlSet\services\ADSync\Parameters, Path, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:C:\Program Files\Microsoft Azure AD Sync\

Azure AD Connect Upgrade Information: 904 : SynchronizationServiceSetupTask: UpgradeCore: Attempting to upgrade LocalDB instance.

Azure AD Connect Upgrade Information: 904 : SyncServiceAccount: Using a VirtualServiceAccount

Azure AD Connect Upgrade Information: 402 : SyncServiceAccount:ResolveSid path=WinNT://NT SERVICE/ADSync

Azure AD Connect Upgrade Verbose: 903 : Found sid SidTypeWellKnownGroup : S-1-5-80-3245704983-3664226991-764670653-2504430226-901976451

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::DoesRegistrySubKeyExist(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileListS-1-5-80-3245704983-3664226991-764670653-2504430226-901976451.bak)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::DoesRegistrySubKeyExist:False

Azure AD Connect Upgrade Information: 904 : SyncServiceAccount.ResolveSid: SyncServiceAccount.SidString=S-1-5-80-3245704983-3664226991-764670653-2504430226-901976451

Azure AD Connect Upgrade Verbose: 903 : IsMemberOfLocalGroup: Encountered error when enumerating group member. Details: The user name or password is incorrect.

Azure AD Connect Upgrade Verbose: 903 : IsMemberOfLocalGroup: Encountered error when enumerating group member. Details: The user name or password is incorrect.

Azure AD Connect Upgrade Verbose: 903 : IsMemberOfLocalGroup:  

(it loops at this point until you kill the upgrade process if running manually)

On a healthy install, it looks like this -- bolding the area where this successfully seemed to install in the healthy sync environment.

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::SetStringValue(HKEY_LOCAL_MACHINE, SYSTEM\CurrentControlSet\services\ADSync\Parameters, SqlAoaAsyncMode, False)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::SetStringValue

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::SetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, ServiceAccountType, VirtualServiceAccount)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::SetStringValue

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupAdmins, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncAdmins

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupOperators, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncOperators

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupBrowse, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncBrowse

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\MSOLCoExistence\CurrentVersion, GroupPasswordSet, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:ADSyncPasswordSet

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::GetStringValue(HKEY_LOCAL_MACHINE, SYSTEM\CurrentControlSet\services\ADSync\Parameters, Path, {NULL})

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::GetStringValue:C:\Program Files\Microsoft Azure AD Sync\

Azure AD Connect Upgrade Information: 904 : SynchronizationServiceSetupTask: UpgradeCore: Attempting to upgrade LocalDB instance.

Azure AD Connect Upgrade Information: 904 : SyncServiceAccount: Using a VirtualServiceAccount

Azure AD Connect Upgrade Information: 402 : SyncServiceAccount:ResolveSid path=WinNT://NT SERVICE/ADSync

Azure AD Connect Upgrade Verbose: 903 : Found sid SidTypeWellKnownGroup : S-1-5-80-3245704983-3664226991-764670653-2504430226-901976451

Azure AD Connect Upgrade Verbose: 901 : >>> RegistryAdapter::DoesRegistrySubKeyExist(HKEY_LOCAL_MACHINE, SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileListS-1-5-80-3245704983-3664226991-764670653-2504430226-901976451.bak)

Azure AD Connect Upgrade Verbose: 902 : <<< RegistryAdapter::DoesRegistrySubKeyExist:False

Azure AD Connect Upgrade Information: 904 : SyncServiceAccount.ResolveSid: SyncServiceAccount.SidString=S-1-5-80-3245704983-3664226991-764670653-2504430226-901976451

Azure AD Connect Upgrade Information: 904 : SyncServiceAccount:AddToLocalAdministratorsGroup:

Azure AD Connect Upgrade Information: 904 : Starting: Temporarily adding the ADSync account to the local Administrators group...

Azure AD Connect Upgrade Verbose: 903 : The 'WinNT://NT SERVICE/ADSync' has been added to the 'WinNT://SYNC-AAD-01A/Administrators,group' successfully.

Azure AD Connect Upgrade Information: 904 : Finished: Temporarily adding the ADSync account to the local Administrators group. Duration: 0.013 sec.

Azure AD Connect Upgrade Verbose: 903 : RestartBootstrapService:

[22:17:16.308] [  1] [INFO ] ServiceControllerProvider: processing StopService request for: ADSync

[22:17:16.308] [  1] [VERB ] ServiceControllerProvider:  Initial service status: Running

[22:17:16.308] [  1] [VERB ] ServiceControllerProvider:  stopping service and waiting for completion.

[22:17:34.264] [  1] [INFO ] ServiceControllerProvider: StopService status: Stopped

[22:17:34.264] [  1] [INFO ] ServiceControllerProvider: Processing StartService request for: ADSync

[22:17:34.264] [  1] [VERB ] ServiceControllerProvider:  Initial service status: Stopped

[22:17:34.264] [  1] [VERB ] ServiceControllerProvider:  Starting service and waiting for completion.

[22:17:34.717] [  1] [INFO ] ServiceControllerProvider: StartService status: Running

Azure AD Connect Upgrade Verbose: 903 : RestartBootstrapService: completed successfully.

Azure AD Connect Upgrade Information: 904 : SynchronizationServiceSetupTask: UpgradeCore: Upgraded LocalDB instance successfully created.

Azure AD Connect Upgrade Verbose: 903 : RestartBootstrapService:

[22:18:02.480] [  1] [INFO ] ServiceControllerProvider: processing StopService request for: ADSync

[22:18:02.480] [  1] [VERB ] ServiceControllerProvider:  Initial service status: Running

Any thoughts on what I should be looking for here?  We opened a ticket on the 15th and I feel like we are just spinning our wheels at this point.  Hoping someone from the community may have seen this issue previously.

Best regards,

Scott

[#2]

There seems to be something wrong with the account that's being used. Microsoft already addressed this in the Azure AD release notes: "We fixed a bug where auto-upgrade fails when the service account is in "UPN" format." source: https://docs.microsoft.com/en-us/azure/active-directory/hybrid/reference-connect-version-history

Try to use the SamAccountName instead of UPN. I hope this helps you to get in the right direction.

[#3]

It was already on the samaccountname. Thank you. We are about a month in with Microsoft Azure support and it continues to escalate along the support chain of command. I have now supplied extensive logging and apparently a few other folks have had this issue, but the solutions have been different for each.

I will try to post back once we have a solution. Thanks again for your replly.

Best regards,

Scott

[#4]

Did you solved this issue? Seems like security Policy causing the upgrade issue, do you have strict user policies on the server? Like deny logon, Log On as a Service and Log on as a Backup etc? I would suggest you to move the server from the GPO's and try the upgrade process.

[#5 · Accepted solution]

After opening a ticket on August 15th with Azure AD technical support, September 12th, I was informed that my issue was out of scope and would not be supported.  We manage two forests and the sync servers are on the same network in this example.  One server automatically upgraded without issue.  One did not.  Reviewing the logs, the software upgrade failed on enumerating the local administrator group.  Support decided that was an on premise issue, assuming it was a change in our environment, and no further help was forthcoming.  While disappointed by this decision, we continued to work the issue and started over again.

Looking at the local administrator group, we determined there was a security group that was a member from the other forest via trust.  This group had been part of the local administrator group since June 2018.  Removing this group from the server's local administrator group allowed Azure AD Connect to install on a new server and upgrade on the other server after a swing migration process.

If you see an error with group enumeration, consider stripping groups or reviewing membership of the group generating the error.  

Thanks for the suggestions, we really appreciate the community feedback and investigated them as possible solutions.