Microsoft Secure Tech Accelerator
Apr 03 2024, 07:00 AM - 11:00 AM (PDT)
Microsoft Tech Community

Uncharted AAD Connect problem enabling password hash sync

Copper Contributor

Hi community!

 

I am experiencing an issue with AAD Connect that I can't seem to find any other mentions of and so i'm a bit lost.

We are unable to enable password hash sync in our AAD Connect environment.

On the 'Configuration Complete' page, we get 'Unable to configure password hash synchronization. Please consult the event log for additional information.'

 

And when checking the configuration afterward in the AAD Connect app and online, Password hash sync is disabled.

 

The event log shows in quick succession each time we try and enable password sync:

informational event 904-

ProvisioningWebServiceAdapter::ExecuteWithRetry: Action ProvisioningWebServiceAdapter::SetCompanyDirSyncFeatures, Exception: Microsoft.Online.Coexistence.ProvisionException: An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: 56924796-502a-4037-80fe-cbe67ca01d80 Server Name: . ---> System.ServiceModel.FaultException`1[Microsoft.Online.Coexistence.Schema.AdminWebServiceFault]: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support.

 

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Online.Coexistence.Schema.IProvisioningWebService.SetCompanyDirsyncFeatures(Int32 dirsyncFeatures)

at Microsoft.Online.Coexistence.ProvisionHelper.<>c__DisplayClass52_0.<SetCompanyDirSyncFeatures>b__0()

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

--- End of inner exception stack trace ---

at Microsoft.Online.Coexistence.ProvisionHelper.AdminWebServiceFaultHandler(FaultException`1 adminwebFault)

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

at Microsoft.Azure.ActiveDirectory.Synchronization.ProvisioningWebServiceAdapter.ProvisioningWebServiceAdapter.ExecuteWithRetry(String actionName, Action action).

 

Followed by Error event 6306-

The server encountered an unexpected error while performing an operation for the client.

 

"BAIL: MMS(120116): ..\server.cpp(8914): 0x80004005 (Unspecified error): An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: 56924796-502a-4037-80fe-cbe67ca01d80 Server Name: .Azure AD Sync 1.2.70.0"

 

Now usually i'd search by event ID and part of the details text to find similar problems that have been resolved in the past - but i really can't seem to find any for this particular issue.

 

What has been tried:

-Re-install AAD Connect

-Try AAD Connect on a different domain controller

-Use different credentials for the on-prem domain in AAD Connect (verified with the correct permissions set against https://docs.microsoft.com/en-us/azure/active-directory/hybrid/reference-connect-accounts-permission...)

-Updates and reboots for DC

-Ran Wireshark to try and identify if there is a network problem - seemed ok

-Ran the powershell commands at http://blog.cyberadvisors.com/aadconnect-password-sync-issue-resolved to enable using powershell. This set password sync enabled at the AAD Connect app but it was still disabled at the tenant.

Events as a result of the powershell commands-

Warning event 660-

Server reported that password hash synchronization is disabled for azure ad tenant. Attempting to enable. Attempt: 1

Informational event 111-

ProvisioningServiceAdapter::ExecuteWithRetry: Action SetCompanyDirSyncFeatures, Unexpected Exception: Microsoft.Online.Coexistence.ProvisionException: An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: e8bb4807-e248-4783-8ac1-f8066fbec9c4 Server Name: . ---> System.ServiceModel.FaultException`1[Microsoft.Online.Coexistence.Schema.AdminWebServiceFault]: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support.

 

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Online.Coexistence.Schema.IProvisioningWebService.SetCompanyDirsyncFeatures(Int32 dirsyncFeatures)

at Microsoft.Online.Coexistence.ProvisionHelper.<>c__DisplayClass52_0.<SetCompanyDirSyncFeatures>b__0()

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

--- End of inner exception stack trace ---

at Microsoft.Online.Coexistence.ProvisionHelper.AdminWebServiceFaultHandler(FaultException`1 adminwebFault)

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.ExecuteWithRetry(String actionName, Action action).

Error event 652-

Failed credential provisioning batch. Clearing affinity to the current service endpoint: https://adminwebservice.microsoftonline.com/provisioningservice.svc. Error: Microsoft.MetadirectoryServices.ServerDownException: Unexpected exception thrown. Action: SetCompanyDirSyncFeatures, Exception: An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: e8bb4807-e248-4783-8ac1-f8066fbec9c4 Server Name: . ---> Microsoft.Online.Coexistence.ProvisionException: An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: e8bb4807-e248-4783-8ac1-f8066fbec9c4 Server Name: . ---> System.ServiceModel.FaultException`1[Microsoft.Online.Coexistence.Schema.AdminWebServiceFault]: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support.

 

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Online.Coexistence.Schema.IProvisioningWebService.SetCompanyDirsyncFeatures(Int32 dirsyncFeatures)

at Microsoft.Online.Coexistence.ProvisionHelper.<>c__DisplayClass52_0.<SetCompanyDirSyncFeatures>b__0()

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

--- End of inner exception stack trace ---

at Microsoft.Online.Coexistence.ProvisionHelper.AdminWebServiceFaultHandler(FaultException`1 adminwebFault)

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.ExecuteWithRetry(String actionName, Action action)

--- End of inner exception stack trace ---

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.ExecuteWithRetry(String actionName, Action action)

at Microsoft.Azure.ActiveDirectory.Connector.PasswordChangeNotificationExtension.EnableAADPasswordSyncCompanyFeature(Int32 attemptCount)

at Microsoft.Azure.ActiveDirectory.Connector.PasswordChangeNotificationExtension.SetPasswords(IList`1 allPasswords, String forestInfo).

Error event 6900-

The server encountered an unexpected error while processing a password change notification:

 

"Unexpected exception thrown. Action: SetCompanyDirSyncFeatures, Exception: An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: e8bb4807-e248-4783-8ac1-f8066fbec9c4 Server Name: .

 

at TargetExtensionManager.ExportPasswords(TargetExtensionManager* , ECMAInformation* ecmaInformation, DynamicArray<ActiveDirectoryPasswordChange \*>* targetPasswordChanges, Char* forestInfo)

 

 

InnerException=>

An error occurred. Error Code: 108. Error Description: The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support. Tracking ID: e8bb4807-e248-4783-8ac1-f8066fbec9c4 Server Name: .

 

at Microsoft.Online.Coexistence.ProvisionHelper.AdminWebServiceFaultHandler(FaultException`1 adminwebFault)

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

at Microsoft.Azure.ActiveDirectory.Connector.ProvisioningServiceAdapter.ExecuteWithRetry(String actionName, Action action)

 

 

InnerException=>

The cause of the error is not clear. This operation will be retried during the next synchronization. If the issue persists, contact Technical Support.

 

 

Server stack trace:

at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

at Microsoft.Online.Coexistence.Schema.IProvisioningWebService.SetCompanyDirsyncFeatures(Int32 dirsyncFeatures)

at Microsoft.Online.Coexistence.ProvisionHelper.<>c__DisplayClass52_0.<SetCompanyDirSyncFeatures>b__0()

at Microsoft.Online.Coexistence.ProvisionHelper.InvokeAwsAPI[T](Func`1 awsOperation, String opsLabel)

 

 

InnerException=>

none

"

 

The environment:

-2x Server 2008 R2 SP1 Domain Controllers

-No proxy acting on DCs

-Traffic originatinating from DCs is not inspected or blocked at the firewall level

-Tenant was previously synced with another domain controller (we are integrating a purchased business that already had o365, but we did not) used guide at https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-install-existing-tenan...

-Exchange hybrid configured

 

I'm thinking maybe the tenant is problematic, perhaps because of some lingering settings or data from the old AADconnect implementation. But who knows

 

Hoping someone out there has some ideas!

 

Thanks for reading

David

0 Replies