Exchange Online Improvements to Accelerate Replication of Changes to Azure Active Directory
Published Sep 09 2019 07:00 AM 84.3K Views

As part of continued improvements to the Exchange Online service we are releasing an improvement to the User object management experience.

Today, when you create or modify user’s properties via Exchange Admin Center (EAC), Exchange Online PowerShell or other API, the change replicates to Azure Active Directory (AAD) through a sync mechanism which can take some time to complete. Simply put, you might not see the result of your change in AAD for a while.

This “back-sync” mechanism (as you may have heard it referred to) from Exchange Online to AAD has served us well for many years making sure the data in both directories (which sometimes includes an on-premises directory if you have AAD Connect or Exchange Hybrid enabled) remains in sync. This sync mechanism provides key functionality that AAD and Exchange Online depend upon.

However, this back-sync mechanism can sometimes become slow (due to various reasons that we won’t go into here (we suspect it's someone called Dave in building 30)). If that happens, changes might not appear in AAD in a timely manner. This prevents admins from seeing a properly updated graph view in AAD or making additional changes like licensing or UPN changes (these changes mastered in AAD vs. mastered in EXO).

The Exchange Online and AAD teams have worked together to provide a new mechanism to synchronize the changes that originate in EXO and need to replicate to AAD without relying on the current back-sync mechanism.

Just as it is the case today, two specific attributes of tenant admin accounts (Phone number and Mobile phone number) will not be editable from Exchange Online after this feature is released. Because those can be used to validate password changes, Admin needs to modify them directly in AAD.

Once this improvement is deployed to your tenant, when you make user object changes in Exchange the changes will now be dual-written to AAD and EXO. The end result is that the replication of those properties should be close to immediate and changes made in EXO will immediately reflect in AAD when the cmdlet completes successfully.

For deleted items, incremental property updates are synced to EXO, however same will not be synced to AAD until the deleted item is restored as per guidance mentioned here. Once deleted item is restored, dual-write syncs incremental updates that happened to the item during deleted state on best effort basis.

For Groups, sync from EXO to AAD by Dual-write is supported only for group with RecipientTypeDetails (RTD) as MailNonUniversalGroup, MailUniversalDistributionGroup, and MailUniversalSecurityGroup. Groups with other RTD, e.g., ExchangeSecurityGroup, DynamicDistributionGroup are not supported.

Note that this implies that there is now an AAD dependency when making such management changes. If AAD is unavailable for some reason when we attempt this dual-write you may see errors when executing an Exchange cmdlet or management action. This error will reference AAD. Note, that this change should be transparent to all the Exchange management operations you are doing today. There is absolutely no need to change your scripts or your usage of EAC or other Exchange APIs. You may see new errors, but they should be treated the same as you do today: re-run the cmdlet. If the issue persists when you open a support ticket, just mention that the error in Exchange is due to AAD and support should be able to help you redirect to the correct teams.

Here is an example of the cmdlet error that you might see. You can always get more detailed information if you get the full error trace:

 

[MultiTenant\BL0PR00MB0354] PS C:\Users\chsun> set-mailbox testDareen@dualwritetesting.onmicrosoft.com -CustomAttribute1 "demo test"
An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.

    + CategoryInfo          : NotSpecified: (:) [Set-Mailbox], UnableToWriteToAadException
    + FullyQualifiedErrorId : [Server=BL0PR00MB0354,RequestId=6669e8ca-2919-450f-93d7-5757b75bef45,TimeStamp=3/15/2019
    9:07:01 PM] [FailureCategory=Cmdlet-UnableToWriteToAadException] B6FABBF,Microsoft.Exchange.Management.RecipientT
  asks.SetMailbox
    + PSComputerName        : bl0pr00mb0354.namprd00.prod.outlook.com

 

How to know if this has rolled out to your tenant?

To find out if this change has already rolled out to your tenant, you can run the following CMDlet; note that if the value does not display for your tenant at all, you can interpret that as “False”:

 

PS C:\Users\admin> Get-OrganizationConfig | fl IsDualWriteEnabled
IsDualWriteEnabled : False

 

Audit log entries with actions by “Microsoft Substrate Management”?

Once dual-write is enabled on your tenant, as part of dual-write operations, you will see audit log entries with actions taken by “Microsoft Substrate Management”. “Microsoft Substrate Management” is a service principal used by Exchange Online during dual-writing operations to AAD. These audit log entries refer to create/update/delete operations executed by EXO to AAD. These entries are informational in nature do not require any action.

We hope you agree that this change is a good thing. We really want your admin and management experience to be as efficient and useful as possible, and this change eliminates one issue we were aware of that caused some pain.

Please do let us know what you think in the comments section below, and we hope you notice the improvement!

The Exchange Team

37 Comments
Copper Contributor
What if I ignore the UnableToWriteToAadException? Will EXO retry syncing to AAD at a later moment on its own or do I absolutely have to re-run the cmdlet?
Microsoft
@Victor - you will need to retry the operation; once this change goes into the effect, the properties in question will not be written to Exchange Online if the write to AAD fails.
Steel Contributor

Oof - I feel bad for Dave in building 30 if the WSYP program is still in place. https://blogs.technet.microsoft.com/uktechnet/2012/04/27/friday-fun-microsoft-wsyp-we-share-your-pai...

Brass Contributor

How does this apply in a scenario such as Undo-SoftDeletedMailbox, where the cmdlet's success does not really tell the whole story about the background replication? Would those operations take a lot longer, or would they have the same failure modes they do today (e.g. potential collisions)?

Copper Contributor

Hi, in total this sounds very good - I only have one concern: what is the performance impact of this change? Updating Azure AD through Graph API is very slow compared to the old native exchange online cmdlets. Have you measured a mass update for let’s say 10.000 users with your old and your new script?

 

I also assume changes are only made to Cmdlets with the set,new or update verb? Get commands still grab everything from the underlying AD?

 

would be nice if you could do a comparison and share your results ;p

 

regards

christian 

Copper Contributor

Hi team,

Any plans to improve the forward sync mechanism or way to do it from the customer end?

How can we know if the improvement is deployed to our tenant?

Copper Contributor
Does this improve the speed of AAD Connect? Currently we sync AD changes hourly, which can be problematic. Then Mail changes from O365 have to sync back to AD, which can be very problematic. Dave has alot to answer for!
Steel Contributor

@jamsnz You can change the scheduler to even 5 minutes if you need to.

 

Set-ADSyncScheduler -CustomizedSyncCycleInterval d.HH:mm:ss

 

Microsoft

@KrisDeb You cannot change the Sync cycle interval less than 30 minutes. When you run Get-ADSyncScheduler, the AllowedSyncCycleInterval is set to 30 minutes. Any values set to a cycle lower than 30 minutes will default to 30. 

Steel Contributor

@Josh Villagomez Oh, thanks for that, you are right, of course.

Copper Contributor

Hi Team,

Receiving following error:
We have different accounts for local AD,azure AD and Exchange... could the issue lie there?

error
An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information+ CategoryInfo : NotSpecified: (:) [Enable-UMMailbox], UnableToWriteToAadException
+ FullyQualifiedErrorId : [Server=DB6PR8503MB0117,RequestId=d79f7b8f-b887-46e2-ad19-878f4cdc02ce,TimeStamp=9/24/20
9 6:24:02 AM] [FailureCategory=Cmdlet-UnableToWriteToAadException] 111CFF3D,Microsoft.Exchange.Management.Tasks.U
EnableUMMailbox
+ PSComputerName : outlook.office365.com
Microsoft

@jenmertens1605 for Enable-UMMailbox with RequestId=d79f7b8f-b887-46e2-ad19-878f4cdc02ce, looks the Source of Authority of the organization and the user is at on-prem. So Enable-UMMailbox should be run against on-prem. But this request looks running via Remote Powershell (RPS) in the Cloud machine DB6PR8503MB011. Looks to me the error is by design.

 

However, what is the behavior before dual-write enabled in Sept? You can run Enable-UMMailbox via RPS? If so, please open a support ticket and we will investigate and fix it.

Microsoft

@ChSun Does the dual-write design apply to hybrid user identities? This blog post mentions "when you make user object changes in Exchange the changes will now be dual-written to AAD and EXO." However, it does not say if the changes applies to cloud-only user objects or synchronized objects such as those exported by Azure AD Connect, where AD is the SOA. Thank you. 

Microsoft

@Josh Villagomez , yes, dual-write applies to hybrid user, too. Actually it's depends on SOA. If SOA is at on-premise, then no need to dual-write. Tenant admin is not able to make changes in EXO side (for dual-write properties, at least). But if SOA transferred to cloud, then we will dual-write the user if changes made on EXO side.

Copper Contributor

Just found the post. We're observing long delays (10 minutes or more in some cases) replicating primary SMTP address from Exchange Online to AAD mail attribute. Does not fit well to what was described above. Is this something that is expected? 

Not related. There is a sometimes significant delay in changes being replicated within the service the last few weeks. Just be patient.

Copper Contributor

Would DualWrite have removed the ability to run a "Set-MailUser user -WindowsEmailAddress zzz@externaldomain.yyy" command in ExchangeOnline for a MailUser object that is synced from OnPrem? I understand that you typically cannot update synced objects from the Cloud, but this command used to work. And the values are correct OnPrem, but not in the Cloud.


For example, I have a MailUser object on-prem (multiple, actually) which is synced to ExchangeOnline. The ExternalEmailAddress is correct (points to an external domain). On-prem, the PrimarySmtpAddress and WindowsEmailAddress match the ExternalEmailAddress.


Previously, in ExchangeOnline, the PrimarySmtpAddress and WindowsEmailAddress would match ExternalEmailAddress when first created. Then something would change and on a subsequent AD Connect Sync, they would no longer match. But, I could use the Set-MailUser command in ExchangeOnline to set WindowsEmailAddress to be that external address, and everything would start working again.


Now, if I try to run a Set-MailUser command in the cloud, it fails with this message:

 

 

An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online.
However, it failed. Detailed error message:
Unable to update the specified properties for on-premises mastered Directory Sync objects or objects currently
undergoing migration.

 


I'm not sure when DualWrite was enabled in my tenant, but this issue started very recently. Any thoughts?

 

(This is being done, by the way, to enable Google GSuite Calendar Availability sharing with the external domain. When the WindowsEmailAddress doesn't match the ExternalEmailAddress, or maybe PrimarySmtpAddress, availability lookups fail.)

Iron Contributor

It looks like this has negatively affected the Exchange Online on-boarding migration tool. Every migration batch we try to run (migration type: G Suite) now fails, with the following error for each user in the batch:

Error: MigrationProvisioningPermanentException: An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. Detailed error message: Unable to update the specified properties for on-premises mastered Directory Sync objects or objects currently undergoing migration. The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.

I've been fighting with this for days, restarting batches, creating new batches, small batches, large batches. Same result every time: complete failure. Office 365 support so far has been no help, though the ticket is still open. It certainly seems however that there are underlying infrastructure issues in Office 365 that are causing this and other issues.

Brass Contributor

Yeah, still doesn't update and work quickly.  Made a change and added an Alias to the Attributes of my user account...30min later...still rejects the email as user address not found. Admin doesn't show the attribute as another email address in Exchange online admin GUI etc....Ridiculous.  Latest ADSync installed, manually do delta/initial syncs...still crazy lag.  Even doing basic things in the Exchange online Admin GUI takes 30-45 seconds, like click a page and it takes forever to load, then fill info.

Copper Contributor

Looks like Microsoft broke everything

Same error when trying to convert user mailbox to shared mailbox, when edit Aliases in user mailbox.

All work has stopped.

 

Error executing request. An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. Detailed error message: Exception happened when doing AAD dual-write. For more details, please check RemoteException if using RPS client or innerException otherwise. RequestId : da43aad9-bde9-47ae-a976-40009cf66134 The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.
------------------------------------------
An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. Detailed error message: Exception happened when doing AAD dual-write. For more details, please check RemoteException if using RPS client or innerException otherwise. RequestId : 5596039b-0644-47e0-9d65-99ce48ffb2ba The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.
--------------------------------------------- -------------

Copper Contributor

Please help me with the below error when trying to change the smtp address of shared mailboxes or trying to add users into the shared mailboxe. any advice/ suggestions or work around is appreciated.

thumbnail_shared x sync error.jpg

Copper Contributor

Get the same error in a non-synced and non-hybrid environment. How to deal with it ?

I retried hours after hours and still the same error. Some settings seams to be editable but not the Displayname, alias etc..

 

An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. Detailed error message:
Resource 'e********' does not exist or one of its queried reference-property objects are not present. DualWrite (Graph) RequestId: 7****-***-**-***-**
The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.
+ CategoryInfo : NotSpecified: (:) [Set-DistributionGroup], UnableToWriteToAadException
+ FullyQualifiedErrorId : [Server=**,RequestId=****,TimeStamp=01.03.2022 14:29:38] [FailureCategory=Cmdlet-UnableToWriteToAadException] 7D0FCF5D,Microsoft.Exchange.Management.RecipientTasks.SetDistributionGroup
+ PSComputerName : outlook.office365.com
Copper Contributor

we used to be able to change smtp addresses via the o365 console. Appears this update has been rolled out to multiple customers of mine now and We have lost this functionality. Changes have to now be done on-prem and synced to 365. 

 

Getting this error in EXO/O365:

 

An Azure Active Directory call was made to keep object in sync between Azure Active Directory and Exchange Online. However, it failed. Detailed error message: Unable to update the specified properties for on-premises mastered Directory Sync objects or objects currently undergoing migration. DualWrite (Graph) RequestId: The issue may be transient and please retry a couple of minutes later. If issue persists, please see exception members for more information.

 

Is there a way to turn off the dual-write feature? 

Copper Contributor

This is an absolute pain of a feature when you're syncing users from AD -> AAD but not utilizing Exchange Hybrid or do not have the schema extended for Exchange in AD. Microsoft please fix this so customers can manage Exchange attributes in the cloud (IE, proxy addresses)

Microsoft

@Chris Blackburn Could you share more about your broken scenario, with complete error message and repro steps.

Microsoft

@migorengchilli It is as per design. On-premises mastered properties are not expected to be managed from online.

Copper Contributor

I'm a partner working with an Enterprise client and finishing up a migration from a 3rd party hosted resource forest Exchange design. ADConnect was syncing this resource forest and their Corporate AD forest, however, their local AD forest never had an Exchange server and the schema was never extended for Exchange. We needed to use 1.6.x in order to facilitate defining attributes across multiple directories (when we started the project there was a bug regression in the 2.0.x version that prevented us from seeing this page), and as part of the migration, we had Exchange Hybrid deployment enabled in the install._This allowed us to represent a single object in Azure AD with the immutableID coming from the corporate AD forest and Exchange attributes contributing from the resource forest. The metaverse presented the 2 source forests' attributes very nicely.

adconnect-exchange-hybrid.pngadconnect-uniquely-identify.png

_As we're decom'ing the 3rd party hosted resource forest from the M365 design, we cut off Directory Sync as one of the final steps (Set-MsolDirSyncEnabled -EnableDirSync $false) to shift user mailboxes, shared mailboxes, and distribution lists to Cloud only, and this worked rather flawlessly (avoiding having to rebuild DLs in EO, etc). The main reason for moving to cloud-only temporarily (less than 24 hours) was to clear the immutableID of the shared mailboxes (which equated to 35% of their total mailboxes) so we could manage their proxy addresses and other facets of Exchange management purely in the cloud.
But we needed to reenable DirSync against their Corporate AD forest for SSO with Password Hash sync, however, this AD was never extended for the Exchange schema, and told them this is not something they want to be in the business of managing.

_Once we reinstalled ADConnect (to remove the old resource forest references, get the client up to a supported version, and not have Exchange hybrid deployment enabled in ADConnect), we did our first sync and the Synchronization Service did its job and matched the users in the Corporate AD forest nicely against their immutable ID, and changed them from Cloud back to Windows Server managed. In doing so, it cleared all the Exchange managed attributes that were previously in the resource forest.

adconnect-user-update.png

_Yet, We still cannot manage the proxyaddresses for these users in the cloud.

_"In a perfect world" it would be great to go into the MSOnline module, run a command (say Get-MsolDirSyncConfiguration) and see if Exchange Hybrid deployment is enabled or disabled. If it's enabled, then things like proxyaddresses, etc would be within dual write scope. If it's disabled, you can manage them in Exchange Online).Another "negative" impact of the reinstall of the DirSync and the Corporate AD users from Cloud to Windows Server synced is that not have the schema extended for Exchange and not having the proxyaddresses populated, the email aliases and X.500 addresses from the resource forest were wiped out and many users are getting failures on email replies, updating calendar messages, etc. You can see when it did the first sync it wiped these out, and it cleared the msExchRemoteRecipientType attribute (which in my mind should be another flag that the object its not under the dual write scope)

What's interesting here is that I cannot add an X.500 address en-mass to all my users via Powershell, because I get the same error

adconnect-x500.png

BUT if i do this via Exchange Online admin center, it adds it with no issues
adconnect-x500-success.pngadconnect-x500-list.png
And all of my AD Synced users daily have (check out Correlation ID d8135576-6305-4b86-9b03-75de490163d8) have the Microsoft Substrate Management (which is tied to ADConnect) error

adconnect-Microsoft-Substrate-Management-failure.png

Copper Contributor

@Yajvendra@The_Exchange_Team
I've fought and fought with the UI for Tech Community to share this in the chat, yelled, and swore a few times, and have resorted to putting this into a Word doc on my OneDrive. I'd love to talk further because this is an active issue, so please feel free to message or at tag me if you have other questions

The Challenges of Dual Write Scope

Microsoft

@Chris Blackburn Referring the details provided in the document “The Challenges of Dual Write Scope”, realize your setup has Dir-Sync enabled and not have Exchange hybrid deployment enabled in ADConnect, so understand the Users are on-premises mastered, then getting following error is by-design, as on-premises mastered properties are not expected to be managed online.

“Unable to update the specified properties for on-premises mastered Directory Sync objects”

However, as you mentioned you can edit property in Exchange Online Admin Center, so suggest you raise case with Support for it.

Copper Contributor

Well today a bunch of accounts were created in our EXO and the culprit was “Microsoft Substrate Management”. After researching it some more, it looks like now users can create accounts in EXO through Microsoft Bookings. And they show up in the address book with no mailbox. This is a not something we want, and so I will be turning this feature Off in our tenant. 

Microsoft

@RoboRob This is expected behavior for Microsoft Bookings product documented here. Each new Bookings calendar creates a corresponding mailbox in Exchange, and a related entry in Azure Active Directory (AAD), where the entry is listed as an unlicensed user. You may delete the mailbox in Exchange to do away with these unlicensed user entries.

What if I want to know the original actor instead “Microsoft Substrate Management” for the activity by running the Audit log. Does it shows If I disable duelwrite by running the following command: 

PS C:\Users\admin> Get-OrganizationConfig | fl IsDualWriteEnabled
IsDualWriteEnabled : False

 I am looking for the original actor who made the changes from Exchange Online. 

mfrabbihasan27gmailc_0-1705557497518.png

 

Microsoft

@mfrabbihasan27gmailc Disabling Dual-Write is not supported for a tenant. From changes done, there could be possibility to get details of which cmdlet initiated Dual-Write to make the change for the user…you may request such details raising a support case.

Copper Contributor

See Below

Copper Contributor

Dual Write Back Error using AD connect for SMTP Proxy addresses

 

The problem is most of you have forgot to add the smtp: or SMTP: infront of the email address

 

to fix pls open AD onprem

 

I not done so already make sure Advanced Features is enabled

"View" from the file menu > "Advanced Features"

 

Find the user > right click Properties > Attribute Editor 

 

Search for "proxyAddresses" in the list > "Edit"
add examples:

SMTP:email address removed for privacy reasons

smtp:email address removed for privacy reasons

smtp:email address removed for privacy reasons

smtp:email address removed for privacy reasons

 

 

a new customer of ours had the same issue for many years and this was an easy fix

 

Copper Contributor

Not sure I like this. If, like us, you export audit logs to an external SIEM, all you get to see is "ServicePrincipal_xxxxxxx-1d02-4186-aba2-eb1a91866024" made the change  in AD. The Exchange group addition is then somewhere completely different (in time and structure) (and doesn't contain the user name that was added - only the GUID) :sad:

 

Sure I can go and find it in the unified log, but if I'm trying to find out WHO made the change I have to unpick about 5 levels of nesting in the powershell. I can't search on either the name of the distribution group or name of the member. It's not exactly very auditable.  

 

And how long is that kept for?  I have no control over its retention period. 

 

Co-Authors
Version history
Last update:
‎Aug 25 2022 06:23 AM
Updated by: