My user has a mailbox both on-premises and in Exchange Online. Help!

[The_Exchange_Team]

In support, we hear from people in this “duplicate mailbox” situation relatively often. When this happens, our customers ask us for the best way to recover. In my experience, there are typically two main scenarios under which accounts can get into this situation.

Duplicate (new) mailbox in Exchange Online

An Exchange Online license was applied to the user before the Exchange GUID got synchronized from on-premises Active Directory. For synchronized accounts, having the Exchange GUID synchronized from on-premises is used to tell Exchange Online that the mailbox hasn’t been migrated yet, and is what allows customers to pre-license accounts prior to migration.

Pre-licensing is recommended for many reasons. For example, if the user is licensed, we bypass the validation that all email addresses must match an accepted domain. Another example where pre-licensing accounts is helpful is so that the mailbox doesn’t get disconnected immediately post-migration due to not having a license. This is especially important if “partial” licenses are used (i.e. apply the license for Azure AD, SharePoint Online, but not for Exchange Online). The bottom line is – if the Exchange GUID hasn’t been synchronized when the Exchange Online license is applied, Exchange Online has no way of knowing that there is an on-premises mailbox, so it dutifully provisions a brand new (empty one) for the user.

Duplicate (new) mailbox on-premises

Assume that a mailbox was migrated to Exchange Online, and then the on-premises account was mailbox-enabled again. In most cases, this ends up happening due to utilizing an Identity management tool of some sort (even if it is a PowerShell script!) which decides that users should be mailbox-enabled when they really shouldn’t be. Here, you end up with the opposite end result: a migrated mailbox is in Exchange Online, and a brand new (empty) mailbox is created on-premises.

I’m already in this state; now what?

No matter how you get into this situation, the recommended recovery process will be the same. You can find that recently documented process here:

How to recover when a mailbox exists in both Exchange Online and on-premises

In this blog post, we did want to acknowledge that we know there might be other ways to recover from this scenario. We spent quite a bit of time discussing how to best approach this and document a process that repeatedly gives you the best chance of success with no data loss. For example, one of the more common methods in the past was to disconnect the Exchange Online mailbox by removing the license, migrate the on-premises mailbox, then perform a restore (New-MailboxRestoreRequest) of the previous Exchange Online mailbox into the newly migrated mailbox. The main reason that we do not suggest this as a recovery method is that recovery from Exchange Online disconnected mailboxes cannot always be guaranteed (there can be factors like passing of time etc. which can make the process not be successful). If 99% of the time, it will work flawlessly, but that 1% of the time it doesn’t work, we do not suggest it because you might lose data as a result.

Note that If there are changes in functionality that allow us to document additional recovery methods, we will make sure to update the documentation with those additional methods.

We’d love your feedback on if the documented process works for you and what your thoughts are.

Ben Winzenz

[Kyle Berwaldt]

We see this scenario far more often than we would like due to issues with a third party identity management solution that is executing built in functions that utilizes enable-mailbox.  The bigger question is why doesn't enable-mailbox check to see if there is already Exchange properties on an AD object?  It can be difficult to have an identity management solution check to see if the account is already in the cloud. 

[Shaun Jennings]

We force this condition in our environment due to the fact that we are not allowed to have a migration endpoint. In order to get the on-premises mailbox and the cloud mailbox to connect and have our AAD Connect sync properly, we make sure to copy the Exchange Online GUID of the mailbox and add it to the proper AD property. We then force the mailbox into a remote mailbox.

<Connect to O365>

$oEmailGuid = Get-Mailbox -Identity $UPN | Select-Object ExchangeGUID

<Connect to on-premises AD>

Set-ADUser $SamAcct -Replace @{msExchRecipientDisplayType = "-2147483642" } -Credential $cred
Set-ADUser $SamAcct -Replace @{msExchRecipientTypeDetails = “2147483648” } -Credential $cred
Set-ADUser $SamAcct -Replace @{msExchRemoteRecipientType = “4” } -Credential $cred
Set-ADUser $SamAcct -Replace @{targetAddress = $TAddress } -Credential $cred

<Connect to on-premises Exchange>

Set-RemoteMailbox $UPN -ExchangeGuid $oEmailGuid.ExchangeGuid

[bwinzenz]

@Kyle Berwaldt this will largely depend on how the Identity management system is mailbox-enabling these accounts and may also depend on the Exchange version.

If I try and mailbox-enable a user whose mailbox was migrated to Exchange Online from my Exchange 2016 server using PowerShell, I get the following error:

Enable-Mailbox on this mail user is disallowed because the mailbox has been migrated.

Exchange 2010 allowed me to mailbox-enable the same user with no error.

[Kyle Berwaldt]

@bwinzenz , Thank you!  Thats great to hear that the issue has been resolved in 2016.

The identity management solution is running the cmdlet enable-mailbox.  We are still on Exchange 2013 CU21 where the cmdlet will still successfully run :(.

[Caleb Caraway]

@Kyle Berwaldt I don't think the EXO mail attribute writes back, but even if it did you'd still have the gap in the initial replication. We use dynamic 365 licensing policies based on AD properties, along with enable-remotemailbox. This has prevented the issue from happening anymore and creates the object directly in EXO.

[Phillip Kenyon]

Interested in seeing the 'recomended' process, having executed the other method a few times.

[Jeff Arndt]

In the linked article, step 7. Could you go in to more detail about this command, the one to restore the Exchange on-prem disconnected mailbox to Exchange Online?

New-MailboxRestoreRequest -RemoteHostName mail.contoso.com -RemoteCredential $cred -SourceStoreMailbox “exchange guid of disconnected mailbox” -TargetMailbox “exchange guid of cloud mailbox” -RemoteDatabaseGuid “guid of on-premises database” -RemoteRestoreType DisconnectedMailbox

I guess I run this in an Exchange Online Powershell, and it will pull in content over the Hybrid connection?

[bwinzenz]

@Jeff Arndt correct you run it from Exchange Online PowerShell. Similar to how Hybrid mailbox moves to Exchange Online utilize remote credentials, this allows you to copy the content from the on-premises disconnected mailbox into the cloud mailbox. It does require that you have MRS Proxy enabled on your EWS virtual directories (enabled by default on Exchange 2013 and newer), and that you allow external access to /ews/mrsproxy.svc even if it is just from Office 365 IP's.

[Lukas_Najman]

Hello,

does this recovery method also work with Exchange 2010?

I've just tried to create such "New-MailboxRestoreRequest" in our O365 tenant and got an error:

Remote server '<server name>' version (14.3.399.0 ServerCaps:, ProxyCaps:, MailboxCaps:, legacyCaps:05FFFF) isn't supported. Missing API: 'TenantHint'.

    + CategoryInfo          : NotSpecified: (:) [New-MailboxRestoreRequest], MRSRemotePermanentException

    + FullyQualifiedErrorId : [Server=VI1PR02MB4031,RequestId=dccb4fca-f7d6-4b4c-a929-d15aac70bbbe,TimeStamp=9/30/2019 1:14:08 PM] [FailureCategory=Cmdlet-MRSRemotePermanentException] 7B1FD142,Microsoft.Exchange.Management.Migration.MailboxReplication.MailboxRestoreRequest.NewMailboxRestoreRequest

    + PSComputerName        : outlook.office365.com

Normal mailbox moves from on-prem -> O365 work OK

Thanks in advance for your feedback.

Lukas

[Jeremy Bradshaw]

One problem with the recommended solution (when keeping the EXO mailbox), and this would be true for anytime you're keeping the EXO mailbox, but users' Outlook profiles are using the on-premises mailbox - new Outlook profile is required because the ExchangeGuid of the mailbox is going to be different.

I think I'll submit a pull request for the linked docs page to add a note about this.  If/when I do, I'll also add the use case which I think is the far more common one - that the problem state is reached by Exchange Online provisioning a new mailbox upon the EXO license being assigned prematurely.  The opposite (where Exchange on-premises creates a new mailbox), I've never seen that happen.  I get that it could, but it would be extremely rare in comparison to the scenario where EXO creates the 2nd mailbox.

I still think there's room for improvement in EXO around this problem.   For example, there could be an option added to OrganizationConfig to disable automatic mailbox provisioning upon license assignment.  It's too common that this issue comes up.  MS Support's first line is NOT ready for this problem.  It's true that customers should not assign EXO licenses to newly created users until after they've done Enable-RemoteMailbox (or New-RemoteMailbox / new > Office 365 mailbox in the GUI), and then waited for 2 AAD Connect sync cycles to pass, but the likelihood that many customers will continue to make this error is very high, and time has proven this.

Recommending pre-licensing is not a safe idea to me.  The example reason given, to avoid the mailbox being deleted immediately after migration - that just reveals how the 30-day grace period, sometimes it just doesn't happen.  Pre-licensing would avoid that, yes, but it will also tee up 10's/100's of 1000's of customers to pre-license prematurely (PPL - new term!).

[Jeremy Bradshaw]

I'm hoping to find somebody out there who has tried this process (New-MailboxRestoreRequest of on-premises soft-deleted mailbox into EXO live mailbox) recently?  I find that it doesn't actually work.  The error I'm getting is unclear because it says "could not find a recipient with ExchangeGuid <guid of on-premises mailbox>.  When I dig in, I find this fatal error is happening after a 100% successful (otherwise) copy of all items from the on-premises mailbox to the cloud mailbox, so it's failing on some kind of wrap-up step.  Here are the last two entries in my restore request's report:

>Get-MailboxRestoreRequest -Name T2_test5 | Get-MailboxRestoreRequestStatistics -IncludeReport | select -ExpandProperty Report | select -ExpandProperty Entries | select -Last 2


CreationTime               : 7/21/2020 12:16:36 AM
ServerName                 : YTBPR01MB4016
Type                       : Informational
TypeInt                    : 0
Flags                      : None
FlagsInt                   : 0
Message                    : Copy progress: 799/799 messages, 25 MB (26,215,094 bytes)/25 MB (26,215,094
                             bytes), 0/0 folders completed.
MessageData                : {0, 1, 0, 0...}
MessageBytes               : {10, 68, 67, 111...}
Failure                    :
BadItem                    :
ConfigObject               :
MailboxSize                :
SessionStatistics          :
ArchiveSessionStatistics   :
MailboxVerificationResults : {}
DivergenceFixupResults     : {}
DebugData                  :
Connectivity               :
SourceThrottleDurations    :
TargetThrottleDurations    :
UnknownElements            :
UnknownAttributes          :
XmlSchemaType              :
LocalizedString            : 7/21/2020 12:16:36 AM [YTBPR01MB4016] Copy progress: 799/799 messages, 25 MB
                             (26,215,094 bytes)/25 MB (26,215,094 bytes), 0/0 folders completed.
Identity                   :
IsValid                    : True
ObjectState                : New

CreationTime               : 7/21/2020 12:16:36 AM
ServerName                 : YTBPR01MB4016
Type                       : Error
TypeInt                    : 4
Flags                      : Failure, Fatal
FlagsInt                   : 18
Message                    : Fatal error RecipientNotFoundPermanentException has occurred.
MessageData                : {0, 1, 0, 0...}
MessageBytes               : {10, 29, 70, 97...}
Failure                    : RecipientNotFoundPermanentException: Cannot find a recipient that has mailbox
                             GUID '2ed5d0ca-54e2-4226-a4ce-a48848e18c0f'.
BadItem                    :
ConfigObject               :
MailboxSize                :
SessionStatistics          :
ArchiveSessionStatistics   :
MailboxVerificationResults : {}
DivergenceFixupResults     : {}
DebugData                  :
Connectivity               :
SourceThrottleDurations    :
TargetThrottleDurations    :
UnknownElements            :
UnknownAttributes          :
XmlSchemaType              :
LocalizedString            : 7/21/2020 12:16:36 AM [YTBPR01MB4016] Fatal error
                             RecipientNotFoundPermanentException has occurred.
Identity                   :
IsValid                    : True
ObjectState                : New

If there is some special trick which is outside and above the linked docs article, I would be thrilled to hear it.

[Shaun Jennings]

@Jeremy Bradshaw 

We had a similar issue with hybrid accounts. We couldn't restore because the account was in legal hold. We also could not restore into a new account because the Exchange GUIDs did not match.

[Jeremy Bradshaw]

@Shaun Jennings Thanks for the response, I'm thinking with Inspire on the go, my chances of getting any attention here are slim right now.  Unfortunately for me, there is no such hold in place in and I'm following the instructions to a tee in a lab environment (preparing for real customer work).  It is seemingly a flawed process as of this time.

[Shaun Jennings]

@Jeremy Bradshaw  It should work. We did get one to finish properly (out of 4) but it has to be perfect and back to the same Exchange GUID. I have not tried on-premises recovery, but I'm sure it is the same thing. Is the main AD account restored with the same Object GUID as before?

[Jeremy Bradshaw]

@Shaun Jennings the on-premises original mailbox is soft-deleted (the ExchangeGuid in the error message is that mailbox's).  The original user in on-premises is now a RemoteUserMailbox and its ExchangeGuid has been set to match that of the EXO-based mailbox.  I'm doing New-MailboxRestoreRequest in EXO v2 PS like this:

New-MailboxRestoreRequest `
 - Name <PSmtp> `
 - BatchName "Migration via Restore" `
 - RemoteCredential $Credential `
 - RemoteHostName <on-premises EWS/MRS proxy FQDN> `
 - RemoteDatabaseGuid <on-premises MDB Guid> `
 - RemoteRestoreType DisconnectedMailbox `
 - SourceStoreMailbox <ExchangeGuid of soft-deleted on-premises MBX> `
 - TargetMailbox <ExchangeGuid of EXO MBX> [ENTER]

It seems like I'm following the instructions in the "How to recover when a mailbox...." article perfectly, with the exception that for step 7, I'm also including the Name and BatchName parameters.  Otherwise I've executed everything exactly as instructed.  It's really strange too whereas I can see in the report that the on-premises mailbox is connected to and its items are successfully copied to the cloud mailbox.

Going back to your question about the main AD account - it was never deleted from on-premises AD, nor from Azure AD, it was mailbox-enabled in EXO before EXO was aware that there was already a mailbox for it in on-premises.

[Shaun Jennings]

@Jeremy Bradshaw  I know this is a dumb question but does the on-premises Exchange GUID match the one on EXO?

[Jeremy Bradshaw]

@Shaun Jennings The on-premises mailbox has a different ExchangeGuid than the EXO mailbox.  The only place I can see this on-premises mailbox now is with Get-MailboxStatistics, and I'm using the "MailboxGuid" property for the -SourceStoreMailbox parameter on my New-MailboxRestoreRequest command.  MailboxGuid from Get-MailboxStatistics is the same as ExchangeGuid from Get-Mailbox (but in this case, I can't do Get-Mailbox since the mailbox has been disabled).

The original mailbox is currently disconnected/soft-deleted.  The original user has been Enable-RemoteMailbox'd and Set-RemoteMailbox'd with -ExchangeGuid <Exch.Guid from EXO>

So the answer is yes and no, ha ha. Let me know if I'm missing something though.

[Shaun Jennings]

@Jeremy Bradshaw That is where we had issues. If we had to restore to a different ExchangeGuid/MailboxGuid, it failed. If we restored to the same, it worked.

[Jeremy Bradshaw]

@Shaun Jennings I understand what you're saying, except I don't know how we can get them to match.  The one scenario I can think of, but haven't ever seen so can't really confirm is the 2 mailbox scenario where the 2nd or most recently-created mailbox is an erroneously created on-premises mailbox for an already-migrated mailbox, or a cloud mailbox where the on-premises Remote<User/Shared/Room/Equipment>Mailbox had its ExchangeGuid set to match the cloud mailbox (i.e. the scenario described in this blog post in the "Duplicate (new) on-premises mailbox" section.

I really do hope that the Exchange Team can come back and see our conversation here, and hopefully help us with the other scenario (which I argued in my earlier comment is WAY more common), where the 2nd/new mailbox is created in EXO due to PPL (premature pre-licensing).

With COVID-19's impact on companies adopting Teams, this dual mailbox problem scenario is getting to be very common.  The specific parameters of this scenario are:  Customers jump into Office 365 and license all their freshly synced users.  In that process, the purposely do not setup Hybrid Exchange, or enable the "Exchange Hybrid Deployment" optional feature in AAD Connect because aren't ready for EXO migration yet.

Even before COVID-19 though, I've seen very commonly that customers license new users too early, before the users are setup in on-premises with a mailbox or as a remote mailbox.

I feel like it may actually be worthwhile to open a user voice for a property on OrganizationConfig which can be set to turn off automatic EXO mailbox provisioning.  That or a default behavior of automatically disabling the EXO P1/P2 app from within O365/M365 licenses.  If I do, I'll come back here with the link to beg people for votes.

[bwinzenz]

@Jeremy Bradshaw @Shaun Jennings Sorry guys - I just got back from vacation and just now catching up. First things first...

Exchange guid does not have to match. Kind of the whole point of this remote restore concept is you are restoring from the on-prem disconnected mailbox into a *different* mailbox in EXO. I've tested it many times (and worked with a customer just today to do one!)

Secondly - remember that this is a "restore" request, not a "migration". If the restore request is "copying" items, I'm really not concerned about the errors referencing cannot find Exchange guid, or even if the restore request fails (at the very end). In the example provided, you can see that 799 items were successfully copied. That is really all that is important here. Yes that means there is likely room for improvement, and it is possible there could have been a code change on the EXO side that is causing the failure, but focus on the item copy. As long as the messages show up in the target (EXO) mailbox, everything worked.

Copy progress: 799/799 messages

 Secondly - regarding your scenario of premature licensing, the ONLY time there should be an issue with licensing creating a mailbox in EXO is if the user is licensed BEFORE the Exchange guid is synchronized. If a company is going to set up AAD Connect and synchronize users, they should be syncing Exchange attributes. It isn't that hard. It is just a check box in the AAD Connect wizard. They don't have to configure full Hybrid using the Hybrid Configuration Wizard. Just sync Exchange attributes.

Hope this helps - happy to have a further discussion offline if it makes more sense. Drop your email address here and I'll reach out.

[Jeremy Bradshaw]

@bwinzenz Thanks for responding and for the answer confirming that the successful / full count of copied messages is an indicator of success, at least in terms of what we're after here.

I do think that if that is how it is supposed to work, then the linked instructions should say so.  Otherwise, how would anyone not in the product engineering group know that this failure is nothing to worry about?  You must agree that not having this info as an outsider (me) substantially impacts my interpretation of whether or not I've succeeded with the process.

Regarding the next point, I agree that it's simple to understand how this stuff is working and how to avoid the issue.  So you're concluding that it is easy to not pre-license too early.  Well I'm concluding that it is extremely easy to pre-license too early, and so much so that it is highly likely to happen to most customers.  If customers create a new AD user (no Exchange tools involved), sync it to AAD, and then it gets licensed before they've touched the on-premises object with an Exchange tool, boom it's got an orphaned mailbox in EXO, as far as Exchange on-premises is concerned.  So yes your point is true that it is simple to avoid, but it's not obvious or advertised as a warning anywhere and every single customer has to go through this learning experience the hard way.

In summary, both your points are putting all onus on customers to just figure things out because they're 'simple' according to the Exchange product group.  Meanwhile, I think I've raise perfectly reasonable points that state the opposite.  Finally, to further strengthen my argument, particular to point #1 about the New-MailboxRestoreRequest failing and it still actually being a success, it seems a long way to go to create this blog post, only to not mention this crucial twist (that the job will fail, and that this is OK).  Not sure where to go from here, so I'll just say thanks for confirming about this, and hopefully something changes for the better about this process.

[bwinzenz]

@Jeremy Bradshaw I appreciate your comments. I do understand there can be circumstances where pre-licensing can end up with a duplicate mailbox. I will freely admit that having 20+ years of experience with Exchange (and Exchange Online) gives me somewhat of a biased opinion in terms of expectations of administrators and their understanding of how things work. In short, what I think is simple might not be so simple for others that don't have the same experience I do. Documentation can *always* be improved! :) If you have a suggestion for where a warning might be helpful, I can check around and see if that might be a possibility.

Regarding the restore failing, I don't want to state that it is expected to always fail. It normally won't. There may be a recent code bug, or there could be other extenuating circumstances that are leading to the failure you are seeing. When I've tested this previously, it doesn't fail. My point was simply to explain that with a restore, the important thing is to see that the items were copied. If you are seeing failures logged for restores, my suggestion would be to open a support case so we can look into it and see if there could be a code bug or other issue causing the failure.

[Jeremy Bradshaw]

@bwinzenz Thanks for the response.  Regarding where a warning might be placed - I'm not sure where would be best since it's a topic spanning AAD licensing, AAD Connect syncing, and everything around and in between.  Because of that (the difficulty of documenting this where the warning will have preventative value), I feel like it's actually something that could merit a feature addition, allowing hybrid Exchange customers to somehow turn off auto-provisioning of mailboxes until the on-premises steps are completed (e.g. either New-RemoteMailbox or Enable-RemoteMailbox, and sync of ExchangeGuid).  I can feel this one going the UserVoice path, as much as I fear that is a graveyard for unpopular-but-great ideas.

I will open a case for the failing New-MailboxRestoreRequest and see what can be found out there.  In case the error is common or expected under certain conditions, a warning note block could be added to the linked docs page from this blog post.  I'll report back once I've gone through support.

[Jeremy Bradshaw]

Hello again @bwinzenz , I've just finished working with MS Support on this issue where the New-MailboxRestoreRequest fails as shown in my earlier comment.  The case # was 21556719.  They were very friendly and helpful (as there was another subtopic on the go as well).  In the end, they could not confirm whether or the job should be considered successful or not in this situation.  Instead the guidance was simply to avoid this situation altogether by not doing things that render a mailbox in both on-premises and EXO.

I've now done this process in 3 different customer tenants, plus my lab tenant and the outcome is 100% consistent, always fails with FailedOther and the error message about the mismatch of ExchangeGuid's.  It seems like when the New-MailboxRestoreRequest Cmdlet was updated with the -Remote*** parameters, they didn't add logic to the actual working job to prevent this error.

I'm going to go now and submit the PR for the "How to recover" article so that it is at least documented based on the real world behavior of the process.

Thanks for the support earlier and also thanks @Shaun Jennings as well.

[Jack_Chen1780]

I know how to fix the problem now, but how can I avoid this happen again? what is the way to confirm Exchange GUID is synced from Azure to Azure AD for certain users ?

I found when I pre-licensing Exchange Online, some users with OnPrem mailboxes will get their online mailboxes created, but not all the users. I  have followed https://docs.microsoft.com/en-us/azure/active-directory/hybrid/tshoot-connect-attribute-not-syncing

and checked Azure AD Connect :

the user with duplicated mailboxes does have msExchMailboxGuid in synced attributes.

[HHantelman]

Hi I am running through these instructions on a handful of mailboxes. The steps and merge to EXO works like a charm. The question I have is what happens if they have an archive mailbox connected to their account? When I run Disable-Mailbox, it disconnects them both. Then I use the GUID info to run the restore into EXO mailbox. As far as I can tell this is only the primary mailbox data being restored. Is that correct? If so, how do I get their archive mailbox data into their new EXO archive too?