In Part 1 of this series, we talked about cross-tenant (sometimes referred to tenant to tenant or T2T) mailbox migrations. In Part 2, we’ll cover how to troubleshoot issues you may encounter during cross-tenant mailbox migrations. There are several tools we wanted to mention that can be useful when troubleshooting.
Here is a table with the key elements:
|
We cannot stress enough how important it is to ensure the configuration is correct. Ensuring this saves time! |
|
|
The most common errors in T2T migrations and how to address them. The errors are found using Test-MigrationServerAvailability or Get-MigrationUserStatistics / Get-MoveRequestStatistics. |
|
|
This simulates a move request cross-tenant and will identify configuration issues at user level. When you get the same error for multiple users, it is likely that configuration is incorrect at organization level. |
|
|
Powerful commands to troubleshoot your migrations and verify progress. The -IncludeReport report is basically a move report which shows all the things that happened in the migration. -DiagnosticInfo provides information on durations, throttling and possible underlying causes for stalls. |
|
|
The report will check user level organization level configuration for T2T migration so that you are prepared for migrations. |
|
|
Permissions related cmdlets: |
Know what permissions are migrated in Exchange Online and if they are broken. |
|
Sometimes, we cannot migrate all the data from source to target, so we skip it. There are 4 main categories of skipped items:
You can view what has been skipped (for example permissions that couldn’t be mapped to a user are considered bad items) in move reports. Sometimes, if there is significant data loss (not migrated), the admin will need to approve to complete migration. |
|
|
Duration estimates for T2T migrations based on data retrieved for 50% (P50) and 90% (P90) of the statistics seen for that migration type. If you exceed P90, your migration may be slow. |
Test-MigrationServerAvailability
A very useful tool when troubleshooting cross-tenant mailbox migration is Test-MigrationServerAvailability. You run this command in the target tenant after the migration endpoint has been created. You can run Get-MigrationEndpoint to view the identity of the endpoint.
Test-MigrationServerAvailability -TestMailbox <user@contoso.com> -Endpoint <T2T Migration Endpoint Name>
If the Result is Success, you can proceed with the migration of the user.
Note that a successful test doesn’t guarantee you will be able to migrate the user without any issues, but it is a good starting point to ensure the minimum prerequisites are met.
For a list of common failures, please see the Migration Failures section here: Cross-tenant mailbox migration
Here is an example, a situation where the source archive mailbox is a few bytes over 100GB.
Test-MigrationServerAvailability in target tenant:
Mailbox statistics for archive in the source tenant:
In situations where a source primary, archive, or dumpster is larger than the target tenant’s user quotas you will see errors like ArchiveExceedsTargetQuotaPermanentException, MailboxExceedsTargetQuotaPermanentException, or ArchiveDumpsterExceedsTargetQuotaPermanentException. In this case, you can contact Microsoft support for recommendations and see which options are best for you. Support may be able to provide an exception for an individual mailbox (we cannot process bulk mailboxes) and if you plan on having auto-expanding on the target tenant side and allocate room to grow in the target tenant. Depending on the situation and if you have enabled the auto-expanding archive feature on the source tenant, you can wait until the AuxArchive is provisioned and MainArchive / primary mailbox size is reduced below the 100GB quota (this can take up to 30 days but it’s generally 7 days on average). Or, if possible, ask the user to clean up unnecessary data in the source mailboxes if auto-expanding is not an option in the target tenant. Sometimes, source mailboxes that are or were on hold at some point, have Recoverable Items quota set to 100GB as opposed to the default 30GB, and in this case, you might just want to enable litigation hold on the target tenant mail user to also increase the quotas there.
Coming back to Test-MigrationServerAvailability, if this fails for multiple users, then likely there is something wrong at the organization level and the cross-tenant mailbox migration validation script discussed in the next section can be helpful. For example, if you get an ‘Access is denied’ error (screenshot below) for some or all users, here are some possible causes at organization level:
- Erroneous application registration;
- Wrong credentials or expired credentials used in the migration endpoint; or
- Source tenant organization relationship’s OAuthApplicationId does not match target migration endpoint's ApplicationId, or the org relationship is not using tenant ID in DomainNames
CrossTenantMailboxMigrationValidation script
When it comes to validating a large cross-tenant mailbox migration (CTMM), a better tool you can rely on is the Cross-Tenant Mailbox Migration validation script written by Alberto Pascual Montoya, published in the official CSS-Exchange GitHub repo and referenced by our official CTMM documentation.
This script will check and validate various related user and organization settings. It is very useful to run it before CTMM takes place to ensure the mailbox(es) can be migrated, and if they can, that they won’t face any validation issues during the move.
You can run the script to validate the configuration between both organizations by running:
.\CrossTenantMailboxMigrationValidation.ps1 -CheckOrgs -LogPath <LogFileLocation>
You can also run it to validate the objects on the target tenant by comparing them with the objects in the source tenant by running:
.\CrossTenantMailboxMigrationValidation.ps1 -CheckObjects -CSV <CSVFileLocation> -LogPath <LogFileLocation>
As you can see below, it will highlight any discrepancies found, and if the target object is not synced from any AD, it will ask if you would like to correct the discrepancy on the go:
There are other scenarios covered by the script too. For example, running it only from the source tenant side and collecting the needed data that can be sent to the target tenant admins and they run the script against their tenant; or simply collecting the data for support purposes.
Analyzing migration reports
Move request and migration user Exchange Online PowerShell commands are other tools that can help us troubleshoot cross-tenant migration.
Let’s do a quick walkthrough of these commands:
When we create a new migration batch, in the background, a Get-MigrationUser object is created for each user specified in the CSV file for migration. If things go well during a partial validation, a Get-MoveRequest is created for each MigrationUser.
In the example below, we can see that we have a migration batch (Get-MigrationBatch) with 1 user (one user was specified in the CSV file) and a migration user object (Get-MigrationUser) was created for the user, but because we don’t have a mail user with this identity, we failed the migration at this early stage. The move request was not created (Get-MoveRequest fails).
This user is not found at all on the target tenant (wrong identity of the user in the CSV):
And if I had run Test-MigrationServerAvailability on the EmailAddress specified in the CSV, before creating the batch, the error would have been relatively obvious:
Also, you’d want to ensure that the users specified in the migration CSV file don’t have a move request already (or they are not a part of another batch).
There might be situations where you might see that migration is stuck at creating the move request, that is in injection workflow stage:
If State: Waiting takes too long, you can look at DiagnosticInfo on Get-MigrationUserStatistics (check the Durations section in this blog post for more info) to see if you have more information on where migration is stuck.
In this case, we can see that it waited 19 minutes to inject the move request:
And 0 move requests injected successfully since we have 0 InjectedRequestCount and no timestamp at LastSuccessfulInjectionTime or LastInjectionTime.
After waiting 19 minutes for the injection of the move request, we finally got a permanent failure: ErrorCrossTenantSourceUserIsInHoldOrRetentionPolicyAppliedPermanentException, and the service gave up trying to inject the move request. The lesson is to check Test-MigrationServerAvailability before creating the batch (prerequisite missed) and avoid this waiting time and failure that could have been easily detected.
You might wonder when we use Get-MigrationUserStatistics versus Get-MoveRequestStatistics. It mostly depends on preference. I am more used to Get-MoveRequestStatistics, but in the situation where we don’t have a Get-MoveRequest created (like in the example above), we are forced to use Get-MigrationUserStatistics. Normally I append -DiagnosticInfo “verbose,showtimeslots” to make sure I get the most details. Also, if you are looking to check skipped Items, you will append -IncludeSkippedItems to Get-MigrationUserStatistics. If using Get-MoveRequestStatistics, it is enough to use -IncludeReport.
Let’s take another example where a move request was injected successfully by the service (New-MoveRequest) but this one failed. We will troubleshoot using the Get-MoveRequestStatistics command (since we have a Get-MoveRequest in place):
Move requests in cross-tenant migrations will have Onboarding_CrossTenant WorkloadType and the SourceServer and TargetServer are Exchange Online servers, usually in different forests.
In this case, the move request failed because we don’t have the Cross-tenant User Data Migration license on source or target tenant.
Get-MoveRequestStatistics EXOMailbox1 |FL MailboxIdentity, WorkloadType, SourceServer, TargetServer, RemoteHostName, FailureType, Message, FailureTimestamp
If I want to check the full details of the failure, I can run a command like this:
$stats = Get-MoveRequestStatistics <User> -IncludeReport
$stats.Report.Failures
If you’d want to have a quick overview of failures encountered, you would run:
$stats.Report.Failures | group Failuretype | FT -a
To check if any items would be skipped during the move, you can run the following command:
Get-MoveRequestStatistics <User> | FL DataConsistency*, BadItem*, LargeItem*, MissingItem*, Skipped*
For more information on DCS (Data Consistency Score) please see Improving Migrations Using Data Consistency Scoring and Track and prevent migration data loss. For troubleshooting DCS and approving skipped items, please see Migrations with Data Consistency Score (DCS) – more than you ever wanted to know! - Microsoft Community Hub.
If you need to look at the available properties of the source or target user before and after the move, you can use the IncludeReport switch in Get-MoveRequestStatistics.
Run the following in the target tenant after move is completed:
$stats = Get-MoveRequestStatistics <User> -IncludeReport
Then you would look into the properties like this:
$stats.report.SourceMailboxBeforeMove.Props
$stats.report.TargetMailUserBeforeMove.Props
$stats.report.SourceMailuserAfterMove.Props
$stats.report.TargetMailboxAfterMove.Props
Suppose that we want to see if migration service stamped the ExternalEmailAddress on the source recipient after the move (according to the TargetDeliveryDomain set in the migration batch), we would run this command:
$stats.report.SourceMailUserAfterMove.Props | ? PropertyName -eq "ExternalEmailAddress" | FL PropertyName, Values
Troubleshooting migration of permissions
Let’s teach you how to examine migration reports to see if permissions were migrated correctly and if there are any permissions errors in move reports. We provide PowerShell commands for the target tenant to help corroborate outputs with the migration reports analysis.
We will first check mailbox Full Access permissions, which should move during the cross-tenant migration, and we check them by using the move reports and PowerShell commands.
In this example, we moved MyMailbox10 and MyMailbox19, from one tenant to another:
We can see that MyMailbox10 has maintained the FullAccess permission on MyMailbox19 after migration:
We can confirm this by looking at the property called ExchangeSecurityDescriptor (which is MsExchMailboxSecurityDescriptor in AD) and noticing the SID values in there.
Note: you first need to store the Get-MoveRequestStatistics <user> -IncludeReport into a variable, in this example $stats19.
$stats.Report.SourceMailboxBeforeMove.Props | ? PropertyName -eq "ExchangeSecuritydescriptor"
$stats.Report.TargetMailboxAfterMove.Props | ? PropertyName -eq "ExchangeSecuritydescriptor"
- Source mailbox: before move the SID belongs to MyMailbox10 in the source tenant.
- Target mailbox: after move the SID belongs to MyMailbox10 in the target tenant.
If you have multiple SID values listed here, you can use these commands to display them in a more readable format:
$valueAfterMove = ($stats19.Report.TargetMailboxAfterMove.Props | ? PropertyName -eq "ExchangeSecurityDescriptor").values.Strvalue
$sdobj = New-Object System.Security.AccessControl.RawSecurityDescriptor($valueAfterMove)
$sdObj.DiscretionaryAcl | select -Skip 1 |FT AceQualifier,AccessMask, SecurityIdentifier, AceFlags, IsInherited
The AccessMask 1 here means Full Access Permission:
The same information is also available in the DebugEntries in the move report when we look for “MailboxSecurityDescriptor’.
$stats.report.DebugEntries | ? LocalizedString -match "MailboxSecurityDescriptor"| % {[string] $_}
Besides Get-MailboxPermission cmdlet, we can look at the ExchangeSecurityDescriptor property with the Get-Mailbox command:
get-mailbox mymailbox19 | select -ExpandProperty ExchangeSecurityDescriptor | select -ExpandProperty discretionaryACL
A quick reminder that Send on Behalf permissions are NOT migrated, so we won’t see anything in move reports.
Send As permissions
The Send-As permissions are migrated. In target tenant, after migration:
You can similarly check the permissions in the source tenant.
When checking the move reports for Send-As permission, you can put the DebugEntries in a Notepad++ file and search for the user’s SID (source user SID and target user SID). These permissions would fall into the category UserSecurityDescriptor. You might see many AD permissions (screenshot below where entries were separated by new line), so it can be quite hard to spot them.
If you want to list all SIDs with Send-As, you can search specifically for ‘CR’ and the following GUID: ab721a54-1e2f-11d0-9819-00aa0040529b.
$stats.report.DebugEntries | ? LocalizedString -match "UserSecurityDescriptor"| % {[string] $_} | clip
Then paste from clipboard to Notepad++, for example:
To have them listed on new lines, you can do the following:
(OA; replace with \n(OA;
(A; replace with \n(A;
(OD; replace with \n(OD;
These are called Ace Types, for a full list see documentation.
Then, since Send-As is an Extended Right, we will see CR ab721a54-1e2f-11d0-9819-00aa0040529b, which is the SendAs Extended Right GUID.
Checking the first SID in the source tenant:
Checking the first SID in the target tenant:
On-premises, looking at the LDP dump of the security descriptor of the user, you would see something like this:
Mailbox folder permissions
We also migrate mailbox folder permissions.
In the target tenant, after migration, you will run Get-MailboxFolderPermission <user>:\<Folder> to check permissions present. For example:
Based on this output alone, we would think that there are no permissions issues.
But expand the User property and notice that Cloud3 is of Unknown user type:
And, if we check the move report, we will see TargetPrincipal errors and FolderACL issues for cloud3.
In this case, the DataConsistencyScore is usually Good instead of Perfect, and we will have corresponding BadItems and Failures recorded if we don’t find the principal user with permissions, in the target tenant.
Here is a quick command to see the alias of the principal user that is not found in the target tenant (Cloud3 in our example).
$stats.report.BadItems | select Kind, Folder, ScoringClassifications, {$_.Failure.DataContext} | ft -a
We can further look into DataContext with |FT or |FL for full details:
$stats.report.BadItems | select {$_.Failure.DataContext} | ft -a
Or look at the entire failure on specific bad items. BadItems[1] is the second CorruptFolderAcl in the screenshot above (the first line count starts from 0, not 1).
$stats.report.BadItems[1].Failure
For migration of Calendar and FreeBusy data folder permissions entries, we can look at folder ACL in Report.DebugEntries:
$stats.report.DebugEntries | ? LocalizedString -match "FolderAcl"| % {[string] $_}
Checking migration durations
Now that you rock at troubleshooting migration of permissions (which was lengthy and a bit boring), we will get into another topic: duration of the cross-tenant migrations and when to recognize you have an issue.
To see how much time your migration took and how it is progressing, you can run these commands:
$stats = Get-MoveRequestStatistics <User> -IncludeReport -DiagnosticInfo “Verbose,showtimeslots”
$stats |FL *duration*
In my example, I see that overall duration was about 30 minutes (it was a very small 35 MB mailbox) but I have 51 days of TotalFailedDuration:
If I check the $stats.DiagnosticInfo property, we will see these durations in a more detailed and accurate way:
Overall Move = 57 days (about 2 months) and 28 min, out of which:
- In Progress = 6 min (actual copying of the data)
- Suspended = 21 min (I chose to complete it at a certain point after initial sync, so it went into suspended mode)
- Failed = 51 days and about 8 hours (it was in a failed state for 51 days)
- Completed = 5 days and 16 hours (the move completed 5 days ago but I ran the Get-MoveRequest command now, 5 days after completion)
You can check the table from Microsoft 365 and Office 365 migration performance and best practices | Microsoft Learn for duration estimates during mailbox migrations in Exchange Online.
For example, a mailbox with a size of less than 10 GB is estimated to be migrated within 1 day. You can open a case with Microsoft Support if P90 is exceeded and it is because of our service (for example it is not a configuration issue on your side that wasn’t remediated fast enough).
In my situation above, I had a permanent failure, and left the move sit there for 51 days in a Failed state. This doesn’t count against the P90, as I neglected the move. The InProgress duration was 6 min for my 35MB mailbox which meets the P90 estimation (90% of mailbox migrations less than 10GB would complete in 1 day).
We would like to thank Anshul Dube, Roman Powell and Nino Bilic for contributing and reviewing this blog post.
Mirela Buruiana and Alberto Pascual Montoya
Microsoft
