Migrations with Data Consistency Score (DCS) – more than you ever wanted to know!

[The_Exchange_Team]

Data Consistency Score (DCS) is a (somewhat) new feature for Office 365 migrations that scores the fidelity of migrated data, allowing admins to identify inconsistencies and integrity issues between source and target data when performing a migration to or from Office 365 (onboarding and offboarding). DCS is meant to replace the existing Bad Item Limit and Large Item Limit (BIL / LIL) model and related shortcomings.

The DCS model identifies and tracks data that cannot be successfully migrated from an on-premises source environment to Office 365 (e.g., corrupt data, items larger than the service allows, or data that is found to be missing in the target but present in the source). The DCS model examines this data based on quantity (count of the items that cannot be migrated and must be skipped by the MRS service) and quality or importance (DCS differentiates between user data and metadata / system data). We then calculate a score based on the total amount of data that might be skipped during migration and how significant it is for the user.

This means that the admin is no longer required to guess the number to use for Bad Item Limit (BIL) or Large Item Limit (LIL) in advance, as the DCS mechanism takes care of this automatically. The admin is therefore better informed of the possibility of data loss during migration and can take necessary actions when appropriate. DCS allows admins to only have to deal with approvals at the end of the migration, when they have all of the information that they need to make an informed decision. Previously, there was a hard choice to make between data resiliency and efficiency. The best option for data resiliency was to interrupt the admin every time the migration process encountered an item that could not be migrated and request approval to continue. But the best option for efficiency was for admins to try to guess a number of items that they could skip without any regard to the scenario behind skipping them, and then hopefully remember to come back afterwards to review what was skipped. Avoiding carte blanche BIL and LIL values encourages the admin to look at the information they have about the items that could not be migrated and come up with their own policies for next steps, such as what to approve, what to work around, what to tell their users, and what to escalate to support.

We are constantly tuning the various thresholds used to determine the DCS to ensure that problematic data loss does not occur during migrations. Exchange Online administrators do not have access to thresholds details (this is done service side).

Here are a couple of examples of bad item types that are not considered as impactful and will trigger a DCS of “Good” (instead of “Investigate”) when encountered:

• Orphaned delegate permissions where the user was removed from Active Directory (for example, the delegate left the company) but the delegate’s permission was not removed from the mailbox being migrated and the permission cannot be mapped to a valid user account. We will dig more into this later in the article where we discuss source/target principal mapping exceptions.
• Corrupted Search Folder criteria where you might see something like this in the Bad Items failure: “Unable to set restriction on table, Operation: StorageDestinationFolder.ApplyRestrictions”

The DCS model tracks 3 main categories of items that would be skipped by the MRS in Exchange Online:

• Bad items (corrupt data / properties of the items or orphaned permissions);
• Large items (for example, in a Hybrid migration, the maximum item size is 150MB; an item above this size is considered too large for the Office 365 service and cannot be migrated to Exchange Online. In an IMAP migration or public folder migration, the default MaxReceiveSize is 35MB for Exchange Online user mailboxes and public folder mailboxes (a size that can be increased up to 150MB); and
• Missing items (for example, in an IMAP migration where a user or automated process like MRM retention policies or a mobile device (wiping data) would delete (or move data to archive mailbox or even to another folder in the same primary mailbox) from the live target mailbox to where you are migrating, but the migration process is still syncing from the source IMAP mailbox). See this reference here on Missing Items.

As a good practice, before starting a migration, you can check for items that might cause issues. Here are some suggestions:

• For mailbox integrity checks: see New-MailboxRepairRequest and our Troubleshooting Failed Migrations blog post;
• For finding large Items: you can use a script similar to the following:

Get-Mailbox -ResultSize Unlimited | Get-MailboxFolderStatistics -IncludeAnalysis -FolderScope All | Where-Object {(($_.TopSubjectSize -Match "MB") -and ($_.TopSubjectSize -GE 35.0)) -or ($_.TopSubjectSize -Match "GB")} | Select-Object Identity, TopSubject, TopSubjectSize, DisplayName | Export-CSV -path "C:\temp\report.csv" -notype

And set, for example, the TopSubjectSize to 35MB or 150MB (depending on the type of migration and the MaxReceiveSize on the Office 365 mailbox). For example, public folder migration will be limited by MaxReceiveSize that you see on the PF mailbox via Get-Mailbox -PublicFolder <Mailbox01> | FL MaxReceiveSize, and by default it is 35MB. For a hybrid migration or PST import, the limit is 150MB.

• For corrupt permissions (pretty common among bad items) there is this document/script: Who is linked to whom in your spider web of mailbox permissions. Migration batch permissions analysi...
• For Calendar items inconsistencies, CalCheck is the right tool to use.

If you didn’t check these prior to migration, the good news is that DCS will detect the bad/large/missing items and you can identify them from the migration reports and migration GUI.

Transitioning to DCS

While the new DCS feature was introduced, DCS and BIL/LIL models coexisted to avoid breaking any scripts in use by admins. When one would specify a BIL or LIL, the old model would be triggered which prevented the use of DCS.

If you still see or use BIL/LIL, we recommend not using them and instead using DCS. DCS will take the guesswork out of migration and will help you avoid large data loss (in cases where BIL/LIL parameters are set very high).

Once a batch is submitted without BIL/LIL, that batch will use the DCS method and cannot be switched to use BIL/LIL. This means that even if you set bad item limit when DCS is in Investigate status due to bad items, the manually set limit will be ignored, and you still need to approve the items flagged by DCS. Similarly, batches submitted with BIL/LIL cannot switch to use DCS.

Note that the BIL and LIL parameters will eventually be replaced by DCS.

In what types of migrations is DCS used?

DCS supports the following types of migration to Exchange Online:

• Mailbox migrations (hybrid onboarding and offboarding, cutover, staged, IMAP, Google Workplace (G-Suite), cross-tenant MRS moves);
• Public folder migrations; and
• Mailbox restore requests.

At this time, PST imports don’t use DCS, but they will in the future. This will have no impact on PST import when using DCS, as admins are not required to approve skipped items when importing data from a PST file.

Admins are prompted for intervention and approval when DCS status is Investigate and the admin is performing a hybrid migration, Google Workspace (G-Suite) migration, or public folders migration. These are the only migration types that need completion (Complete-MigrationBatch); thus, for such migrations where we have a risk of data loss admins must approve to complete the migration.

As a side note, the DCS model and especially DCS failures are frequently seen in the context of mailbox restores in Exchange Online, but this is expected and you don’t need to worry about it. There is more information on this later in the article.

More information on DCS can be found here:

• Improving Migrations Using Data Consistency Scoring
• Track and prevent migration data loss

DCS says Investigate? Let’s investigate!

Now let’s focus on troubleshooting low scores; that is when things are bad and significant data is skipped, and how admins can overcome this.

With regard to how we calculate the score for migration batches, the DCS of the batch is equal to the worst DCS of any user within the batch. This behavior helps administrators know immediately whether there is data loss that should be investigated.

We will focus on an “Investigate” score as this is within the admin’s control, whereas for a “Poor” score you will have to contact Microsoft Support for assistance. In general, you don’t need to worry about a Poor score because they are rare.

When you get a score that is too low (Investigate) on migration, you may see a similar error in the move request failures in PowerShell or the keyword “Investigate” and in the Skipped item details view in the classic Exchange admin center (EAC) you might see “DataConsistencyTransientException: The data consistency score (Investigate) for this request is too low…”

If the migration receives a grade of Investigate, you have a few options:

1. Be aware of the skipped items, and communicate with the affected user;
2. Approve skipped items manually to allow the migration to succeed (hybrid, Google Workplace (G-Suite) and PF migrations); or
3. Correct the situation and re-migrate or resume/retry when possible.

Checking the skipped items

Like the score says, we should Investigate skipped items. There are 2 ways to retrieve the skipped items: (1) from the classic Exchange admin center – we will focus on that in this post; and (2) with Exchange Online PowerShell.

Classic Exchange Admin Center

Here is a reference of the Migrations tab in the classic EAC using a public folder migration example:

Mailbox offboarding example:

Mailbox onboarding example:

View details of the migration batch and skipped item details:

Let us comment a bit on the screenshots above for onboarding scenario:

• There is a migration batch called “DCS” with a type of “Exchange Remote Move” (hybrid migration) which has an Investigate score.
• The batch contains only 1 migration user (DCStest) with the Investigate score.
• In “Skipped Item details” view, we see 4 items that were skipped.
• The “Subject” column is very useful in identifying those large items that were skipped; in real life, you should be able to find items based on this.
• The column “Kind” indicates the type of the skipped item (bad / large / missing) and not the “kindness” or “niceness” level of the item (heh).
• The “Scoring classifications” indicates that these are IPM.Notes that are Large items type and the Large item that is found in Inbox is considered more significant than the ones in Sent items or Outbox (as seen in “Folder Name” column).
• Speaking of “Folder Name” column, in this example, all the folders where these large items reside (Inbox, Sent items, Outbox) are categorized as Well-Known Folders types (WKFType) because these are the default folders in a mailbox. However, if there was a large item in a user-created folder (e.g., a custom folder called “Training”), the folder name might not be as user friendly since in some cases a hexadecimal folder ID is displayed.

NOTE: In a situation where we have Missing Folders, the items that were inside of these folders are not counted individually against Items Skipped Column.

Example: in these 114 skipped items, the 3 missing folders are counted as 3 items.

PowerShell log collection and analysis

In this section we will show you how to retrieve the Skipped Items using PowerShell for your scenario (migration, PST import).

First, generate XML reports for the move request / sync request / mailbox import request / public folder mailbox move request or migration user statistics or a restore request to analyze. Note that you will need to connect to Exchange Online PowerShell and have the appropriate permissions to export these reports. If engaging with Microsoft Support, please provide these XML reports so that we can have a better understanding of your problem.

Migration user for Cutover/Staged/IMAP/Google Workplace (G Suite)/Public Folder/Hybrid migrations

Get-MigrationUserStatistics <user@contoso.com> -IncludeSkippedItems -IncludeReport -DiagnosticInfo verbose | Export-Clixml C:\Temp\EXO_MigrationUserStatistics_User1.xml

Hybrid moves

Get-MoveRequestStatistics user@contoso.com -IncludeReport -DiagnosticInfo verbose | Export-Clixml C:\Temp\EXO_MoveRequestStatistics_User1.xml

IMAP/Google Workplace (G Suite) migrations

Get-SyncRequest -Mailbox <user@contoso.com> | Get-SyncRequestStatistics -IncludeReport -DiagnosticInfo verbose | Export-Clixml C:\Temp\EXO_SyncReqStats.xml

Public folder migrations

Get-PublicFolderMailboxMigrationRequest | where {$_.TargetMailbox -eq "<Mailbox1>"} | Get-PublicFolderMailboxMigrationRequestStatistics -IncludeReport -DiagnosticInfo verbose | Export-Clixml C:\Temp\EXO_PFMailbox1.xml

PST imports using Office 365 import service

$i = Get-MailboxImportRequest -Mailbox <user@contoso.com> | Select Mailbox, RequestGuid, Status
$path = ‘C:\temp’
$i | foreach { Get-MailboxImportRequestStatistics $_.RequestGuid -IncludeReport -DiagnosticInfo verbose | Export-Clixml "$path\EXO_Import_$($_.Mailbox)_$($_.RequestGuid).xml” }

Mailbox restores

Get-MailboxRestoreRequestStatistics <user@contoso.com> -IncludeReport -DiagnosticInfo verbose | Export-Clixml C:\Temp\EXO_RestoreReq.xml

Checking the logs

Import the XML data into a variable. For example:

$ustats = Import-Clixml C:\Temp\EXO_MigrationUserStatistics_User1.xml
$r = Import-Clixml C:\Temp\EXO_MoveRequestStatistics_User1.xml

Whether we are looking at the migration user statistics or move request statistics, we can retrieve the logs to see if we have any large/bad/missing items encountered, if we have any limits set (BIL/LIL), information related to data consistency score, and the factors causing the low scores.

You can either use the XML files or you use Exchange Online PowerShell without exporting to XML. For example:

• Migration User Statistics: $ustats=Get-MigrationUserStatistics user@contoso.com -IncludeSkippedItems -IncludeReport -DiagnosticInfo "Verbose"
• Move Request Statistics: $r= Get-MoveRequestStatistics user@contoso.com -IncludeReport -DiagnosticInfo Verbose

For Migration User Statistics we will use the $ustats variable to extract the information and that will shed more light on our investigation.

To check the DCS value and why the score is Investigate, use this command:

$ustats| fl data*

The above output shows that the score is Investigate because of large item(s) found in the mailbox.

To check how many items are skipped, use this command:

$ustats | fl *item*

Based on the screenshots above, we can see that:

• We have no large/bad items limits set (we are using DCS).
• We have 2 skipped items.
• We did not approve the skipped items yet.
• We have a preview of skipped items.
• We have a score of Investigate due to large items encountered.

To investigate further, run the following command:

$ustats.Report.LargeItems | fl Kind,Folder,Subject,MessageSize,ScoringClassifications,DateSent,DateReceived,Failure

For bad items, run a similar command:

$ustats.Report.BadItems | FT foldername,subject,failure

We can see all the bad items grouped by their type, using this command:

$ustats.report.BadItems | group kind

As you can see, every large/bad item has a failure associated with it.

To get more information on a specific bad/large item failure, use the following command (where 0 is the index of item):

$ustats.Report.LargeItems[0].Failure

Or you can check the failures report:

$ustats.Report.Failures[0]

The following command should index all the failures so you can easily check the details of a specific failure:

$i=0;
$ustats.report.Failures | foreach { $_ | Select-Object @{name="index";expression={$i}},timestamp,failurecode,failuretype,failureside;$i++} | ft

To see all failures grouped by their type, use this command:

$ustats.Report.Failures | group FailureType | ft -AutoSize

The same information about large/bad items that were skipped can be seen in skipped items:

$ustats.SkippedItems | group Kind

To get a quick overview of the skipped items, run this command:

$ustats.SkippedItems |ft Kind,Subject,MessageSize,ScoringClassifications,DateSent,DateReceived,FolderName,Failure -AutoSize

We can see some of the information from command $ustats | fl *item*, data* also in SubscriptionSnapShot in DiagnosticInfo :

$ustats.DiagnosticInfo

Now that we have checked the Migration User Statistics, let’s see what we have for our Move Request Statistics. Please keep in mind that most of the commands used above can be also used for the $r variable as well.

$r=Get-MoveRequestStatistics dcs@contoso.com -IncludeReport -DiagnosticInfo "verbose"

Check what failures are present for the migration; for this demo, we took as an example the move request statistics XML file. This command should provide a list with all errors and the count (number of bad/large items that have an associated failure).

$r.Report.Failures | group FailureType

To check the entire message for a failure, take them one by one. Normally, the last encountered error is a PermanentException:

Index Failures:

$i=0;
$r.report.Failures | foreach { $_ | Select-Object @{name="index";expression={$i}},timestamp,failurecode,failuretype,failureside;$i++} | ft

Get more details about one of the failures using the index number:

$r.Report.Failures[0]

Let’s take another example of DiagnosticInfo in which we will do a breakdown of all parameters:

Optionally, you can copy paste this section of <SkippedItemCounts> </SkippedItemCounts> to an XML file editor to see it more clearly:

In the report above, we have 15697 Corrupt items and 11 Large Items.

The Corrupt Items are the following: 4 calendar, 11 large items / user created (probably a custom form), 7 messages, 15695 bad folder permissions, 2 folder rules.

Focusing on majority of skipped items (FolderACL) most of them (12642) are Source Principal Mapping errors and the rest (3055) are Target Principal Mapping errors.

In short, the Source Principal Mapping Exception means that the user with permissions over the user mailbox which is being migrated is not found on the source side (on-premises Exchange / AD in an onboarding scenario). The user(s) with permissions that were tied to some SIDs are not found in AD and their related SIDs are orphaned SIDs since we cannot map them to the associated users that had permission over the mailbox being migrated. You can check / clean these permissions on-premises, but this is not normally impacting migration as they are skipped anyway (we assume that permissions are not needed since those users cannot be found).

The Target Principal Mapping Exception means the user with permissions over the user mailbox that is being migrated is not found on the Target (Office 365 in the case of onboarding). This can be the case if the user is not synced from on-premises to Office 365.

These permission failures (Source/Target Principal Mapping Exceptions) will typically not cause a score of Investigate. We will get migration failed when reaching 100,000 corrupt ACLs.

More details on Source and Target Principal error can be found here.

When it comes to the Missing items we can use $stats.MissingItemsEncountered or we can check the MailboxVerification report by using this pretty long command:

$ver = $r.report.MailboxVerification
$ver | ? {$_.FolderTargetPath-match $folderName} | select @{Name="MissingItemInTargetCount"; Expression={$_.MissingItemsInTargetBucket.BucketCountAndSize.Count}} , @{Name="ItemCountBeforeMove"; Expression={$_.Source.Count}}, @{Name="ItemCountAfterMove"; Expression={$_.Target.Count}},WKFType, FolderTargetPath | ?{$_.ItemCountBeforeMove -gt 0 -or $_.ItemCountAfterMove -gt 0} | ft –a

To take a peek at some samples that were found to be missing in the target, use the command in PowerShell or Exchange admin center (Items Skipped):

$r.Report.MailboxVerification.MissingItemsInTargetBucket.DivergentItemsSamples | ft Kind, Subject, WellKnownFolderType, DateReceived

If you've analyzed these reports but still require assistance from Microsoft Support, please open a support case and send us the relevant logs for your scenario.

Poor score can happen

Note that this example is a manufactured one, created intentionally for the purpose of this blog post. In real life, if you get a Poor score for DCS, please raise a support case with Microsoft. We don’t allow completing the migration with a Poor score; when you try to approve / complete, you will see a warning similar to the one shown below and will need to contact Support.

Here we have a migration batch with “Poor” score: 1 Migration User (Mailbox1) has Poor Score, the other two users have Perfect Score. Batch score is Poor, based on the lowest score:

A quick view of the DCS on the Move Request Statistics for Mailbox1:

If you try to approve / complete this batch, you will get the warning because of Mailbox1 Poor’s DCS score:

If you click on Yes, you will see the following:

In the end, the Batch is “Completed with errors” and you need to contact support for the Migration User where the DCS is Poor if you want to complete that migration.

Approving skipped items

Now that you are aware of the items skipped and why we have the Investigate score and considering you can go ahead and complete the migration, you would need to Approve these skipped items.

Let’s say that you are doing a hybrid migration using migration batches and you have received a score of Investigate; you would run the following commands in PowerShell to approve the finalization of the mailbox migration or approve them through the classic Exchange admin center (easier way):

Set-MigrationBatch -ApproveSkippedItems or Set-MigrationUser -ApproveSkippedItems

If you are using MoveRequests directly, without using Migration Batches (applicable to hybrid moves), then run:

Set-MoveRequest -SkippedItemApprovalTime $([DateTime]::UtcNow)

If you are using hybrid remote moves with migration batches, we don’t recommend making changes on the individual moves as you will have inconsistency between batch / migration user and the move request status and you might also encounter an error related to SkippedItemApprovalTime being in the future (ApprovalTimeInFutureException) - depending on the regional settings on the admin who created the batch. If you still want to approve skipped items at the move request level and not batch / migration user level and you get the time error, you can try using this command (setting 5 minutes in the past or more):

Set-MoveRequest -SkippedItemApprovalTime $([DateTime]::UtcNow.AddMinutes(-5))

Note: For hybrid migrations, public folder migration, Google Workspace (G-Suite), approval is required.

How to see if we approved?

There might be situations when admin approved the skipped items but there are still Skipped items pending approval.

In DiagnosticInfo of the MRS request statistics (for example GetMoveRequestStatistics for hybrid and Get-SyncRequestStatistics for IMAP), you will see <timestamps> under <TimeTracker> for the request. In the example below we can see that the admin approved skipped items two times, first on October 25th but as we last encountered a skipped item 2 days later, the admin would need to approve again, and this was done on November 2nd.

<Timestamp Type="SkippedItemApprovalTime" Time="2020-11-02T10:34:12.0507654Z" />
 <Timestamp Type="LastSkippedItemEncounteredTime" Time="2020-10-27T17:09:42+00:00" />
<Timestamp Type="SkippedItemApprovalTime" Time="2020-10-25T07:37:25.4262119Z"></Timestamp>
 <Timestamp Type="LastSkippedItemEncounteredTime" Time="2020-10-27T17:09:42+00:00"></Timestamp>

A note about mailbox restores and DCS

Mailbox restores often fail during mailbox verification because the target mailbox is a "live" mailbox. When other clients move items as they are being restored, it prevents MRS from being able to find these items during mailbox verification. Previously, restores would have failed with TooManyMissingItemsPermanentException or TooManyBadItemsPermanentException. Now they fail with DataConsistencyPermanentException instead. Restores do not stop their progress when these errors are hit – they are only flagged at the very end of the restore. This means that all of the content has already been restored by the time you see this message. This is really informational that items were moved during the restore. It does not require any action on behalf of the admin. You can see which folders were missing the items by getting the restore report and looking at the MailboxVerification section.

$stats = Get-MailboxRestoreRequestStatistics -Identity <id> -IncludeReport

This command will show you which folders didn't have the same count between source and destination (thus, MRS counted them as missing).

$stats.Report.MailboxVerification | where { $_.Source.Count -ne $_.Target.Count }

I would like to thank you for reading this article and many thanks to the contributors to this blog post: Mirela Buruiana, Sruthi Sreeram, William Rall, Angus Leeming, Nino Bilic, Brad Hughes, Zacharie Zambalas.

Mihai Grigoruta

[Sam_T]

@The_Exchange_Team 

I scrolled though the 25? pages of this article and I'm not sure I see the value.

Every mailbox migration of a significant size where the systems have been in place for a while has "corrupted" items that do not get migrated. But if you are using well maintained Microsoft Exchange servers, Microsoft Outlook clients and Microsoft-supported OWA clients, the responsibility for the "corruption" can only lie with Microsoft for failing to maintain the integrity of these items and allow them to reach the nebulous state of "corrupted". There is no option that I have seen to check for "corruption" prior to the migration and no Microsoft mechanism that I have seen to repair the "corruption" induced by Microsoft apps.
The net result is that admins end up having to either set the "bad" item limit to a higher number to avoid the Microsoft induced "corruption", then let the mailbox migrate or set the "bad" item limit to zero, let the migration fail, then review the logs for any "corrupted" items and go back and address them individually in each users mailbox. The rerun the repaired mailbox. That's very very time consuming.
If you raise the "bad" item limit to allow the migration to complete you are then not migrating the "corrupted" items. In the past I've reviewed many of these reportedly "corrupted" items in user mailboxes and I've never seen anything apparently "corrupted" about any of them.
But lets think about what happens when you don't migrate any of these "corrupted" items. You have then chosen to effectively and very deliberately, delete messages from a user mailbox - i.e you have deleted user data. During the course of a larger migration you may end up deleting hundreds or thousands of user messages - all deliberately. This is a serious problem, not only for compliance and legal reasons but you may also be deleting very important user messages which may have financial or other lasting implications.
Did I miss something or is there still no way for Microsoft to identify and repair "corrupted" items that were obviously left "corrupted" by their own software?
Or is the recommendation to review all 25 pages of this article for each "corrupted" message that you find and gather more information about it but still not fix it and again delete user data ?
Sorry I'm not seeing anything of practical use to address Microsoft-"corrupted" items but a technical rabbit hole. Please let me know if I missed your solution to the aforementioned problem.

[imsh3rlock3d]

@Mihai Grigoruta

@The_Exchange_Team 

A quick clarification needed here:

1. DCS will be performed by EXO right after the submitting the batch and before starting the actual migration, right?

2. The reports to be pulled and inspected to "investigate" will be before performing the actual data migration, or during migration or after migration at 95%? Which one is it?

3. Do you have a TechCommunity blog with troubleshooting mailbox corruption caused by Outlook client? Please share.. Time to ask users to cleanup their mess!

4. If Outlook client causes mailbox corruption, why can't server win by reparing it and pushing it down to the client? The "Exchange" idea of "sync across all devices" is defeated here, is it?

[Ziv Rivkis]

@The_Exchange_Team 

Here are some of the challenges we found using the DCS for migrations in the last few months:

1. The DCS is calculated on every sync so if you perform an incremental/delta sync and the DCS is Perfect, it doesn't guarantee that the DCS will remain as such when you perform final cutover. We have many examples where DCS changes from Perfect/Good to Investigate on final cutover when the delta sync was performed minutes earlier and had a Perfect/Good score.
2. There is no "limit" to how many items are considered before the score is set to Investigate. Again, we have many examples where DCS is set to poor for a single CorruptFolderRule, ACL or another transient error.
3. The "SkippedItemApprovalTime" duration - in some cases this has to be set multiple times before MRS accepts the "approval". One indicator that the approval was not "accepted" is to watch the StatusDetail of Get-MoveRequestStatistics and if it's "stuck" on WaitingForJobPickup for too long then you have to Suspend the move request, set the ApprovalTime again and Resume the move request.

4. Here are a couple of examples of bad item types that are not considered as impactful and will trigger a DCS of “Good” (instead of “Investigate”) when encountered:

   1. Orphaned delegate permissions where the user was removed from Active Directory (for example, the delegate left the company) but the delegate’s permission was not removed from the mailbox being migrated and the permission cannot be mapped to a valid user account. 

   2. These permission failures (Source/Target Principal Mapping Exceptions) will typically not cause a score of Investigate. We will get migration failed when reaching 100,000 corrupt ACLs.

      We had the exact opposite experience where we had scores of "Investigate" due to as few as 2 CorruptACL items. 

We saved all the XMLs that we generated and are happy to share with the Exchange Team. We understand why DCS was introduced but feel that there is still a lot of work left making it "usable" for admins who have to migrate thousands of mailboxes at a time. Our case with MS on how to best address DCS went unresolved because there was so little internal information on it that the support engineers couldn't help us. Their attempts to get more info from product team was met with "it's working as it should" even though we presented many instances where it doesn't appear to be working as designed.

[richochet]

A great article, thank you.  I read with interest the comments as there are some valid questions here.  I have not had hands on experience of reviewing DCS results as my most recent Exchange migration work has not required me to do this.  I will taking note of the article, and comments to see what the outcome is.

[Mirela_Buru]

@Ziv Rivkis , thank you for the feedback. Here are some answers to the points mentioned by you but this needs a proper analysis, so if you can provide me with the existing case number and optionally your contact details /time zone in a private message so that I can look into it and contact you, that would be great.

1. Correct, DCS is (re)calculated at every incremental sync (including final sync). Since we are migrating from a live mailbox where there are continuous changes and additions, that would explain a change in the DCS score. However, if you think we are recalculating it based on the same data /metadata from a mailbox, we would like to check this.

2. Also, correct, there is no known threshold, but it matters more the type of data that we cannot migrate. However, if you really mean Poor Score (not even Investigate), this should be very rare situation and not based on a single corrupt folder rule. Again, we would need to check it.

3. This would normally happen when Last Skipped Item Encountered was AFTER approval. This means you need to approve the new skipped items encountered (acknowledge them and hence approve it again). If you think this is not the case here, again we would need to look into examples.

4. This should not happen, if you confirm in the migration report metadata (DiagnosticInfo -> SkippedItemCounts section that you only had 2 Corrupt ACLs), we likely need to address this.

We are also open to suggestions on how exactly to improve this feature so that admins can truly benefit from it and would be nice to have a discussion with you.

[Ziv Rivkis]

Not sure why the comment was posted twice. Deleted one of the versions.

[Ziv Rivkis]

@Mirela_Buru @The_Exchange_Team 

Here are my comments based on your feedback:

1. I believe that DCS may be calculated on the same data/metadata or at least if the incremental sync (manually run) and the cutover occur in close proximity to each other, something isn't updated in time and DCS is being recalculated on the data that it was already calculated on. I had instances where I would run a manual Incremental Sync and DCS was Good/Perfect. I would then kick off a completion job that would "fail" (go back to Synced state) with DCS of "Investigate". When I reviewed the items identified, they were items that were originally synced without issues because they were "old" items in the mailbox (Calendar entry from 2018 and similar). 
2. You are correct, apologies for using the wrong term. We are seeing "Investigate" DCS on CorruptFolderACL, CorruptMailboxRule, CorruptItem issues. I can provide examples of this (some are attached to the case # provided in the DM).
3. This is a challenge, because the mailbox is "live" and is able to send/receive emails (as well as generate other items like Rules and grant Permissions) during the migration. If I set the SkippedItemApprovalTime and the user happens to send/receive an email or generate a new item during the cutover that is deemed corrupt will the DCS change to "Investigate" even though I set the SkippedItemApprovalTime? If the answer is yes, then it could explain my situation (but only if the item was created during the "cutover" time period). I've seen the DCS change to "Investigate" on old items, similarly to my comment #1. Some mailboxes generate a large delta between "prestage" and "cutover" stages (including final delta sync) that can result in items becoming corrupt but I would expect the items time stamp to be very recent.
4. I wasn't able to find the .xml file for the user that had 2 CorruptACLs and went into "Investigate". Unfortunately, I only started to keep the xml files after this became an issue for me. I did find a user with 139 CorruptACLs; should that have triggered investigate? The article states that 100,000 corrupt ACLs with trigger a failure but doesn't state how many will trigger "Investigate". I can provide the xml if required.

I believe that more parameters are required for the feature or different "weighting" of the score should be applied to items based on age and IPM type. For example, I should be able to specify -IgnoreCalendarItems or -IgnoreMailboxFolderRules and similar. I think having the ability to control "-Start" and "-End" for any DCS parameters would be great, eg. "-startDCS -start:$startdate -end:$enddate". This will allow us to ensure that the data created during the specified time range is stringently validated while the rest isn't. Not all items are treated "the same" by users so they shouldn't treated the same by the DCS. In the past, organizations accepted the fact that some items will be corrupt and lost during the migration and I am yet to see a "perfect" migration for every single mailbox.

It would be very helpful to be able to control how and where the DCS score is being calculated as well being able to reduce its impact on the batch (for example, still allow the completion of the batch but exclude the "Investigate" move requests automatically rather than block the entire batch). Additionally, since the "magic" behind DCS calculations is known, it might be very helpful to release a tool similar to IdFix for Azure AD that will allow admins to proactively review their environments and address the issues prior to migrations.

My suggestions are biased toward my needs and the needs of the clients I deal with so it would be great to see what other admins feel is missing as well.

[davidmbahm]

I've also been using DCS for migrations over the last couple of months and wanted to echo I'd also experienced the behaviors @Ziv Rivkis has mentioned. A migration batch will sync and I'll see the investigation status and I will go and approve those items. I'll then schedule the batch to complete at a certain date and that morning I won't see the email reporting the success and go and find several mailboxes are waiting approval.

At first I thought it was that additional items were flagged, but for the most recent batch I found that mailboxes that didn't complete were ones that were not previously flagged for investigation. I think the final incremental sync is evaluating items differently, because the mailboxes that didn't complete did show as having skipped items but had a Good DCS score. So in my experience it seems that you need to review and approve any mailbox with skipped items, whether DCS scores it as Investigate or not.

For our migrations any findings have been limited to Corrupt ACL or Rules, not generally seeing corrupted mail items or large items impacting our migrations.

[Mirela_Buru]

@imsh3rlock3d and @Sam_T , thank you for the feedback,  we will soon address your comments too. 

Meanwhile, to @Ziv Rivkis and @davidmbahm , this is to add more clarity:

With DCS = Investigate, when one approves the items to be skipped, we are telling the system that it can skip the items that cannot be migrated, items that have been discovered so far. We can always discover more items that cannot be migrated for various reasons (as it is a live mailbox) and require approval for those as well. So, yes, the feature is designed in a such way that the admin is made aware of the both existing and NEW items that are to be skipped. One thing that is interesting is that Content Verification only happens at the end of the migration for moves, so we sadly don’t see certain classes of corruption until the very end.
It can also happen that the Investigate Score was due to another scoring factor that it was for Good (and likely this would happen at the end of the move, in mailbox verification phase, after trying to complete the move). So let's say we check the Score before Final Sync, this is Good for now and Data Consistency Scoring Factors are Corrupt ACLs and SourcePrincipalError Then in the Final Sync the Score switched to Investigate and the factors causing this score would be in that case MissingItemInTarget and not the Corrupt ACLs / Source principal mapping errors. There could also be situation where the CorruptACLs could have triggered Investigate due to a side-effect. For example, if they were all related to the Inbox then it would be Investigate because there is special handling of things affecting the Inbox folder (rather than Junk folder as example).

Related to Ziv's remark: “for example, still allow the completion of the batch but exclude the "Investigate" move requests automatically rather than block the entire batch.” – that’s what we already do. One can set CompleteAfter on a batch/user that needs skipped-item approval (it will only respect that once it has been approved). But all skipped items encountered that trigger Investigate score need approval and we discover them continuously, till the very end of the move.

I understand this can be annoying - to keep coming and approve latest skipped items - but by not doing this, we would basically go back to old model, where admins would blindly consent to data loss in advance without knowing what bad items are being skipped during migration. This would defeat the purpose of DCS feature.

Regarding the improvement suggestions, I understand that you would like DCS to be more granular and flexible in regards to corruptions which are not related to the actual user data (separate more the end-user data like their e-mails and meetings from Mailbox Rules and ACLs) or new data vs old data.

[Mihai_Grigoruta]

@Sam_T   

Thank you for the interest shown in the new Migration feature DCS. I know the article is a bit long but we wanted to give everyone the possibility to investigate in depth a migration to Office365.

In regards to the corrupt data, I can explain why this can happen:

• Logical corruption on items can be introduced by any app that can access the mailbox, especially in older versions of Exchange
• There is a lot less validation on those versions than newer versions, so while some set of properties on a calendar item might be allowed to be set on an old version, the new version may reject that combination of properties
• Usually these items were corrupted by a 3rd party client of some kind that has put the source data into a logically corrupt state (created meetings without a valid StartDate or similar).

So the corrupt data will be skipped by our system only if you allow the data loss. And not even then the data is not lost because, using this article, you can identify it and back it up on a PST file if that information is really important.

Normally when dealing with corrupt data an Exchange admin would move the mailbox from one database to another or use the New-MailboxRepairRequest command.

Hope this brings more clarity to your journey in data migration :smiling_face_with_smiling_eyes:

[Mihai_Grigoruta]

@imsh3rlock3d 

These are the clarifications your requested:

1. DCS will be performed by EXO right after the submitting the batch and before starting the actual migration, right?

DCS will be used in the entire migration process since we check the items that will be migrated. DCS will be used after submitting and starting the batch if no Bad Items or Large Items Limit are specified. As items are being copied during migration, the score is dynamically calculated during the entire migration process. When score is Investigate or Poor, action needs to be taken by admin in order to complete the migration (switchover if we are talking about Hybrid or Google Workspace migrations that need completion of the batch). 

2. The reports to be pulled and inspected to "investigate" will be before performing the actual data migration, or during migration or after migration at 95%? Which one is it?
Strictly speaking from DCS point of view, you would need to inspect the migration reports if you receive a Score of Investigate or Poor during migration (at any phase you get that). When Score is Good (not Perfect) you might want to check why is that. Some parts of the report (Mailbox verification / missing items) will only be available after you try to complete once. However normal bad items and large items will be available for inspection as soon as they are encountered.

3. Do you have a TechCommunity blog with troubleshooting mailbox corruption caused by Outlook client? Please share.. Time to ask users to cleanup their mess!
We are not aware of such article or fact (Outlook client causing corruptions at the mailbox level).

4. If Outlook client causes mailbox corruption, why can't server win by reparing it and pushing it down to the client? The "Exchange" idea of "sync across all devices" is defeated here, is it?
As mentioned above, we are not sure of what Mailbox corruptions caused by Outlook client you refer to. Regardless of how the corruption was introduced, by which client, as the corruption already took place, we can't repair it or migrate it, it is invalid and we wouldn't know at server side what would be the desired valid values to be set there.

[Sam_T]

@Mihai_Grigoruta 

@The_Exchange_Team 

Thanks for the response re mailbox items that have reached the nebulous state of "corrupted".  It appears that you missed most of what I posted.

If I can review the response, you've clearly stated that:

"Logical corruption on items can be introduced by any app" So that includes Microsoft Exchange Server, Microsoft Outlook, Microsoft ActiveSync clients, Microsoft Edge (for OWA) or another Microsoft approved browser. Thank you for acknowledging that these Microsoft applications may indeed introduce "corruption". This is consistent with Microsoft introducing "corruption" and not having the checks in place to ensure that the "corruption" does not make it to the Exchange database.

"Usually these items were corrupted by a 3rd party client of some kind" is a blanket statement abut the quality of 3rd party applications, so its logical to assume that you have statistics to back the assertion. Can you please share your statistics that support 3rd party clients that may or may not have been created by Microsoft Partners, causing the majority of the "corruption". I have been in Exchange environments with no third party apps and we always found "corrupted" items during migrations. The Exchange servers and clients were well maintained so there was no reason to believe that the "corruption" was caused by anything other than a Microsoft application such as Exchange or Outlook.

"Usually these items were corrupted by a 3rd party client of some kind that has put the source data into a logically corrupt state". This appears to be an acknowledgement Exchange is generally vulnerable to exploits from 3rd party apps as it does little checking.

"So the corrupt data will be skipped by our system only if you allow the data loss"
This a roundabout way of saying that if you want the mailbox migrations to complete without data loss, you may spend hundreds or thousands of hours trying to remediate the "corrupted" items. The New-MailboxRepairRequest has 40+ different types of -CorruptionType values. For a larger migration you could assign a full-time individual to do nothing but look at every last "corrupted" item, attempt endless variations of New-MailboxRepairRequest repairs and then still keep their fingers crossed that the repair did not truncate business-critical information.

As mentioned in my previous post, when you "skip" items that were "corrupted" by Microsoft applications, you have then chosen to effectively and very deliberately, delete messages from a user mailbox - i.e. you have deleted user data. During the course of a larger migration you may end up deleting hundreds or thousands of user messages - all deliberately. This is a serious problem, not only for compliance and legal reasons but you may also be deleting very important user messages which may have financial or other lasting implications.

The ability to "skip" items, or the New-MailboxRepairRequest command or DCS still does nothing to address the fact that after 25+ years, Microsoft is still allowing "corrupted" items, that are most likely "corrupted" by Microsoft supported software, to make it to the database.

[davidddd]

The New-MailboxRestoreRequest can also fail to complete with the Data Consistency Score of "investigate". 

Since there appears to be zero documentation on how to proceed in this case, and there is no "ApproveSkippedItems" switch, how can we actually complete this restore?

[Mihai_Grigoruta]

@davidddd The New-MailboxRestoreRequest command is discussed at the end of the article. You do not need to complete this type of request and the DCS is only there to inform you of certain mailbox modifications made by the user since normally the restore is done on an active mailbox.

You have the necessary commands to check for more details but if you still need help you can send me a message with your email address and from there we can investigate further.

[davidddd]

Thanks Mihai, I wondered if that might be the case.  Should be documented since it is assumed and it leaves the final state as "in progress" instead of "completed with issues".  I am curious though,  I wasn't able to find a way to export the list of items that had issues.  I am guessing there is a way.

[greatquux]

There is still no way to approve skipped items in an IMAP migration. It's quite annoying.

[alavd]

Please add more documentation and resolution steps for New-MailboxRestoreRequest.

My 28 requests has ben finished and 8 was Failed with: "Fatal error DataConsistencyPermanentException has occurred."

FailureCode : -2146233088
FailureType : DataConsistencyPermanentException
FailureSide :
Message : Error: The data consistency score (Investigate) for this request is too low (due to MissingItemInTarget).

how can we actually complete this restore?

[Mirela_Buru]

Hi @alavd . this behavior is mentioned in "A note about mailbox restores and DCS"
Mailbox Restore requests don't need admins to approve the skipped items and the restores should be normally completed already, look at PercentComplete if 100% in order to confirm completion. The mailboxes where we are restoring to, are live mailboxes and there are system processes like MRM and/or situations where end-users move or delete the items restored in the Target mailbox, hence the Data Consistency Error about the missing items in target. You can use these commands to export the missing items in the target and view them.

$stats = Get-MailboxRestoreRequestStatistics "<id>"-includereport 
$e = $stats.Report.Entries | Where{$_.flags -match "mailboxverificationresults"}
$affected_entry = $e.MailboxVerificationResults | where { $_.DataConsistencyScore -notlike 'Perfect' }
$affected_entry.MissingItemsInTargetBucket.DivergentItemsSamples | export-csv -notypeinformation missingItemsUserA.csv