Improving Migrations Using Data Consistency Scoring

Published Jan 13 2020 11:51 AM 20K Views

Hello!

As you may have seen from the Microsoft 365 roadmap item, the O365 migration team has been working on improvements to the way we detect inconsistencies or data loss during migrations or moves.

We’ve historically used a configurable parameter called the ‘Bad Item Limit’ to define the number of items that you, the admin, are okay with dropping during migrations. We allow the admin to use their discretion to set this bad item limit to as low or as high as they feel comfortable with. On the service side, we can see that many migrations hit a large number of ‘bad items’ which are simply inconsistencies in metadata that the end-user may not even notice. This then resulted in many admins running their migrations with a very high Bad Item Limit by default. The problem is that the current implementation has limited capabilities to alert the admin when there are bad items that the end-user will notice, or significant amounts of truly ‘bad items’.

As our experience around migrations has grown over time, we’ve learned to distinguish between ‘expected’ and ‘unexpected’ inconsistencies and have built functionality to expose this to admins. We call this mechanism Data Consistency Scoring or DCS. Based on the number and type of data inconsistencies we detect, your migration will be categorized as Perfect, Good, Investigate, or Poor.

Migrations that end up in the Investigate bucket would require additional admin approval (self-approval via the UI or cmdlet) for completion. Migrations marked as Poor cannot be completed without escalating to support. By doing this, we are taking the guessing out of the “How many bad items am I OK with?” equation. We never had an official recommendation on what to set your ‘bad items’ limits to, and we are hoping this helps to deal with ambiguity that resulted.

Now that the DCS mechanism is fully rolled out, any new migration/migration batch that is started without a value set for the Bad Item Limit (-BadItemLimit parameter) or Large Item Limit (-LargeItemLimit parameter) will use the new DCS method. The Bad Item Limit mechanism will still be available for use and overrides DCS whenever explicitly specified, as we want to allow you time to modify your scripts to work with the new DCS method.

Update: the BadItemLimit and LargeItemLimit parameters will be completely replaced by DataConsistencyScore in January 2021.

For more details, take a look at the official documentation and guidelines for DCS! Let us know what you think!

O365 Migration Team

9 Comments
Super Contributor

Great. That Bad Item Limit ambiguity was really weird to me :)

Occasional Visitor

This is very timely information.  I've been troubleshooting why a move request would not finish for the last few days.

Senior Member

I am using the new method, DCS, and have migrations failing, support are telling me to add a baditem limit to my requests.

 

Is DCS really for mainstream?

Super Contributor

Maybe support is not up to speed yet with new features (especially L1). You said migrations are failing. Do you get Poor results as described in this article?

Senior Member

Hi,

 

No mailboxes had a 'fail' score. For the 3 mailboxes which failed to complete migration: 1 had a score of 'perfect' and the others were 'good'. 

 

Last night whilst trying to complete the migration they failed and all 3 had the following error:

 

30/01/2020 22:05:47 [DBAPR02MB6199] Stage: FinalIncrementalSync. Percent complete: 95.
30/01/2020 22:05:52 [DBAPR02MB6199] Mailbox store finalization is complete.
30/01/2020 22:05:52 [DBAPR02MB6199] SessionStatistics updated.
30/01/2020 22:05:52 [DBAPR02MB6199] Content verification: source mailbox:
01524bdc-ae3a-418a-8675-6f0abfc84bd9, target mailbox: 01524bdc-ae3a-418a-8675-6f0abfc84bd9, flags: Default.
30/01/2020 22:05:52 [DBAPR02MB6199] Started Data Guarantee wait.
30/01/2020 22:06:25 [DBAPR02MB6199] Mailbox contents verification complete: 209 folders, 8536 items, 529.7 MB
(555,447,067 bytes).
30/01/2020 22:06:25 [DBAPR02MB6199] Transient error DataConsistencyTransientException has occurred. The
system will retry (60/60, 60/600).

Occasional Visitor

Hello,

Running a restore mailbox request and noticed a DataConsistencyScore of investigate.

The restore to date is approx 35 gb and has taken 17:44 hours,  The restore has not reported any bad items only 5 large items.

 

Does an Admin need to intervene in this scenario?  

How does one approve a mailbox restore request?

 

I have read the documentation in your link but it only mentions PS commands for migrations or move requests, nothing on restore requests.

Thanks

 

New Contributor

I am in the same boat as ratzq described. DataConsistencyScore of investigate during mailbox restore and there is no way to override DCS using Baditemlimit

Frequent Contributor

I noticed a bug with this feature DCS.  You can recreate it like this:

  • Create a new batch, say onboarding mailboxes in a Hybrid deployment, and set Bad Item Limit to something (let's say 10), and leave Large Item Limit blank.
  • Next, in EXO PowerShell change the move requests' Large Item Limits to something other than 0, let's say 10 again.
  • If there are large items, the large item limit will be ignored and the move will fail.
    • You'll see this in the Get-MoveRequestStatistics:  LargeItemLimt 10
    • And this won't matter it will just be ignored.

I'm testing the solution of also updating the migration batch in the GUI with the new Large Item Limit value.  But it's strange that I would need to when other overrides like -CompleteAfter still work just fine via Set-MoveRequest.

Frequent Contributor

FYI in case anyone is back wondering the same.  It seems as though you can still override the Data Consistency Score by changing the limits on the batch itself while also doing Set-MoveRequest -BadItemLimit.

 

I also notice the Set-MoveRequest cmdlet in the EXO v2 PS module (v2.0.3) has been plastered with some double Write-Warning's about the fact that -BadItemLimit -LargeItemLimit parameters are going away.

 

Example of the double Write-Warning special

>Set-MoveRequest User@Domain.tld -BadItemLimit 40
WARNING: When an item can't be read from the source database or it can't be written to the destination database, it will be considered corrupted. By specifying a non-zero BadItemLimit, you are
 requesting Exchange not copy such items to the destination mailbox. At move completion, these corrupted items will not be available at the destination mailbox.
WARNING: The request is currently being managed as a part of a migration batch.  Changes to the request may be overwritten by the Migration Service or could impact the status expressed by the
migration batch or migration user.
WARNING: Setting a bad item limit may conceal unexpected data loss.  Please consider using the Data Consistency Score feature by not specifying the Bad or Large Item Limits.  Setting bad item
limits will be disallowed in the near future, even when the threshold (0) has not been exceeded.
WARNING: The request is currently being managed as a part of a migration batch.  Changes to the request may be overwritten by the Migration Service or could impact the status expressed by the
migration batch or migration user.
WARNING: Setting a bad item limit may conceal unexpected data loss.  Please consider using the Data Consistency Score feature by not specifying the Bad or Large Item Limits.  Setting bad item
limits will be disallowed in the near future, even when the threshold (0) has not been exceeded.

That 1x each of the original warning with -BadItemLimt, then 2x the usual warning about the move being part of a migration batch, and then 2x the new warning with the threat of the coming change.  So it definitely must be a serious threat.  That's 5 warnings, making for 1 large text wall of yellow.

 

I feel like the manual override should not be discontinued.  It should simply be updated in documentation that sometimes "BadItems" are not just corrupted items which are unsalvageable, sometimes they are indeed salvageable or even just worthwhile of letting the user know about.  Why the harsh discontinuance of these two significant features is beyond me.  Central Bad-Decisions'Ville.

%3CLINGO-SUB%20id%3D%22lingo-sub-1106022%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1106022%22%20slang%3D%22en-US%22%3E%3CP%3EGreat.%20That%20Bad%20Item%20Limit%20ambiguity%20was%20really%20weird%20to%20me%20%3A)%3C%2Fimg%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1105920%22%20slang%3D%22en-US%22%3EImproving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1105920%22%20slang%3D%22en-US%22%3E%3CP%3EHello!%3C%2FP%3E%0A%3CP%3EAs%20you%20may%20have%20seen%20from%20the%20Microsoft%20365%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fmicrosoft-365%2Froadmap%3Ffilters%3D%26amp%3Bsearchterms%3D46690%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Eroadmap%20item%3C%2FA%3E%2C%20the%20O365%20migration%20team%20has%20been%20working%20on%20improvements%20to%20the%20way%20we%20detect%20inconsistencies%20or%20data%20loss%20during%20migrations%20or%20moves.%3C%2FP%3E%0A%3CP%3EWe%E2%80%99ve%20historically%20used%20a%20configurable%20parameter%20called%20the%20%E2%80%98Bad%20Item%20Limit%E2%80%99%20to%20define%20the%20number%20of%20items%20that%20you%2C%20the%20admin%2C%20are%20okay%20with%20dropping%20during%20migrations.%20We%20allow%20the%20admin%20to%20use%20their%20discretion%20to%20set%20this%20bad%20item%20limit%20to%20as%20low%20or%20as%20high%20as%20they%20feel%20comfortable%20with.%20On%20the%20service%20side%2C%20we%20can%20see%20that%20many%20migrations%20hit%20a%20large%20number%20of%20%E2%80%98bad%20items%E2%80%99%20which%20are%20simply%20inconsistencies%20in%20metadata%20that%20the%20end-user%20may%20not%20even%20notice.%20This%20then%20resulted%20in%20many%20admins%20running%20their%20migrations%20with%20a%20very%20high%20Bad%20Item%20Limit%20by%20default.%20The%20problem%20is%20that%20the%20current%20implementation%20has%20limited%20capabilities%20to%20alert%20the%20admin%20when%20there%20are%20bad%20items%20that%20the%20end-user%20%3CEM%3Ewill%20%3C%2FEM%3Enotice%2C%20or%20significant%20amounts%20of%20truly%20%E2%80%98bad%20items%E2%80%99.%3C%2FP%3E%0A%3CP%3EAs%20our%20experience%20around%20migrations%20has%20grown%20over%20time%2C%20we%E2%80%99ve%20learned%20to%20distinguish%20between%20%E2%80%98expected%E2%80%99%20and%20%E2%80%98unexpected%E2%80%99%20inconsistencies%20and%20have%20built%20functionality%20to%20expose%20this%20to%20admins.%20We%20call%20this%20mechanism%20Data%20Consistency%20Scoring%20or%20DCS.%20Based%20on%20the%20number%20and%20type%20of%20data%20inconsistencies%20we%20detect%2C%20your%20migration%20will%20be%20categorized%20as%20Perfect%2C%20Good%2C%20Investigate%2C%20or%20Poor.%3C%2FP%3E%0A%3CP%3EMigrations%20that%20end%20up%20in%20the%20Investigate%20bucket%20would%20require%20additional%20admin%20approval%20(self-approval%20via%20the%20UI%20or%20cmdlet)%20for%20completion.%20Migrations%20marked%20as%20Poor%20cannot%20be%20completed%20without%20escalating%20to%20support.%20By%20doing%20this%2C%20we%20are%20taking%20the%20guessing%20out%20of%20the%20%E2%80%9CHow%20many%20bad%20items%20am%20I%20OK%20with%3F%E2%80%9D%20equation.%20We%20never%20had%20an%20official%20recommendation%20on%20what%20to%20set%20your%20%E2%80%98bad%20items%E2%80%99%20limits%20to%2C%20and%20we%20are%20hoping%20this%20helps%20to%20deal%20with%20ambiguity%20that%20resulted.%3C%2FP%3E%0A%3CP%3ENow%20that%20the%20DCS%20mechanism%20is%20fully%20rolled%20out%2C%20any%20new%20migration%2Fmigration%20batch%20that%20is%20started%20without%20a%20value%20set%20for%20the%20Bad%20Item%20Limit%20(-BadItemLimit%20parameter)%20or%20Large%20Item%20Limit%20(-LargeItemLimit%20parameter)%20will%20use%20the%20new%20DCS%20method.%20The%20Bad%20Item%20Limit%20mechanism%20will%20still%20be%20available%20for%20use%20and%20overrides%20DCS%20whenever%20explicitly%20specified%2C%20as%20we%20want%20to%20allow%20you%20time%20to%20modify%20your%20scripts%20to%20work%20with%20the%20new%20DCS%20method.%3C%2FP%3E%0A%3CP%3EBut%20the%20long-term%20goal%20is%20to%20eventually%20do%20away%20with%20Bad%20Item%20Limit%20and%20Large%20Item%20Limit%20altogether.%3C%2FP%3E%0A%3CP%3EFor%20more%20details%2C%20take%20a%20look%20at%20the%20official%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fmailbox-migration%2Ftrack-prevent-data-loss-dcs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Edocumentation%3C%2FA%3E%20and%20guidelines%20for%20DCS!%26nbsp%3BLet%20us%20know%20what%20you%20think!%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22author%22%3EO365%20Migration%20Team%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1105920%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20O365%20migration%20team%20has%20been%20working%20on%20improvements%20to%20the%20way%20we%20detect%20inconsistencies%20or%20data%20loss%20during%20migrations%20or%20moves...%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1105920%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAdministration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMigration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOffice%20365%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETips%20'n%20Tricks%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETools%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1107405%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1107405%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20very%20timely%20information.%26nbsp%3B%20I've%20been%20troubleshooting%20why%20a%20move%20request%20would%20not%20finish%20for%20the%20last%20few%20days.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142145%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142145%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20using%20the%20new%20method%2C%20DCS%2C%20and%20have%20migrations%20failing%2C%20support%20are%20telling%20me%20to%20add%20a%20baditem%20limit%20to%20my%20requests.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIs%20DCS%20really%20for%20mainstream%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142160%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142160%22%20slang%3D%22en-US%22%3E%3CP%3EMaybe%20support%20is%20not%20up%20to%20speed%20yet%20with%20new%20features%20(especially%20L1).%20You%20said%20migrations%20are%20failing.%20Do%20you%20get%20Poor%20results%20as%20described%20in%20this%20article%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142196%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142196%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ENo%20mailboxes%20had%20a%20'fail'%20score.%20For%20the%203%20mailboxes%20which%20failed%20to%20complete%20migration%3A%201%20had%20a%20score%20of%20'perfect'%20and%20the%20others%20were%20'good'.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ELast%20night%20whilst%20trying%20to%20complete%20the%20migration%20they%20failed%20and%20all%203%20had%20the%20following%20error%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A47%20%5BDBAPR02MB6199%5D%20Stage%3A%20FinalIncrementalSync.%20Percent%20complete%3A%2095.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Mailbox%20store%20finalization%20is%20complete.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20SessionStatistics%20updated.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Content%20verification%3A%20source%20mailbox%3A%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E01524bdc-ae3a-418a-8675-6f0abfc84bd9%2C%20target%20mailbox%3A%2001524bdc-ae3a-418a-8675-6f0abfc84bd9%2C%20flags%3A%20Default.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Started%20Data%20Guarantee%20wait.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A06%3A25%20%5BDBAPR02MB6199%5D%20Mailbox%20contents%20verification%20complete%3A%20209%20folders%2C%208536%20items%2C%20529.7%20MB%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E(555%2C447%2C067%20bytes).%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A06%3A25%20%5BDBAPR02MB6199%5D%20Transient%20error%20DataConsistencyTransientException%20has%20occurred.%20The%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3Esystem%20will%20retry%20(60%2F60%2C%2060%2F600).%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1151558%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1151558%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3ERunning%20a%20restore%20mailbox%20request%20and%20noticed%20a%20%3CSTRONG%3EDataConsistencyScore%3C%2FSTRONG%3E%20of%20investigate.%3C%2FP%3E%3CP%3EThe%20restore%20to%20date%20is%20approx%2035%20gb%20and%20has%20taken%2017%3A44%20hours%2C%26nbsp%3B%26nbsp%3BThe%20restore%20has%20not%20reported%20any%20bad%20items%20only%205%20large%20items.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDoes%20an%20Admin%20need%20to%20intervene%20in%20this%20scenario%3F%20%26nbsp%3B%3C%2FP%3E%3CP%3EHow%20does%20one%20approve%20a%20mailbox%20restore%20request%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20read%20the%20documentation%20in%20your%20link%20but%20it%20only%20mentions%20PS%20commands%20for%20migrations%20or%20move%20requests%2C%20nothing%20on%20restore%20requests.%3C%2FP%3E%3CP%3EThanks%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1460038%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1460038%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20in%20the%20same%20boat%20as%20ratzq%20described.%26nbsp%3B%3CSTRONG%3EDataConsistencyScore%3C%2FSTRONG%3E%3CSPAN%3E%26nbsp%3Bof%20investigate%20during%20mailbox%20restore%20and%20there%20is%20no%20way%20to%20override%20DCS%20using%20Baditemlimit%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E
Version history
Last update:
‎Dec 21 2020 12:11 PM
Updated by: