Home
%3CLINGO-SUB%20id%3D%22lingo-sub-1106022%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1106022%22%20slang%3D%22en-US%22%3E%3CP%3EGreat.%20That%20Bad%20Item%20Limit%20ambiguity%20was%20really%20weird%20to%20me%20%3A)%3C%2Fimg%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1105920%22%20slang%3D%22en-US%22%3EImproving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1105920%22%20slang%3D%22en-US%22%3E%3CP%3EHello!%3C%2FP%3E%0A%3CP%3EAs%20you%20may%20have%20seen%20from%20the%20Microsoft%20365%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fmicrosoft-365%2Froadmap%3Ffilters%3D%26amp%3Bsearchterms%3D46690%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Eroadmap%20item%3C%2FA%3E%2C%20the%20O365%20migration%20team%20has%20been%20working%20on%20improvements%20to%20the%20way%20we%20detect%20inconsistencies%20or%20data%20loss%20during%20migrations%20or%20moves.%3C%2FP%3E%0A%3CP%3EWe%E2%80%99ve%20historically%20used%20a%20configurable%20parameter%20called%20the%20%E2%80%98Bad%20Item%20Limit%E2%80%99%20to%20define%20the%20number%20of%20items%20that%20you%2C%20the%20admin%2C%20are%20okay%20with%20dropping%20during%20migrations.%20We%20allow%20the%20admin%20to%20use%20their%20discretion%20to%20set%20this%20bad%20item%20limit%20to%20as%20low%20or%20as%20high%20as%20they%20feel%20comfortable%20with.%20On%20the%20service%20side%2C%20we%20can%20see%20that%20many%20migrations%20hit%20a%20large%20number%20of%20%E2%80%98bad%20items%E2%80%99%20which%20are%20simply%20inconsistencies%20in%20metadata%20that%20the%20end-user%20may%20not%20even%20notice.%20This%20then%20resulted%20in%20many%20admins%20running%20their%20migrations%20with%20a%20very%20high%20Bad%20Item%20Limit%20by%20default.%20The%20problem%20is%20that%20the%20current%20implementation%20has%20limited%20capabilities%20to%20alert%20the%20admin%20when%20there%20are%20bad%20items%20that%20the%20end-user%20%3CEM%3Ewill%20%3C%2FEM%3Enotice%2C%20or%20significant%20amounts%20of%20truly%20%E2%80%98bad%20items%E2%80%99.%3C%2FP%3E%0A%3CP%3EAs%20our%20experience%20around%20migrations%20has%20grown%20over%20time%2C%20we%E2%80%99ve%20learned%20to%20distinguish%20between%20%E2%80%98expected%E2%80%99%20and%20%E2%80%98unexpected%E2%80%99%20inconsistencies%20and%20have%20built%20functionality%20to%20expose%20this%20to%20admins.%20We%20call%20this%20mechanism%20Data%20Consistency%20Scoring%20or%20DCS.%20Based%20on%20the%20number%20and%20type%20of%20data%20inconsistencies%20we%20detect%2C%20your%20migration%20will%20be%20categorized%20as%20Perfect%2C%20Good%2C%20Investigate%2C%20or%20Poor.%3C%2FP%3E%0A%3CP%3EMigrations%20that%20end%20up%20in%20the%20Investigate%20bucket%20would%20require%20additional%20admin%20approval%20(self-approval%20via%20the%20UI%20or%20cmdlet)%20for%20completion.%20Migrations%20marked%20as%20Poor%20cannot%20be%20completed%20without%20escalating%20to%20support.%20By%20doing%20this%2C%20we%20are%20taking%20the%20guessing%20out%20of%20the%20%E2%80%9CHow%20many%20bad%20items%20am%20I%20OK%20with%3F%E2%80%9D%20equation.%20We%20never%20had%20an%20official%20recommendation%20on%20what%20to%20set%20your%20%E2%80%98bad%20items%E2%80%99%20limits%20to%2C%20and%20we%20are%20hoping%20this%20helps%20to%20deal%20with%20ambiguity%20that%20resulted.%3C%2FP%3E%0A%3CP%3ENow%20that%20the%20DCS%20mechanism%20is%20fully%20rolled%20out%2C%20any%20new%20migration%2Fmigration%20batch%20that%20is%20started%20without%20a%20value%20set%20for%20the%20Bad%20Item%20Limit%20(-BadItemLimit%20parameter)%20or%20Large%20Item%20Limit%20(-LargeItemLimit%20parameter)%20will%20use%20the%20new%20DCS%20method.%20The%20Bad%20Item%20Limit%20mechanism%20will%20still%20be%20available%20for%20use%20and%20overrides%20DCS%20whenever%20explicitly%20specified%2C%20as%20we%20want%20to%20allow%20you%20time%20to%20modify%20your%20scripts%20to%20work%20with%20the%20new%20DCS%20method.%3C%2FP%3E%0A%3CP%3EBut%20the%20long-term%20goal%20is%20to%20eventually%20do%20away%20with%20Bad%20Item%20Limit%20and%20Large%20Item%20Limit%20altogether.%3C%2FP%3E%0A%3CP%3EFor%20more%20details%2C%20take%20a%20look%20at%20the%20official%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fexchange%2Fmailbox-migration%2Ftrack-prevent-data-loss-dcs%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Edocumentation%3C%2FA%3E%20and%20guidelines%20for%20DCS!%26nbsp%3BLet%20us%20know%20what%20you%20think!%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22author%22%3EO365%20Migration%20Team%3C%2FSPAN%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-1105920%22%20slang%3D%22en-US%22%3E%3CP%3EThe%20O365%20migration%20team%20has%20been%20working%20on%20improvements%20to%20the%20way%20we%20detect%20inconsistencies%20or%20data%20loss%20during%20migrations%20or%20moves...%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1105920%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAdministration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMigration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOffice%20365%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETips%20'n%20Tricks%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ETools%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1107405%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1107405%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20is%20very%20timely%20information.%26nbsp%3B%20I've%20been%20troubleshooting%20why%20a%20move%20request%20would%20not%20finish%20for%20the%20last%20few%20days.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142145%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142145%22%20slang%3D%22en-US%22%3E%3CP%3EI%20am%20using%20the%20new%20method%2C%20DCS%2C%20and%20have%20migrations%20failing%2C%20support%20are%20telling%20me%20to%20add%20a%20baditem%20limit%20to%20my%20requests.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIs%20DCS%20really%20for%20mainstream%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142160%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142160%22%20slang%3D%22en-US%22%3E%3CP%3EMaybe%20support%20is%20not%20up%20to%20speed%20yet%20with%20new%20features%20(especially%20L1).%20You%20said%20migrations%20are%20failing.%20Do%20you%20get%20Poor%20results%20as%20described%20in%20this%20article%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1142196%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1142196%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ENo%20mailboxes%20had%20a%20'fail'%20score.%20For%20the%203%20mailboxes%20which%20failed%20to%20complete%20migration%3A%201%20had%20a%20score%20of%20'perfect'%20and%20the%20others%20were%20'good'.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ELast%20night%20whilst%20trying%20to%20complete%20the%20migration%20they%20failed%20and%20all%203%20had%20the%20following%20error%3A%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A47%20%5BDBAPR02MB6199%5D%20Stage%3A%20FinalIncrementalSync.%20Percent%20complete%3A%2095.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Mailbox%20store%20finalization%20is%20complete.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20SessionStatistics%20updated.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Content%20verification%3A%20source%20mailbox%3A%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E01524bdc-ae3a-418a-8675-6f0abfc84bd9%2C%20target%20mailbox%3A%2001524bdc-ae3a-418a-8675-6f0abfc84bd9%2C%20flags%3A%20Default.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A05%3A52%20%5BDBAPR02MB6199%5D%20Started%20Data%20Guarantee%20wait.%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A06%3A25%20%5BDBAPR02MB6199%5D%20Mailbox%20contents%20verification%20complete%3A%20209%20folders%2C%208536%20items%2C%20529.7%20MB%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E(555%2C447%2C067%20bytes).%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3E30%2F01%2F2020%2022%3A06%3A25%20%5BDBAPR02MB6199%5D%20Transient%20error%20DataConsistencyTransientException%20has%20occurred.%20The%3C%2FFONT%3E%3CBR%20%2F%3E%3CFONT%20face%3D%22courier%20new%2Ccourier%22%20size%3D%223%22%3Esystem%20will%20retry%20(60%2F60%2C%2060%2F600).%3C%2FFONT%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1151558%22%20slang%3D%22en-US%22%3ERe%3A%20Improving%20Migrations%20Using%20Data%20Consistency%20Scoring%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1151558%22%20slang%3D%22en-US%22%3E%3CP%3EHello%2C%3C%2FP%3E%3CP%3ERunning%20a%20restore%20mailbox%20request%20and%20noticed%20a%20%3CSTRONG%3EDataConsistencyScore%3C%2FSTRONG%3E%20of%20investigate.%3C%2FP%3E%3CP%3EThe%20restore%20to%20date%20is%20approx%2035%20gb%20and%20has%20taken%2017%3A44%20hours%2C%26nbsp%3B%26nbsp%3BThe%20restore%20has%20not%20reported%20any%20bad%20items%20only%205%20large%20items.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDoes%20an%20Admin%20need%20to%20intervene%20in%20this%20scenario%3F%20%26nbsp%3B%3C%2FP%3E%3CP%3EHow%20does%20one%20approve%20a%20mailbox%20restore%20request%3F%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20read%20the%20documentation%20in%20your%20link%20but%20it%20only%20mentions%20PS%20commands%20for%20migrations%20or%20move%20requests%2C%20nothing%20on%20restore%20requests.%3C%2FP%3E%3CP%3EThanks%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E

Hello!

As you may have seen from the Microsoft 365 roadmap item, the O365 migration team has been working on improvements to the way we detect inconsistencies or data loss during migrations or moves.

We’ve historically used a configurable parameter called the ‘Bad Item Limit’ to define the number of items that you, the admin, are okay with dropping during migrations. We allow the admin to use their discretion to set this bad item limit to as low or as high as they feel comfortable with. On the service side, we can see that many migrations hit a large number of ‘bad items’ which are simply inconsistencies in metadata that the end-user may not even notice. This then resulted in many admins running their migrations with a very high Bad Item Limit by default. The problem is that the current implementation has limited capabilities to alert the admin when there are bad items that the end-user will notice, or significant amounts of truly ‘bad items’.

As our experience around migrations has grown over time, we’ve learned to distinguish between ‘expected’ and ‘unexpected’ inconsistencies and have built functionality to expose this to admins. We call this mechanism Data Consistency Scoring or DCS. Based on the number and type of data inconsistencies we detect, your migration will be categorized as Perfect, Good, Investigate, or Poor.

Migrations that end up in the Investigate bucket would require additional admin approval (self-approval via the UI or cmdlet) for completion. Migrations marked as Poor cannot be completed without escalating to support. By doing this, we are taking the guessing out of the “How many bad items am I OK with?” equation. We never had an official recommendation on what to set your ‘bad items’ limits to, and we are hoping this helps to deal with ambiguity that resulted.

Now that the DCS mechanism is fully rolled out, any new migration/migration batch that is started without a value set for the Bad Item Limit (-BadItemLimit parameter) or Large Item Limit (-LargeItemLimit parameter) will use the new DCS method. The Bad Item Limit mechanism will still be available for use and overrides DCS whenever explicitly specified, as we want to allow you time to modify your scripts to work with the new DCS method.

But the long-term goal is to eventually do away with Bad Item Limit and Large Item Limit altogether.

For more details, take a look at the official documentation and guidelines for DCS! Let us know what you think!

O365 Migration Team

6 Comments
Super Contributor

Great. That Bad Item Limit ambiguity was really weird to me :)

Occasional Visitor

This is very timely information.  I've been troubleshooting why a move request would not finish for the last few days.

Senior Member

I am using the new method, DCS, and have migrations failing, support are telling me to add a baditem limit to my requests.

 

Is DCS really for mainstream?

Super Contributor

Maybe support is not up to speed yet with new features (especially L1). You said migrations are failing. Do you get Poor results as described in this article?

Senior Member

Hi,

 

No mailboxes had a 'fail' score. For the 3 mailboxes which failed to complete migration: 1 had a score of 'perfect' and the others were 'good'. 

 

Last night whilst trying to complete the migration they failed and all 3 had the following error:

 

30/01/2020 22:05:47 [DBAPR02MB6199] Stage: FinalIncrementalSync. Percent complete: 95.
30/01/2020 22:05:52 [DBAPR02MB6199] Mailbox store finalization is complete.
30/01/2020 22:05:52 [DBAPR02MB6199] SessionStatistics updated.
30/01/2020 22:05:52 [DBAPR02MB6199] Content verification: source mailbox:
01524bdc-ae3a-418a-8675-6f0abfc84bd9, target mailbox: 01524bdc-ae3a-418a-8675-6f0abfc84bd9, flags: Default.
30/01/2020 22:05:52 [DBAPR02MB6199] Started Data Guarantee wait.
30/01/2020 22:06:25 [DBAPR02MB6199] Mailbox contents verification complete: 209 folders, 8536 items, 529.7 MB
(555,447,067 bytes).
30/01/2020 22:06:25 [DBAPR02MB6199] Transient error DataConsistencyTransientException has occurred. The
system will retry (60/60, 60/600).

Occasional Visitor

Hello,

Running a restore mailbox request and noticed a DataConsistencyScore of investigate.

The restore to date is approx 35 gb and has taken 17:44 hours,  The restore has not reported any bad items only 5 large items.

 

Does an Admin need to intervene in this scenario?  

How does one approve a mailbox restore request?

 

I have read the documentation in your link but it only mentions PS commands for migrations or move requests, nothing on restore requests.

Thanks