Mailbox Migration Performance Analysis

When you're migrating on-premises mailboxes to Office365, there are a lot of factors that can impact overall mailbox migration speed and performance. This post will help you investigate and correct the possible causes by using the AnalyzeMoveRequestStats.ps1 script  to analyze the performance of a batch of move requests to identify reasons for slower performance.

The AnalyzeMoveRequestStatsscript provides important performance statistics from a given set of move request statistics. It also generates two files - one for the failure list, and one for individual move statistics.

Step 1: Download the script (see attachment on this post) and import using the following syntax

> . .\AnalyzeMoveRequestStats.ps1

Step 2: Select the move requests that you want to analyze

In this example, we’re retrieving all currently executing requests that are not in a queued state.

> $moves = Get-MoveRequest | ?{$_.Status -ne 'queued'}

Step 3: Get the move reports for each of the move requests you want to analyze

Note: This make take a few minutes, especially if you’re analyzing a large number of moves

> $stats = $moves | Get-MoveRequestStatistics –IncludeReport

Step 4: Run the ProcessStats function to generate the statistics

> ProcessStats -stats $stats -name ProcessedStats1

The output should look similar to the following:

StartTime:    2/18/2014 19:57
EndTime:    3/3/2014 17:15
MigrationDuration:    12 day(s) 19:10:55
MailboxCount:    50
TotalGBTransferred:    2.42
PercentComplete:    95
MaxPerMoveTransferRateGBPerHour:    1.11
MinPerMoveTransferRateGBPerHour:    0.43
AvgPerMoveTransferRateGBPerHour:    0.66
MoveEfficiencyPercent:    86.36
AverageSourceLatency:    123.55
AverageDestinationLatency:   
IdleDuration:    1.16%
SourceSideDuration:    78.93%
DestinationSideDuration:    19.30%
WordBreakingDuration:    9.63%
TransientFailureDurations:    0.00%
OverallStallDurations:    4.55%
ContentIndexingStalls:    1.23%
HighAvailabilityStalls:    0.00%
TargetCPUStalls:    3.32%
SourceCPUStalls:    0.00%
MailboxLockedStall:    0.00%
ProxyUnknownStall:    0.00%

How to read the results

The first step in understanding the results are to understand the definitions of the report items:

Next, you need to identify which side of the migration components is slower by looking at the SourceSideDuration and DestinationSideDurationvalues.

Note: The SourceSideDuration value + DestinationSideDuration value is usually, but not always, equal to 100%.

If you see that the SourceSideDuration value is greater than the normal range of 60-80%, this means the source side is the bottleneck. If the DestinationSideDurationvalue is greater than the normal 20-40%, this means the destination side of the migration is the bottleneck

Causes of source side slowness in onboarding scenarios

There are several possible reasons the source side of the migration in an onboarding scenario may be causing slower than normal performance.

High transient failures

Most common reason for transient failures is the connectivity issue to the on-premises MRSProxy web service on your Mailbox servers. Check the TransientFailureDurations and MailboxLockedStallvalues in the output and also check the failure log generated by this script to verify any failures. The source mailbox may also get locked when a transient failure occurs and this will lower migration performance.

Misconfigured network load balancers

Another common reason for connectivity issues are misconfigured load balancers. If you're load balancing your servers, the load balancer needs to be configured so that all calls for a migration specific request is directed to the same server hosting the MRSProxy service instances.

Some load balancers use the ExchangeCookieto associate all the migration requests to the same Mailbox server where the MRSProxy service is hosted.

If your load balancers are not configured correctly, migration calls may be directed to the “wrong” MRSProxy service instance and will fail. This causes the source mailbox to be locked for some time and lowers migration performance.

High network latency

The Office 365 MRSProxy service makes periodic dummy web service calls to the on-premises MRSProxy service and collects statistics from these calls. The AverageSourceLatencyvalue represents the average duration of these calls. If you see high values for this parameter (greater than 100ms), it can be caused by:

Network latency between the Office 365 and on-premises MRSProxy services is high

In this case, you can try to reduce the network latency by

1. Migrate mailboxes from servers closer to Office 365 datacenters. This is usually not feasible, but can be preferred if the migration project is time critical.
2. Delete empty mailbox folders or consolidate mailbox folders. High network latency affects the move rate more if there are too many folders.
3. Increase the export buffer size. This reduces the number of migration calls, especially for larger mailboxes and reduces the time spent in network latency. You can increase the export buffer size by adding the ExportBufferSizeOverrideKB parameter in the MSExchangeMailboxReplication.exe.config file

Example: ExportBufferSizeOverrideKB="7500"

Important: You need to have Exchange 2013 SP1 installed on your Client Access server to increase the export buffer size. This will also disable cross forest downgrade moves between Exchange 2013 and Exchange 2010 servers.

Source servers are too busy and not very responsive to the web service calls

In this care you can try to release some of the system resources (CPU, Memory, Disk IO etc.) on the Mailbox and Client Access servers.

Scale issues

The resource consumption on the on-premises Mailbox or Client Access servers may be high if you're not load balancing the migration requests or you're running other services on the same servers. You can try distributing the source mailboxes to multiple Mailbox servers and moving mailboxes to different databases located on separate physical hard drives.

Causes of destination side slowness in onboarding scenarios

There are several possible reasons the destination side of the migration in an onboarding scenario may be causing slower than normal performance. Since the destination side of the migration is Office 365, there are limited options for you to try to resolve bottlenecks. The best solution is to remove the migration requests and re-insert them so that migration requests are assigned to less busy Office 365 servers.

Office 365 system resources: This is likely due to insufficient system resources in Office 365. Office 365 destination servers may be too busy with handling other requests associated with normal support of services for your organization.

Stalls due to word breaking: Any mailbox content that is migrated to Office365 is separated into individual words so that it can be indexed later on. This is performed by the Office 365 search service and coordinated by the MRS service. Check the WordBreakingDuration values to see how much time is spent in this process. If it's more than 15%, it usually indicates that the content indexing service running on the Office 365 target server is busy.

Stall due to Content Indexing: Content indexing service on the Office 365 servers is too busy.

Stall due to High Availability: High availability service that is responsible to copy the data to multiple Office 365 servers is too busy.

Stall due to CPU: The Office 365 server's CPU consumption is too high.

Karahan Celikel and the Migration Team