SharePoint Migration Tool calls are THROTTLED by O365 API

Copper Contributor

Hi,

We have a customer with SharePoint 2013 environment, hosted in Azure (as a few VMs).  Let's call the customer Contoso (NDA restriction), for which we are creating Migration solution, based on latest SharePoint Migration Tool v3.2.114.0 (and its PowerShell reference).

 

We implemented a simple PowerShell script, that uses SPMT PS cmdlets to Register SPMT Migration, adds tasks and starts SPTM Migration.

 

 

 

# -------------- MIGRATION WITH SPMT  -------------- #


# Register the SPMT session with SPO credentials#

Register-SPMTMigration -SPOCredential $Global:SPOCredential `

    -SkipSitesWithName "$($Global:subWebs -join ';')" `

    -EnableMultiRound $true `

    -Force


# Add migration task into the session

Add-SPMTTask -SharePointSourceCredential $Global:SPCredential `

    -SharePointSourceSiteUrl $Global:SourceSiteUrl  `

    -TargetSiteUrl $Global:SPOTargetSiteUrl `

    -MigrateAll     # Migrate all lists and libraries!


#Start Migration in the console. #

Start-SPMTMigration 

 

 

 

Source sites are small sites with some documents (on test environment it is from 10 to 200 MB of documents), some members (from 5 to 50). No custom solutions migrated. No large data or large quantity of files.

 

We are now in implementation and internal testing phase, which lasted a few months already. Everything worked good so far, until September 24, 2019.

 

On that day we started to receive strange THROTTLING responses from O365, when SharePoint Migration Tool tried to request any info from the environment.

 

SPMT logs are full of following (or similar) records:

 

2019-10-17 16:02:53.5014|INFO|WF|275|8f837fbd|Start Assessing 'https://intranet-uat.contoso.com/teamsites/Event-Sverige' 

2019-10-17 16:02:53.5014|DEBUG|WF|275|8f837fbd|Scan using previous report 

2019-10-17 16:02:53.5014|DEBUG|WF|275|8f837fbd|Assess source credentials... 

2019-10-17 16:02:53.6734|DEBUG|WF|275|8f837fbd|Assess target credential... 

2019-10-17 16:02:53.6734|DEBUG|WF|275|8f837fbd|Getting schema for site https://contoso.sharepoint.com/teams/Event-Sverige 

2019-10-17 16:02:53.8434|DEBUG|WF|275|8f837fbd|Scanning full web /teams/event-sverige under site https://contoso.sharepoint.com/teams/Event-Sverige 

2019-10-17 16:02:54.5004|DEBUG|WF|275|8f837fbd|Successfully fetch navigation setting for web https://contoso.sharepoint.com/teams/Event-Sverige 

2019-10-17 16:02:54.7384|ERROR|WF|275|8f837fbd|Csom call with TraceCorrelationId 42c30e9f-4052-1000-0076-4b5ecedef351 from SPO Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPTooManyRequestsExceptionToo many requests response from SharePoint Online. Stop the task. ---> Microsoft.SharePoint.Migration.Common.Exceptions.WebTooManyRequestException ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.

   at System.Net.HttpWebRequest.GetResponse()

   at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute()

   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)

   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   --- End of inner exception stack trace ---

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   --- End of inner exception stack trace ---

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.ExecuteQuery(String callerMemberName, String callerFilePath)

2019-10-17 16:02:54.7554|ERROR|WF|275|8f837fbd|Failed to load web navigation trees layer by layer Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPTooManyRequestsExceptionToo many requests response from SharePoint Online. Stop the task. ---> Microsoft.SharePoint.Migration.Common.Exceptions.WebTooManyRequestException ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.

   at System.Net.HttpWebRequest.GetResponse()

   at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute()

   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)

   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   --- End of inner exception stack trace ---

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   --- End of inner exception stack trace ---

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.WrapExceptions(Action logicToRetry)

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.ExecuteQuery(String callerMemberName, String callerFilePath)

   at Microsoft.SharePoint.MigrationTool.MigrationLib.SharePoint.MigSPClientContext.RunWithRetry(Action logicToRetry, Int32 retryCount, Int32 retryIntervalMiliSeconds, Func`2 exceptionHandler, String errorMessage, Boolean resetContextIfRetry)

 

300 times for this migration session (as an example).

Error, marked blue , is different for each occurrence – SPMT makes different requests to Target environment (both on Scanning and Migration phases), and sometimes O365/SPO API refuses these requests with “Too many requests” response.

As a result of that throttling, we often (often!) get “INVALID CREDENTIALS” Fatal Error during migration.

"Invalid credentials","ACTION_STOP","0x0201000C"

2019-10-17 16:03:02.6755|ERROR|WF|275|8f837fbd|Got a fatal exception during scan Microsoft.SharePoint.MigrationTool.MigrationLib.Assessment.Exceptions.InvalidCredentialsExceptionInvalid credentials ---> Microsoft.SharePoint.Migration.Common.Exceptions.WebCredentialException ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized.

   at System.Net.HttpWebRequest.GetResponse()

   at Microsoft.SharePoint.MigrationTool.MigrationLib.Common.Net.HttpWebRequestProxy.GetResponse()

   --- End of inner exception stack trace ---

   at Microsoft.SharePoint.Migration.Common.Exceptions.WebExceptionBase.Create(WebException e)

   at Microsoft.SharePoint.MigrationTool.MigrationLib.Common.Net.HttpWebRequestProxy.GetResponse()

   at Microsoft.SharePoint.Migration.Common.Utilities.RunWithRetry[T](Func`1 logicToRetry, Int32 retryCount, Int64 timeOutSeconds, Int32 retryIntervalMiliSeconds, String errorMessage, Type[] retryExceptionTypes, Type[] nonRetryExceptionTypes)


We say “often”, but not “always”. Sometimes our Migration Sessions ends well, and everything is migrated, despite a lot of “Too many requests” responses (because of Retry Policy in SPMT – it repeats requests). Approximate quantity – 50% failed sessions, 50% successful sessions. But even though we are lucky to see “COMPLETED” status – it takes AN HOUR or more to migrate 50-100 MB package.

Credentials of SPO Admin account are verified - and they are correct, because Migration Script creates a new Target site before launching SMPT Migration session. And, of course, credentials of SPO Admin are NOT CHANGED during Migration Session.

We also tried to CHANGE SPO Admin account, which makes the migration. It helps for some small period (a few migrations), and then throttling comes back.

Of course, we know about https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-o... - but I stress out, that we don't use any custom solutions for migration, we rely only on SPMT (latest version).

All users’ requests are not throttled. End-user experience is completely fine.

Our test CSOM requests are NOT throttled as well. Everything is fine, when we are, for example, uploading several PDFs to Target site with CSOM, with the same credentials.

Regarding tenant usage: Customer has a few active users for now: DEV team and test team. They have 17 active sites only (and the traffic on that is VERY low).

clipboard_image_0.png



SharePoint Online services are completely healthy (based on Admin Center reports).

We have an important question:

We are migrating almost empty sites from UAT environment now. The sites, that have 20, 30, 100 MB of documents. We are doing it from a single Azure machine. Single user does it. We are using latest, out-of-the box SPMT. Why it is getting throttled?

4 Replies
Thanks for flagging this, there are reports lately about more throttling happening on SPO migrations. I will share this with someone of the product team and I hope the will be able to provide some light
Any updates on this topic?

No migration can be launched during the day since it starts to indicate errors like the following one:
MigSPTooManyRequestsException: Too many requests response from SharePoint Online.

@albegut There is nothing to update on this topic. The throttling is there to protect service quality during business hours of the region of the Office 365 tenant. You need to abide by the guidelines

 

The throttling is even more strict (lack of better term) due to the COVID19 situation.  It is incorrect that you can not launch migrations. You can and it will migrate some content, but it will be throttled quicker in comparison to say if you start at 6 PM or weekend.  

@albegutWell, I might say, that we are in process of migration of an Intranet for one of our customers to M365, and it works good enough (in comparison to the time, when I started this thread). We were able to migrate 200 sites with average size from 100 MB to 1 GB.

 

Indeed, migrations work better in non-business hours and during the weekends. It's not so convenient, but at least it works - single migration session took from 30 minutes to 1,5 hours.