SharePoint 429 Error - Throttling

Enric Carrión One suggestion I could mention is how we have tried to deal with this situation while these errors were occurring. As mentioned in my previous reply, we modified our solution to stop hitting SharePoint for the amount of time that is mentioned in the 429 error (usually 2 minutes).

Of course this effected us very negatively as all systems were very very slow and we were asking our users to not use the system unless they really need to. But at least we managed to limp trough this situation until somehow it got resolved on it's own.

At the time we got the suggestion to only start our processes (migration) after business hours (18:00-07:00). That helped us limiting the throttling errors. We were also using ShareGate and they updated their software as well to minimise the 429 errors. 

I had a feeing it also depend on the location of your tenant, ours at the time was in the UK.

Hi Vesa,

we already posted https://github.com/SharePoint/sp-dev-docs/issues/1710 and https://github.com/SharePoint/sp-dev-docs/issues/1711 four days ago, and we're collecting data for a third one today. 

In the meantime, support tickets are open by these customers, although we didn't get specific replies yet.

So, as some productive systems are affected (= unusable) and every day counts, this is extremely concerning for us, and I wanted to get some feedback from the users that originally faced this 2 months ago.

I understand that every issue needs to be properly diagnosed and assessed, so I really appreciate all the efforts you're already making. Please let us know if we can help with any action/test/troubleshooting from our side. There is critical business around this issue and we're willing to help as much as possible.

Thank you!

Justin Spiteri Thanks a lot for your feedback, I'm glad your issues got solved.

Hi Enric,

unfortunately we did not get any official replies from Microsoft neither, other then the ones in this forum, however these 429 errors vanished without us doing nothing. We experienced these errors throughout the month of February and while the 429 errors began without us doing any changes, we still did a lot of suggested improvements in our code such as waiting to hit SharePoint for the timeout specified in the 429 error, using multiple users and decorating HTTP.

However after these changes were implemented, the errors were still occurring. Then at the end of February, they just stopped occurring.

Regards,

Justin

Please share us details around your issue at https://github.com/SharePoint/sp-dev-docs/issues, so that we can route this properly internally thx. 

Hi,

we're currently experiencing the same situation in our tenants and our customer's tenants. Since 2/3 weeks ago, the 429 errors are suddenly increasing without any configuration changes or updates. 

Also using incremental backoff retry and User Agent. 

We created an ASPX page that can be uploaded to any Azure web app and simply performs a CSOM query every 5 sec. against SharePoint online. We can see how this operation returns 429 a significant amount of times in these tenants. In case it's useful for someone, it's attached to this comment.

As we didn't get any official reply from MS yet, did you guys manage to get an explanation or a satisfactory resolution to your cases? Did the errors just go or were there any actions involved?

We feel we're quite in the dark here...

Thank you,

Enric

Dear VesaJuvonen

As to support, we tried to open this through the normal O365 channels and through our distributor. The case was # 30126-7340755.

Unfortunately we were never directed to the site you mentioned and if you have access to this case, you may see that most of our questions in relation to throttling were not answered. The standard reply was;

"As discussed earlier we as a frontline support will not be able to provide you any further support on this scenario. Also according to the scope of the service request this is out of support boundary and we are proceeding with the archival of this service request 30126-7340755"

We were being directed to how to avoid throttling with each support engineer, as against replying to our queries as to how and why the throttling occurred and started out of nowhere.

I believe Justin has followed your advise and replied accordingly.

Many thanks for following this up.

Thanks VesaJuvonen for you reply. We have opened a new issue in the list provided and answering all your questions there. 

These issues were occurring mainly in February. Our target here is to identify what caused these errors and why. We are planning to move more services to SharePoint and such issues will hold us back until we fully migrate our services, since it will effect usability of custom built applications for our Business to run on SharePoint.

Thanks everyone for sharing your finding. We would really appreciate if you could report these issues using our official sp-dev-docs issue list at https://github.com/SharePoint/sp-dev-docs/issues. This issue list is automatically synced with our engineering task list so that we can ensure that your issue does not get hidden or ignored.

Would ask following from each of you, so that we can start solving these one-by-one. 

- Which tenant has the issue?

- When did the issue happen, so that we can check right log entries?

- What was your code performing when this happened?

- Did you use a user-agent string as instructed in https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online?

There should NOT be any world-wide change on this capability which should be this widely impacting, but apparently, issues are seen in multiple tenants, so we need actual facts and technical details to analyze what has happened. We can't really investigate the underlying issue without tenant and timing details.

Thanks for your input advance. Even if your issue seems to be identified as the previously reported issue, would suggest submitting a new issue so that we truly understand how widely these throttling issues are being encountered.

Also in the future - Please use the https://github.com/SharePoint/sp-dev-docs/issues issue list for this kind of issues. This list exists for you to be able to directly report issues to engineering. Thx.

Further to this, the 2 main issues are;

- the disruption such unannounced changes create in production environments

- the lack of information on what changed in Feb 2018

Our exact same conclusion.

Symptoms all started in Feb 2018 and we're also trying to obtain more information from Microsoft, but so far we keep being redirected on 'how to avoid 429 errors' rather than what led to this situation. We have worked with no issues for the past 6 months, including migration.

As you mentioned earlier, we also came across instances where a single call triggers error 429 and other times with heavy load and this does not occur.

The timing of 429 errors starting to appear was early Feb 2018 for me as well. Other threads and people I've had conversations with share this timeline as well. It seems something changed in either the way throttling happens or the thresholds were tightened around this time - still trying to get some official information from Microsoft. 

Sharegate has released a new version which should get a higher priority from Microsoft which than should decrease the number of 429. We did not try this version yet because we finished our migration.

The 429 throttling errors are to protect the tenant so they might occur in different situations in different tenants. Our was situated in the UK and we were throttled suddenly in Feb-2018. Before we did not have any 429 issues and we converted 6Tb before.  When moving off hours we were able to run Sharegate again without many 429 errors. Microsoft has a SLA on availability NOT on performance that is why they will do anything to protect their office365 tenants.