Random CSOM 503 errors

[Charles Willwerth]

Hello,

I am seeing random 503 errors in Fiddler when running CSOM code. As an example (using PnP ExecuteQueryRetry):

using (var clientContext = new ClientContext(url))
  {
   // (code to add credentials goes here)
   clientContext.Load(clientContext.Web);
   var web = clientContext.Web;
   var list = web.Lists.GetByTitle("Status");
   clientContext.Load(list);
   var StatusTask = list.GetItemById(2665); // using this item just to test
   clientContext.Load(StatusTask, item => item["Status"]);
   clientContext.ExecuteQueryRetry();
   if (StatusTask["Status"].ToString() == "Status") // this is where I get a C# error, but in Fiddler I can already see the 503 error
    {
     // do stuff
    }
  }

This code works the vast majority of the time, but sometimes errors out, and it's only one call against SharePoint. If we have a process that runs ExecuteQuery() dozens of times (and we do), it lowers our success rate quite a bit.

I've noticed various threads on Twitter today having this issue but wanted to post here in case other people are looking. 

Example twitter threads: 

https://twitter.com/Frankyyh/status/1242025700133081088

https://twitter.com/pwmather/status/1242048897444446208

[#2]

Very common issue those days with a lot of people working from home

[#3]

@Charles Willwerth We see this sporadically (and quite extensively) at multiple of our customers' tenants since last week. It has a huge impact on our ISV solutions.

So far we have seen no public information from MS about this issue even existing but it is very wide spread and seems to increase during hours of high demand. I guess it is related to more people using Teams and WFH and that this stresses the underlying infrastructure. But even so, we need offical information / recognition from MS!

[#4]

@Robert Lindgren Yes, I'd like to see it acknowledged by Microsoft. I don't think going through all code everywhere to have multiple retries on 503 errors is a good idea at this point, but any sort of support ticket request is met with "hey can't support any custom scripting."