Handling CSOM 429 Errors from PowerShell Scripts

[Cauldron_of_Penguins]

I have a PowerShell script which processes a large number of lists, it *will* get 429 errors.

I could put a sleep in there, but how long? Too long will make an already long-running script unnecessarily slow, so I want to handle the 429's and respect the Retry-After.

I've got a test script like this, which I run many times concurrently for force threshold errors: 

for($i=0; $i -lt 10000; $i++) 
{
    Write-Verbose "Attempt $($i)" -Verbose
    $caml = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery()
    $items = $lib.GetItems($caml)
    try
    {
        $ctx.Load($items)
        $ctx.ExecuteQuery()
        Write-Verbose "Found $($items.Count) items" -Verbose
    }
    catch
    {
        Write-Warning $_
        Start-Sleep -Seconds 20
    }
}

When I watch the scripts running in Fiddler I *do* see the Retry-After header. 

Question: How do I get the Response Headers from my ExecuteQuery?

[#2]

@Cauldron_of_Penguins 

Hope you have to get Retry-After field from catched error object.

try
{
   .......................
}
catch
{   
    Write-Host "Retry-After:" $_.Exception.Response.Headers["Retry-After"] 
}

[#3]

@Kevin Morgan 

Hey Kevin, I put a breakpoint in my catch block. 

$_ looks good (it's the correct exception), but $_.Response is null.

Hit Line breakpoint on 'C:\Users\..\Desktop\Testing\LVT Test.ps1:44'
07:35:59 .\Desktop > $_.Exception.Response.Headers["Retry-After"]
Cannot index into a null array.
At line:1 char:1
+ $_.Exception.Response.Headers["Retry-After"]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : NullArray
 

07:36:05 .\Desktop > $_
Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an 
error: (429)."
At C:\Users\..\Desktop\Testing\LVT 
Test.ps1:39 char:9
+         $ctx.ExecuteQuery()
+         ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException
 

07:36:13 .\Desktop > $_.ResPonse

07:36:22 .\Desktop > $_.ResPonse -eq $null
True

07:36:44 .\Desktop > 

I tried to get the Response details with: 

$streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd()
$streamReader.Close()
Write-Verbose $ErrResp 

This get me the string '429 TOO MANY REQUESTS' which might be workable to identify it as a 429 (I wonder if it will always be in English..?). I don't see the response Retry-After.

[#4]

@Cauldron_of_Penguins 

Have you checked the below posts :

https://techcommunity.microsoft.com/t5/SharePoint-Support-Blog/Decorating-CSOM-calls-in-PowerShell/b...

https://stackoverflow.com/questions/51730540/how-to-get-retry-after-header-in-csom-context

[#5]

Hey @Kevin Morgan 

I had seen that. I have not decorated my requests in my test script, primarily because I am trying to cause 429's there! 

I have also not decorated the real script. I should, and will, but this only reduces the probability of receiving a 429. The process for this script (necessarily) enumerates all the libraries in the tenant and updates them. This tenant is huuuuge. 

So I feel that, even with a well decorated script, I will still get 429s and need to handle them. 

I'm looking at two options:

1) sleep in a suitable location. How long? If I sleep too frequently and too long, the script will run much longer than necessary; If I don't sleep frequently or long enough then I don't solve the problem. MS returns the header for a good reason.

2) Switch to REST. I believe REST will give me the headers, though my quick testing seems to be giving me 406's instead of 429's.

[#6]

@Cauldron_of_Penguins 

I had this same 429 issue.

Doing "decorate" thing did not make a difference, and adding a few seconds of "sleep" between each update helped a little bit but did not fully resolve the problem.

I resolved it by using ExecuteQueryAsync() instead of ExecuteQuery().