Handling CSOM 429 Errors from PowerShell Scripts

%3CLINGO-SUB%20id%3D%22lingo-sub-837789%22%20slang%3D%22en-US%22%3EHandling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-837789%22%20slang%3D%22en-US%22%3E%3CP%3EI%20have%20a%20PowerShell%20script%20which%20processes%20a%20large%20number%20of%20lists%2C%20it%20*will*%20get%20429%20errors.%3C%2FP%3E%3CP%3EI%20could%20put%20a%20sleep%20in%20there%2C%20but%20how%20long%3F%20Too%20long%20will%20make%20an%20already%20long-running%20script%20unnecessarily%20slow%2C%20so%20I%20want%20to%20handle%20the%20429's%20and%20respect%20the%20Retry-After.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI've%20got%20a%20test%20script%20like%20this%2C%20which%20I%20run%20many%20times%20concurrently%20for%20force%20threshold%20errors%3A%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-csharp%22%3E%3CCODE%3Efor(%24i%3D0%3B%20%24i%20-lt%2010000%3B%20%24i%2B%2B)%20%0A%7B%0A%20%20%20%20Write-Verbose%20%22Attempt%20%24(%24i)%22%20-Verbose%0A%20%20%20%20%24caml%20%3D%20%5BMicrosoft.SharePoint.Client.CamlQuery%5D%3A%3ACreateAllItemsQuery()%0A%20%20%20%20%24items%20%3D%20%24lib.GetItems(%24caml)%0A%20%20%20%20try%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%20%20%24ctx.Load(%24items)%0A%20%20%20%20%20%20%20%20%24ctx.ExecuteQuery()%0A%20%20%20%20%20%20%20%20Write-Verbose%20%22Found%20%24(%24items.Count)%20items%22%20-Verbose%0A%20%20%20%20%7D%0A%20%20%20%20catch%0A%20%20%20%20%7B%0A%20%20%20%20%20%20%20%20Write-Warning%20%24_%0A%20%20%20%20%20%20%20%20Start-Sleep%20-Seconds%2020%0A%20%20%20%20%7D%0A%7D%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWhen%20I%20watch%20the%20scripts%20running%20in%20Fiddler%20I%20*do*%20see%20the%20Retry-After%20header.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EQuestion%3A%20How%20do%20I%20get%20the%20Response%20Headers%20from%20my%20ExecuteQuery%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-837789%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ECSOM%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EHttp%20429%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESharepointOnline%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-838716%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-838716%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F169139%22%20target%3D%22_blank%22%3E%40David%20Whitehead%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHope%20you%20have%20to%20get%26nbsp%3B%3CSPAN%3ERetry-After%20field%20from%20catched%20error%20object.%3C%2FSPAN%3E%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-csharp%22%3E%3CCODE%3Etry%0A%7B%0A%20%20%20.......................%0A%7D%0Acatch%0A%7B%20%20%20%0A%20%20%20%20Write-Host%20%22Retry-After%3A%22%20%24_.Exception.Response.Headers%5B%22Retry-After%22%5D%20%0A%7D%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-839093%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-839093%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F38365%22%20target%3D%22_blank%22%3E%40Kevin%20Morgan%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHey%20Kevin%2C%20I%20put%20a%20breakpoint%20in%20my%20catch%20block.%26nbsp%3B%3C%2FP%3E%3CP%3E%24_%20looks%20good%20(it's%20the%20correct%20exception)%2C%20but%20%24_.Response%20is%20null.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-csharp%22%3E%3CCODE%3EHit%20Line%20breakpoint%20on%20'C%3A%5CUsers%5C..%5CDesktop%5CTesting%5CLVT%20Test.ps1%3A44'%0A07%3A35%3A59%20.%5CDesktop%20%26gt%3B%20%24_.Exception.Response.Headers%5B%22Retry-After%22%5D%0ACannot%20index%20into%20a%20null%20array.%0AAt%20line%3A1%20char%3A1%0A%2B%20%24_.Exception.Response.Headers%5B%22Retry-After%22%5D%0A%2B%20~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%0A%20%20%20%20%2B%20CategoryInfo%20%20%20%20%20%20%20%20%20%20%3A%20InvalidOperation%3A%20(%3A)%20%5B%5D%2C%20RuntimeException%0A%20%20%20%20%2B%20FullyQualifiedErrorId%20%3A%20NullArray%0A%20%0A%0A07%3A36%3A05%20.%5CDesktop%20%26gt%3B%20%24_%0AException%20calling%20%22ExecuteQuery%22%20with%20%220%22%20argument(s)%3A%20%22The%20remote%20server%20returned%20an%20%0Aerror%3A%20(429).%22%0AAt%20C%3A%5CUsers%5C..%5CDesktop%5CTesting%5CLVT%20%0ATest.ps1%3A39%20char%3A9%0A%2B%20%20%20%20%20%20%20%20%20%24ctx.ExecuteQuery()%0A%2B%20%20%20%20%20%20%20%20%20~~~~~~~~~~~~~~~~~~~%0A%20%20%20%20%2B%20CategoryInfo%20%20%20%20%20%20%20%20%20%20%3A%20NotSpecified%3A%20(%3A)%20%5B%5D%2C%20MethodInvocationException%0A%20%20%20%20%2B%20FullyQualifiedErrorId%20%3A%20WebException%0A%20%0A%0A07%3A36%3A13%20.%5CDesktop%20%26gt%3B%20%24_.ResPonse%0A%0A07%3A36%3A22%20.%5CDesktop%20%26gt%3B%20%24_.ResPonse%20-eq%20%24null%0ATrue%0A%0A07%3A36%3A44%20.%5CDesktop%20%26gt%3B%20%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20tried%20to%20get%20the%20Response%20details%20with%3A%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CPRE%20class%3D%22lia-code-sample%20language-csharp%22%3E%3CCODE%3E%24streamReader%20%3D%20%5BSystem.IO.StreamReader%5D%3A%3Anew(%24_.Exception.Response.GetResponseStream())%0A%24ErrResp%20%3D%20%24streamReader.ReadToEnd()%0A%24streamReader.Close()%0AWrite-Verbose%20%24ErrResp%20%3C%2FCODE%3E%3C%2FPRE%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThis%20get%20me%20the%20string%20'429%20TOO%20MANY%20REQUESTS'%20which%20might%20be%20workable%20to%20identify%20it%20as%20a%20429%20(I%20wonder%20if%20it%20will%20always%20be%20in%20English..%3F).%20I%20don't%20see%20the%20response%20Retry-After.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-840654%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-840654%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F169139%22%20target%3D%22_blank%22%3E%40David%20Whitehead%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHave%20you%20checked%20the%20below%20posts%20%3A%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FSharePoint-Support-Blog%2FDecorating-CSOM-calls-in-PowerShell%2Fba-p%2F177998%22%20target%3D%22_blank%22%20rel%3D%22noopener%22%3Ehttps%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2FSharePoint-Support-Blog%2FDecorating-CSOM-calls-in-PowerShell%2Fba-p%2F177998%3C%2FA%3E%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fstackoverflow.com%2Fquestions%2F51730540%2Fhow-to-get-retry-after-header-in-csom-context%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fstackoverflow.com%2Fquestions%2F51730540%2Fhow-to-get-retry-after-header-in-csom-context%3C%2FA%3E%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-841451%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-841451%22%20slang%3D%22en-US%22%3E%3CP%3EHey%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F38365%22%20target%3D%22_blank%22%3E%40Kevin%20Morgan%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20had%20seen%20that.%20I%20have%20not%20decorated%20my%20requests%20in%20my%20test%20script%2C%20primarily%20because%20I%20am%20trying%20to%20cause%20429's%20there!%26nbsp%3B%3C%2FP%3E%3CP%3EI%20have%20also%20not%20decorated%20the%20real%20script.%20I%20should%2C%20and%20will%2C%20but%20this%20only%20reduces%20the%20probability%20of%20receiving%20a%20429.%20The%20process%20for%20this%20script%20(necessarily)%20enumerates%20all%20the%20libraries%20in%20the%20tenant%20and%20updates%20them.%20This%20tenant%20is%20huuuuge.%26nbsp%3B%3C%2FP%3E%3CP%3ESo%20I%20feel%20that%2C%20even%20with%20a%20well%20decorated%20script%2C%20I%20will%20still%20get%20429s%20and%20need%20to%20handle%20them.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20looking%20at%20two%20options%3A%3C%2FP%3E%3CP%3E1)%20sleep%20in%20a%20suitable%20location.%20How%20long%3F%20If%20I%20sleep%20too%20frequently%20and%20too%20long%2C%20the%20script%20will%20run%20much%20longer%20than%20necessary%3B%20If%20I%20don't%20sleep%20frequently%20or%20long%20enough%20then%20I%20don't%20solve%20the%20problem.%20MS%20returns%20the%20header%20for%20a%20good%20reason.%3C%2FP%3E%3CP%3E2)%20Switch%20to%20REST.%20I%20believe%20REST%20will%20give%20me%20the%20headers%2C%20though%20my%20quick%20testing%20seems%20to%20be%20giving%20me%20406's%20instead%20of%20429's.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1081844%22%20slang%3D%22en-US%22%3ERe%3A%20Handling%20CSOM%20429%20Errors%20from%20PowerShell%20Scripts%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1081844%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F169139%22%20target%3D%22_blank%22%3E%40David%20Whitehead%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20had%20this%20same%20429%20issue.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EDoing%20%22decorate%22%20thing%20did%20not%20make%20a%20difference%2C%20and%20adding%20a%20few%20seconds%20of%20%22sleep%22%20between%20each%20update%20helped%20a%20little%20bit%20but%20did%20not%20fully%20resolve%20the%20problem.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20resolved%20it%20by%20using%26nbsp%3BExecuteQueryAsync()%20instead%20of%26nbsp%3BExecuteQuery().%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Occasional Contributor

I have a PowerShell script which processes a large number of lists, it *will* get 429 errors.

I could put a sleep in there, but how long? Too long will make an already long-running script unnecessarily slow, so I want to handle the 429's and respect the Retry-After.

 

I've got a test script like this, which I run many times concurrently for force threshold errors: 

 

for($i=0; $i -lt 10000; $i++) 
{
    Write-Verbose "Attempt $($i)" -Verbose
    $caml = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery()
    $items = $lib.GetItems($caml)
    try
    {
        $ctx.Load($items)
        $ctx.ExecuteQuery()
        Write-Verbose "Found $($items.Count) items" -Verbose
    }
    catch
    {
        Write-Warning $_
        Start-Sleep -Seconds 20
    }
}

 

 

When I watch the scripts running in Fiddler I *do* see the Retry-After header. 

 

Question: How do I get the Response Headers from my ExecuteQuery?

5 Replies
Highlighted

@David Whitehead 

 

Hope you have to get Retry-After field from catched error object.

try
{
   .......................
}
catch
{   
    Write-Host "Retry-After:" $_.Exception.Response.Headers["Retry-After"] 
}

 

Highlighted

@Kevin Morgan 

Hey Kevin, I put a breakpoint in my catch block. 

$_ looks good (it's the correct exception), but $_.Response is null.

 

 

Hit Line breakpoint on 'C:\Users\..\Desktop\Testing\LVT Test.ps1:44'
07:35:59 .\Desktop > $_.Exception.Response.Headers["Retry-After"]
Cannot index into a null array.
At line:1 char:1
+ $_.Exception.Response.Headers["Retry-After"]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : NullArray
 

07:36:05 .\Desktop > $_
Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an 
error: (429)."
At C:\Users\..\Desktop\Testing\LVT 
Test.ps1:39 char:9
+         $ctx.ExecuteQuery()
+         ~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException
 

07:36:13 .\Desktop > $_.ResPonse

07:36:22 .\Desktop > $_.ResPonse -eq $null
True

07:36:44 .\Desktop > 

 

 

 

I tried to get the Response details with: 

 

$streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd()
$streamReader.Close()
Write-Verbose $ErrResp 

 

This get me the string '429 TOO MANY REQUESTS' which might be workable to identify it as a 429 (I wonder if it will always be in English..?). I don't see the response Retry-After.

 

Highlighted

Hey @Kevin Morgan 

 

I had seen that. I have not decorated my requests in my test script, primarily because I am trying to cause 429's there! 

I have also not decorated the real script. I should, and will, but this only reduces the probability of receiving a 429. The process for this script (necessarily) enumerates all the libraries in the tenant and updates them. This tenant is huuuuge. 

So I feel that, even with a well decorated script, I will still get 429s and need to handle them. 

 

I'm looking at two options:

1) sleep in a suitable location. How long? If I sleep too frequently and too long, the script will run much longer than necessary; If I don't sleep frequently or long enough then I don't solve the problem. MS returns the header for a good reason.

2) Switch to REST. I believe REST will give me the headers, though my quick testing seems to be giving me 406's instead of 429's.

Highlighted

@David Whitehead 

 

I had this same 429 issue.

 

Doing "decorate" thing did not make a difference, and adding a few seconds of "sleep" between each update helped a little bit but did not fully resolve the problem.

 

I resolved it by using ExecuteQueryAsync() instead of ExecuteQuery().