Sep 04 2019
06:03 PM
- last edited on
Feb 06 2023
04:06 AM
by
TechCommunityAP
Sep 04 2019
06:03 PM
- last edited on
Feb 06 2023
04:06 AM
by
TechCommunityAP
I have a PowerShell script which processes a large number of lists, it *will* get 429 errors.
I could put a sleep in there, but how long? Too long will make an already long-running script unnecessarily slow, so I want to handle the 429's and respect the Retry-After.
I've got a test script like this, which I run many times concurrently for force threshold errors:
for($i=0; $i -lt 10000; $i++)
{
Write-Verbose "Attempt $($i)" -Verbose
$caml = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery()
$items = $lib.GetItems($caml)
try
{
$ctx.Load($items)
$ctx.ExecuteQuery()
Write-Verbose "Found $($items.Count) items" -Verbose
}
catch
{
Write-Warning $_
Start-Sleep -Seconds 20
}
}
When I watch the scripts running in Fiddler I *do* see the Retry-After header.
Question: How do I get the Response Headers from my ExecuteQuery?
Sep 05 2019 05:31 AM
Hope you have to get Retry-After field from catched error object.
try
{
.......................
}
catch
{
Write-Host "Retry-After:" $_.Exception.Response.Headers["Retry-After"]
}
Sep 05 2019 07:50 AM - edited Sep 05 2019 08:50 AM
Hey Kevin, I put a breakpoint in my catch block.
$_ looks good (it's the correct exception), but $_.Response is null.
Hit Line breakpoint on 'C:\Users\..\Desktop\Testing\LVT Test.ps1:44'
07:35:59 .\Desktop > $_.Exception.Response.Headers["Retry-After"]
Cannot index into a null array.
At line:1 char:1
+ $_.Exception.Response.Headers["Retry-After"]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
07:36:05 .\Desktop > $_
Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an
error: (429)."
At C:\Users\..\Desktop\Testing\LVT
Test.ps1:39 char:9
+ $ctx.ExecuteQuery()
+ ~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : WebException
07:36:13 .\Desktop > $_.ResPonse
07:36:22 .\Desktop > $_.ResPonse -eq $null
True
07:36:44 .\Desktop >
I tried to get the Response details with:
$streamReader = [System.IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
$ErrResp = $streamReader.ReadToEnd()
$streamReader.Close()
Write-Verbose $ErrResp
This get me the string '429 TOO MANY REQUESTS' which might be workable to identify it as a 429 (I wonder if it will always be in English..?). I don't see the response Retry-After.
Sep 06 2019 03:57 AM - edited Sep 06 2019 04:00 AM
Sep 06 2019 10:29 AM
Hey @Kevin Morgan
I had seen that. I have not decorated my requests in my test script, primarily because I am trying to cause 429's there!
I have also not decorated the real script. I should, and will, but this only reduces the probability of receiving a 429. The process for this script (necessarily) enumerates all the libraries in the tenant and updates them. This tenant is huuuuge.
So I feel that, even with a well decorated script, I will still get 429s and need to handle them.
I'm looking at two options:
1) sleep in a suitable location. How long? If I sleep too frequently and too long, the script will run much longer than necessary; If I don't sleep frequently or long enough then I don't solve the problem. MS returns the header for a good reason.
2) Switch to REST. I believe REST will give me the headers, though my quick testing seems to be giving me 406's instead of 429's.
Dec 27 2019 09:26 AM
I had this same 429 issue.
Doing "decorate" thing did not make a difference, and adding a few seconds of "sleep" between each update helped a little bit but did not fully resolve the problem.
I resolved it by using ExecuteQueryAsync() instead of ExecuteQuery().