Scroll Down WebPage until End

YeuHarng 

Hi, Yeu.

You can't "scroll down" in a native PowerShell script, but this doesn't mean you need to resort to a tool like Selenium, either.

The FBI has its own REST API which is sufficient for what you're trying to do (noting you have a second thread going on this topic), however, the cyber category isn't individually searchable. It seems cyber can only be obtained using the default category.

• FBI API

Anyhow, here's an example script that you can include further filtering in if you so desire.

Note: You will need to install the ThreadJob PowerShell module for this example, as noted in the comments at the top of the script.

Example

# This script leverages the FBI REST API described at:
# https://api.fbi.gov/docs#!/Wanted/get_wanted

# This script also relies on the ThreadJob module (for downloading files in parallel) being installed from the official PSGallery repository:
# Install-Module -Name ThreadJob -Scope AllUsers;

$SaveLocation = "D:\Data\Temp\Forum\fbi\";

# Do not set PageSize too high as this also relates to how many concurrent downloads will be kicked off.
$PageSize = 20;
$Page = 1;

$UriBase  = "https://api.fbi.gov/@wanted";
$Category = "default";

while (0 -ne ($Response = Invoke-RestMethod -Method Get -URI "$UriBase`?poster_classification=$Category&pageSize=$PageSize&page=$Page" -ContentType "application/json" -UseBasicParsing -ErrorAction:Stop).total)
{
    $Downloads = @();
    $Job = 0;

    try
    {
        foreach ($Url in ($Response.Items.Files | Where-Object { ($_.Name -eq "English") -and ($_.url.EndsWith(".pdf")) }).url)
        {
            $FileName = "$SaveLocation$(($Parts = $Url.Split("/"))[-3])_$($Parts[-2]).pdf";
    
            $Downloads += Start-ThreadJob -Name "fbi$($Job.ToString('X3'))" -ScriptBlock {
                Invoke-WebRequest -Uri $using:Url -OutFile $using:FileName -ErrorAction:Stop;
            };
    
            $Job++;
        }
    
        Wait-Job -Job $Downloads | Out-Null;
        Receive-Job -Job $Downloads -ErrorAction:Continue | Out-Null;
        Remove-Job -Job $Downloads -ErrorAction:SilentlyContinue -Force;

        $Downloads.Dispose();
    }
    catch
    {
        throw;
    }
    finally
    {
        $Page++;
    }
}

Edited: To remove the two-page limit I'd been using for testing (as there's nearly 1,000 records in the default category.)

Cheers,

Lain