Forum Discussion
Curl API help
Got it working script below in reply
I'm having some trouble with the Curl API I want to export all policy hits that are related to the "File containing PII detected in the cloud (built-in DLP engine)" policy.
This is what I've go so far, but I can't get anything related to "policy" to work.
-XGET -k "MYURL" -H "Authorization: Token MYTOKEN"
-d '
{
\"filters\":
{\"policy\":
{\"eq\": \"File containing PII detected in the cloud \(built-in DLP engine\)\"}
},
\"limit\": 2\
}'
It also fails with "https://salliemae.us.portal.cloudappsecurity.com/api-docs/#operators" as the api documentation says (but that looks like a typo)
I when I swap out the policy for filename and it works.
{\"filename\": {\"eq\": \"sheet002.htm\"}
Here's my Powershell script that I created to do this, it will export everything until it runs out. It is reliant on calling curl. I got it from "https://curl.haxx.se" I'm using version 7.59.0. It completed just over 105k for me. I also do a few unnecessary things, rotating keys really shouldn't be necessary so either remove that code or just put the same API key into all 4 locations. (I did it because I was having a lot of time out problems but that was because my requests weren't using indexes on the back end database, so the requests 503'd on me)
cls
cd "PathToCurlExe" #path to the curl exe$response = $null
$skip = 0 #position to start at
$url = "YourUrl"
$failurekey = "Authorization: Token YourApiKey"
$balancer = 0 #rotate through keys
$k1 = 0 #falures on key 1
$k2 = 0 #falures on key 2
$k3 = 0 #falures on key 3
$kb = 0 #falures on key backup
$fail = 0 #total count of failures
$hasnext = $true
$out = $null
do{
#what are we looking for
#lastGlobalMatchDate is used to "activate" database indexes backend has trouble gathering data before timeout if indexes aren't used (once you get past 10k records)
#change sortDirection from asc to dsc if you start to get too many timeout and just work it from the other direction
$data = '{\"filters\": {\"policy\": {\"cabinetmatchedrulesequals\": [\"YOURPOLICY\"]}},\"sortField\":\"lastGlobalMatchDate\",\"sortDirection\":\"asc\",\"limit\": 100,\"skip\":' + $skip + '}'
switch($balancer%3) #rotate keys in round robin
{
0 {$head = "Authorization: Token YourApiKey"}
1 {$head = "Authorization: Token YourApiKey"}
2 {$head = "Authorization: Token YourApiKey"}
}try #pull data
{
$response = &.\curl.exe -XGET -k $url -H $head -d $data
$out = $response | ConvertFrom-Json
}
catch #Problem pulling data go to backup key and try again
{
try
{
Write-Host "Failure on key number: $($balancer % 3)"
$fail += 1
switch($balancer%3) #Keep count of fails per key
{
0 {$k1 += 1}
1 {$k2 += 1}
2 {$k3 += 1}
}
$response = &.\curl.exe -XGET -k $url -H $failurekey -d $data
$out = $response | ConvertFrom-Json
}
catch
{
Write-Host "Failure on Backup key"
$kb += 1
$fail += 1
$skip -= 100 #force try again redo (this with be negated by the increment)
$out.data = $null #if this isn't done error on backup with duplicate last successful
}
}
if($out.data -ne $null)
{
$out.data | select name,ownerAddress,appName,alternateLink | Export-Csv -Append -nti out.csv #dump data to a CSV
$hasnext = $out.hasNext
}
$skip += 100 #go for next set of records
$balancer += 1 #rotate keys
Write-Host "Count: $($skip/100) `nTotal Fails: $fail `nKey0: $k1 `nKey1: $k2 `nKey2: $k3`nBackup: $kb"#progress
} while($hasnext) #keep going until the system says stop
2 Replies
Here's my Powershell script that I created to do this, it will export everything until it runs out. It is reliant on calling curl. I got it from "https://curl.haxx.se" I'm using version 7.59.0. It completed just over 105k for me. I also do a few unnecessary things, rotating keys really shouldn't be necessary so either remove that code or just put the same API key into all 4 locations. (I did it because I was having a lot of time out problems but that was because my requests weren't using indexes on the back end database, so the requests 503'd on me)
cls
cd "PathToCurlExe" #path to the curl exe$response = $null
$skip = 0 #position to start at
$url = "YourUrl"
$failurekey = "Authorization: Token YourApiKey"
$balancer = 0 #rotate through keys
$k1 = 0 #falures on key 1
$k2 = 0 #falures on key 2
$k3 = 0 #falures on key 3
$kb = 0 #falures on key backup
$fail = 0 #total count of failures
$hasnext = $true
$out = $null
do{
#what are we looking for
#lastGlobalMatchDate is used to "activate" database indexes backend has trouble gathering data before timeout if indexes aren't used (once you get past 10k records)
#change sortDirection from asc to dsc if you start to get too many timeout and just work it from the other direction
$data = '{\"filters\": {\"policy\": {\"cabinetmatchedrulesequals\": [\"YOURPOLICY\"]}},\"sortField\":\"lastGlobalMatchDate\",\"sortDirection\":\"asc\",\"limit\": 100,\"skip\":' + $skip + '}'
switch($balancer%3) #rotate keys in round robin
{
0 {$head = "Authorization: Token YourApiKey"}
1 {$head = "Authorization: Token YourApiKey"}
2 {$head = "Authorization: Token YourApiKey"}
}try #pull data
{
$response = &.\curl.exe -XGET -k $url -H $head -d $data
$out = $response | ConvertFrom-Json
}
catch #Problem pulling data go to backup key and try again
{
try
{
Write-Host "Failure on key number: $($balancer % 3)"
$fail += 1
switch($balancer%3) #Keep count of fails per key
{
0 {$k1 += 1}
1 {$k2 += 1}
2 {$k3 += 1}
}
$response = &.\curl.exe -XGET -k $url -H $failurekey -d $data
$out = $response | ConvertFrom-Json
}
catch
{
Write-Host "Failure on Backup key"
$kb += 1
$fail += 1
$skip -= 100 #force try again redo (this with be negated by the increment)
$out.data = $null #if this isn't done error on backup with duplicate last successful
}
}
if($out.data -ne $null)
{
$out.data | select name,ownerAddress,appName,alternateLink | Export-Csv -Append -nti out.csv #dump data to a CSV
$hasnext = $out.hasNext
}
$skip += 100 #go for next set of records
$balancer += 1 #rotate keys
Write-Host "Count: $($skip/100) `nTotal Fails: $fail `nKey0: $k1 `nKey1: $k2 `nKey2: $k3`nBackup: $kb"#progress
} while($hasnext) #keep going until the system says stop