SOLVED

Curl API help

%3CLINGO-SUB%20id%3D%22lingo-sub-218859%22%20slang%3D%22en-US%22%3ECurl%20API%20help%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-218859%22%20slang%3D%22en-US%22%3E%3CP%3EGot%20it%20working%20script%20below%20in%20reply%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20having%20some%20trouble%20with%20the%20Curl%20API%20I%20want%20to%20export%20all%20policy%20hits%20that%20are%20related%20to%20the%20%22%3CSPAN%3EFile%20containing%20PII%20detected%20in%20the%20cloud%20(built-in%20DLP%20engine)%22%20policy.%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EThis%20is%20what%20I've%20go%20so%20far%2C%20but%20I%20can't%20get%20anything%20related%20to%20%22policy%22%20to%20work.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E-XGET%20-k%20%22MYURL%22%20-H%20%22Authorization%3A%20Token%20MYTOKEN%22%3CBR%20%2F%3E-d%20'%3CBR%20%2F%3E%7B%3CBR%20%2F%3E%5C%22filters%5C%22%3A%3CBR%20%2F%3E%7B%5C%22policy%5C%22%3A%3CBR%20%2F%3E%7B%5C%22eq%5C%22%3A%20%5C%22File%20containing%20PII%20detected%20in%20the%20cloud%20%5C(built-in%20DLP%20engine%5C)%5C%22%7D%3CBR%20%2F%3E%7D%2C%3CBR%20%2F%3E%5C%22limit%5C%22%3A%202%5C%3CBR%20%2F%3E%7D'%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIt%20also%20fails%20with%20%22%3CA%20href%3D%22https%3A%2F%2Fsalliemae.us.portal.cloudappsecurity.com%2Fapi-docs%2F%23operators%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ecabinetmatchedrulesequals%3C%2FA%3E%22%20as%20the%20api%20documentation%20says%20(but%20that%20looks%20like%20a%20typo)%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20when%20I%20swap%20out%20the%20policy%20for%20filename%20and%20it%20works.%3C%2FP%3E%3CP%3E%7B%5C%22filename%5C%22%3A%20%7B%5C%22eq%5C%22%3A%20%5C%22sheet002.htm%5C%22%7D%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-218859%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ECloud%20App%20Security%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ECloud%20Discovery%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3Edata%20classification%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-236811%22%20slang%3D%22en-US%22%3ERe%3A%20Curl%20API%20help%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-236811%22%20slang%3D%22en-US%22%3EAnd%20it%20ate%20the%20formatting%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-236808%22%20slang%3D%22en-US%22%3ERe%3A%20Curl%20API%20help%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-236808%22%20slang%3D%22en-US%22%3E%3CP%3EHere's%20my%20Powershell%20script%20that%20I%20created%20to%20do%20this%2C%20it%20will%20export%20everything%20until%20it%20runs%20out.%26nbsp%3B%20It%20is%20reliant%20on%20calling%20curl.%26nbsp%3B%20I%20got%20it%20from%20%22%3CA%20href%3D%22https%3A%2F%2Fcurl.haxx.se%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fcurl.haxx.se%3C%2FA%3E%22%20I'm%20using%26nbsp%3B%20version%207.59.0.%26nbsp%3B%20It%26nbsp%3Bcompleted%26nbsp%3Bjust%20over%20105k%20for%20me.%26nbsp%3B%20I%20also%20do%20a%20few%20unnecessary%20things%2C%20rotating%20keys%20really%20shouldn't%20be%20necessary%20so%20either%20remove%20that%20code%20or%20just%20put%20the%20same%20API%20key%20into%20all%204%20locations.%26nbsp%3B%20(I%20did%20it%20because%20I%20was%20having%20a%20lot%20of%20time%20out%20problems%20but%20that%20was%20because%20my%20requests%20weren't%20using%20indexes%20on%20the%20back%20end%20database%2C%20so%20the%20requests%20503'd%20on%20me)%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3Ecls%3CBR%20%2F%3Ecd%20%22PathToCurlExe%22%20%23path%20to%20the%20curl%20exe%3C%2FP%3E%3CP%3E%24response%20%3D%20%24null%3CBR%20%2F%3E%24skip%20%3D%200%20%23position%20to%20start%20at%3CBR%20%2F%3E%24url%20%3D%20%22YourUrl%22%3CBR%20%2F%3E%24failurekey%20%3D%20%22Authorization%3A%20Token%20YourApiKey%22%3CBR%20%2F%3E%24balancer%20%3D%200%20%23rotate%20through%20keys%3CBR%20%2F%3E%24k1%20%3D%200%20%23falures%20on%20key%201%3CBR%20%2F%3E%24k2%20%3D%200%20%23falures%20on%20key%202%3CBR%20%2F%3E%24k3%20%3D%200%20%23falures%20on%20key%203%3CBR%20%2F%3E%24kb%20%3D%200%20%23falures%20on%20key%20backup%3CBR%20%2F%3E%24fail%20%3D%200%20%23total%20count%20of%20failures%3CBR%20%2F%3E%24hasnext%20%3D%20%24true%3CBR%20%2F%3E%24out%20%3D%20%24null%3CBR%20%2F%3Edo%7B%3CBR%20%2F%3E%23what%20are%20we%20looking%20for%3CBR%20%2F%3E%23lastGlobalMatchDate%20is%20used%20to%20%22activate%22%20database%20indexes%20backend%20has%20trouble%20gathering%20data%20before%20timeout%20if%20indexes%20aren't%20used%20(once%20you%20get%20past%2010k%20records)%3CBR%20%2F%3E%23change%20sortDirection%20from%20asc%20to%20dsc%20if%20you%20start%20to%20get%20too%20many%20timeout%20and%20just%20work%20it%20from%20the%20other%20direction%3CBR%20%2F%3E%24data%20%3D%20'%7B%5C%22filters%5C%22%3A%20%7B%5C%22policy%5C%22%3A%20%7B%5C%22cabinetmatchedrulesequals%5C%22%3A%20%5B%5C%22YOURPOLICY%5C%22%5D%7D%7D%2C%5C%22sortField%5C%22%3A%5C%22lastGlobalMatchDate%5C%22%2C%5C%22sortDirection%5C%22%3A%5C%22asc%5C%22%2C%5C%22limit%5C%22%3A%20100%2C%5C%22skip%5C%22%3A'%20%2B%20%24skip%20%2B%20'%7D'%3CBR%20%2F%3E%3CBR%20%2F%3Eswitch(%24balancer%253)%20%23rotate%20keys%20in%20round%20robin%3CBR%20%2F%3E%7B%3CBR%20%2F%3E0%20%7B%24head%20%3D%20%22Authorization%3A%20Token%20YourApiKey%22%7D%3CBR%20%2F%3E1%20%7B%24head%20%3D%20%22Authorization%3A%20Token%20YourApiKey%22%7D%3CBR%20%2F%3E2%20%7B%24head%20%3D%20%22Authorization%3A%20Token%20YourApiKey%22%7D%3CBR%20%2F%3E%7D%3C%2FP%3E%3CP%3Etry%20%23pull%20data%3CBR%20%2F%3E%7B%3CBR%20%2F%3E%24response%20%3D%20%26amp%3B.%5Ccurl.exe%20-XGET%20-k%20%24url%20-H%20%24head%20-d%20%24data%3CBR%20%2F%3E%24out%20%3D%20%24response%20%7C%20ConvertFrom-Json%3CBR%20%2F%3E%7D%3CBR%20%2F%3Ecatch%20%23Problem%20pulling%20data%20go%20to%20backup%20key%20and%20try%20again%3CBR%20%2F%3E%7B%3CBR%20%2F%3Etry%3CBR%20%2F%3E%7B%3CBR%20%2F%3EWrite-Host%20%22Failure%20on%20key%20number%3A%20%24(%24balancer%20%25%203)%22%3CBR%20%2F%3E%24fail%20%2B%3D%201%3CBR%20%2F%3Eswitch(%24balancer%253)%20%23Keep%20count%20of%20fails%20per%20key%3CBR%20%2F%3E%7B%3CBR%20%2F%3E0%20%7B%24k1%20%2B%3D%201%7D%3CBR%20%2F%3E1%20%7B%24k2%20%2B%3D%201%7D%3CBR%20%2F%3E2%20%7B%24k3%20%2B%3D%201%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%24response%20%3D%20%26amp%3B.%5Ccurl.exe%20-XGET%20-k%20%24url%20-H%20%24failurekey%20-d%20%24data%3CBR%20%2F%3E%24out%20%3D%20%24response%20%7C%20ConvertFrom-Json%3CBR%20%2F%3E%7D%3CBR%20%2F%3Ecatch%3CBR%20%2F%3E%7B%3CBR%20%2F%3EWrite-Host%20%22Failure%20on%20Backup%20key%22%3CBR%20%2F%3E%24kb%20%2B%3D%201%3CBR%20%2F%3E%24fail%20%2B%3D%201%3CBR%20%2F%3E%24skip%20-%3D%20100%20%23force%20try%20again%20redo%20(this%20with%20be%20negated%20by%20the%20increment)%3CBR%20%2F%3E%24out.data%20%3D%20%24null%20%23if%20this%20isn't%20done%20error%20on%20backup%20with%20duplicate%20last%20successful%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%7D%3CBR%20%2F%3Eif(%24out.data%20-ne%20%24null)%3CBR%20%2F%3E%7B%3CBR%20%2F%3E%24out.data%20%7C%20select%20name%2CownerAddress%2CappName%2CalternateLink%20%7C%20Export-Csv%20-Append%20-nti%20out.csv%20%23dump%20data%20to%20a%20CSV%3CBR%20%2F%3E%24hasnext%20%3D%20%24out.hasNext%3CBR%20%2F%3E%7D%3CBR%20%2F%3E%24skip%20%2B%3D%20100%20%23go%20for%20next%20set%20of%20records%3CBR%20%2F%3E%24balancer%20%2B%3D%201%20%23rotate%20keys%3CBR%20%2F%3EWrite-Host%20%22Count%3A%20%24(%24skip%2F100)%20%60nTotal%20Fails%3A%20%24fail%20%60nKey0%3A%20%24k1%20%60nKey1%3A%20%24k2%20%60nKey2%3A%20%24k3%60nBackup%3A%20%24kb%22%23progress%3CBR%20%2F%3E%7D%20while(%24hasnext)%20%23keep%20going%20until%20the%20system%20says%20stop%3C%2FP%3E%3C%2FLINGO-BODY%3E
Deleted
Not applicable

Got it working script below in reply

 

 

 

 

I'm having some trouble with the Curl API I want to export all policy hits that are related to the "File containing PII detected in the cloud (built-in DLP engine)" policy. 

 

This is what I've go so far, but I can't get anything related to "policy" to work.  

 

-XGET -k "MYURL" -H "Authorization: Token MYTOKEN"
-d '
{
\"filters\":
{\"policy\":
{\"eq\": \"File containing PII detected in the cloud \(built-in DLP engine\)\"}
},
\"limit\": 2\
}'

 

It also fails with "cabinetmatchedrulesequals" as the api documentation says (but that looks like a typo)

 

I when I swap out the policy for filename and it works.

{\"filename\": {\"eq\": \"sheet002.htm\"}

2 Replies
best response
Solution

Here's my Powershell script that I created to do this, it will export everything until it runs out.  It is reliant on calling curl.  I got it from "https://curl.haxx.se" I'm using  version 7.59.0.  It completed just over 105k for me.  I also do a few unnecessary things, rotating keys really shouldn't be necessary so either remove that code or just put the same API key into all 4 locations.  (I did it because I was having a lot of time out problems but that was because my requests weren't using indexes on the back end database, so the requests 503'd on me)  

 

cls
cd "PathToCurlExe" #path to the curl exe

$response = $null
$skip = 0 #position to start at
$url = "YourUrl"
$failurekey = "Authorization: Token YourApiKey"
$balancer = 0 #rotate through keys
$k1 = 0 #falures on key 1
$k2 = 0 #falures on key 2
$k3 = 0 #falures on key 3
$kb = 0 #falures on key backup
$fail = 0 #total count of failures
$hasnext = $true
$out = $null
do{
#what are we looking for
#lastGlobalMatchDate is used to "activate" database indexes backend has trouble gathering data before timeout if indexes aren't used (once you get past 10k records)
#change sortDirection from asc to dsc if you start to get too many timeout and just work it from the other direction
$data = '{\"filters\": {\"policy\": {\"cabinetmatchedrulesequals\": [\"YOURPOLICY\"]}},\"sortField\":\"lastGlobalMatchDate\",\"sortDirection\":\"asc\",\"limit\": 100,\"skip\":' + $skip + '}'

switch($balancer%3) #rotate keys in round robin
{
0 {$head = "Authorization: Token YourApiKey"}
1 {$head = "Authorization: Token YourApiKey"}
2 {$head = "Authorization: Token YourApiKey"}
}

try #pull data
{
$response = &.\curl.exe -XGET -k $url -H $head -d $data
$out = $response | ConvertFrom-Json
}
catch #Problem pulling data go to backup key and try again
{
try
{
Write-Host "Failure on key number: $($balancer % 3)"
$fail += 1
switch($balancer%3) #Keep count of fails per key
{
0 {$k1 += 1}
1 {$k2 += 1}
2 {$k3 += 1}
}
$response = &.\curl.exe -XGET -k $url -H $failurekey -d $data
$out = $response | ConvertFrom-Json
}
catch
{
Write-Host "Failure on Backup key"
$kb += 1
$fail += 1
$skip -= 100 #force try again redo (this with be negated by the increment)
$out.data = $null #if this isn't done error on backup with duplicate last successful
}
}
if($out.data -ne $null)
{
$out.data | select name,ownerAddress,appName,alternateLink | Export-Csv -Append -nti out.csv #dump data to a CSV
$hasnext = $out.hasNext
}
$skip += 100 #go for next set of records
$balancer += 1 #rotate keys
Write-Host "Count: $($skip/100) `nTotal Fails: $fail `nKey0: $k1 `nKey1: $k2 `nKey2: $k3`nBackup: $kb"#progress
} while($hasnext) #keep going until the system says stop

And it ate the formatting