Fetch Activity with Curl retrieves limited entries

Hi All, I am trying to fetch activities of last few days and I am using below command, curl "http://mydomain.cloudsecurity.com/api/v1/activities/" -H 'Authorization: Token mytoken" -d ' { "filt...

Cloud App Security

Mike Kassis
Nov 15, 2017
Hi Sanket,

The activities API endpoint has a query limit of 100 records to prevent overloading the endpoint with any one request. If you want to retrieve more than 100 records, you will need to add that logic to your script to update the parameters of the curl request through a loop. For example, if you wanted 500 records you would use this pseudocode:

- Get activities 1-100 (limit 100, skip 0)

- Get activities 101-200 (limit 100, skip 100)

- ...

- Get activities 401-500 (limit 100, skip 400)

To make this loop more intelligent, you would add in some if statements to check the timestamps to see if you should pull more records.

Reminder, the limit/skip properties are not nested under the filters property. Here is a simple example of a body that skips 10 records, gets 10 records, and pulls only activities for Salesforce.

{

"skip":10,

"limit":10,

"filters":{

"service":{

"eq":[11114]

}

}

}

P.S. I would also like to mention that I and a colleague wrote a PowerShell module to make it easier to do these ad-hoc queries straight form command line. Have a look at the following link for how to get started: https://github.com/Microsoft/Cloud-App-Security/wiki/2.-Getting-Started

I hope this helps.

Sanket Yeram

Copper Contributor

Nov 17, 2017

Mike Kassis
Thanks a lot.

I tried by putting it into script.

But when I put my query into the loop, it starts fetching all the events including the one I m interested.

e.g. I m interested lets say only in "Malware Events" which I have shown below, it fetches malware events along with other, whereas I want only malware event.

Here is the script I m using.

#!/bin/bash
for (( i=0;i<200;i=i+100));
do curl -XPOST "https://myportal/api/v1/activities/" -H "Authorization: Token mykey" -d '{ "filters": { "activity.actionType" : { "eq" : "EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL"} }, "limit":100, "skip" : $i } ' >> events.txt
done

Thanks.

Sanket Yeram

Copper Contributor

Nov 17, 2017

Mike Kassis

I just added ' ' around the $i which was missing in earlier case.

Its working perfectly fine now. so final script is. I m playing with timestamp now :)

Thanks