Forum Discussion
Fetch Activity with Curl retrieves limited entries
- Nov 15, 2017
Hi Sanket,
The activities API endpoint has a query limit of 100 records to prevent overloading the endpoint with any one request. If you want to retrieve more than 100 records, you will need to add that logic to your script to update the parameters of the curl request through a loop. For example, if you wanted 500 records you would use this pseudocode:
- Get activities 1-100 (limit 100, skip 0)
- Get activities 101-200 (limit 100, skip 100)
- ...
- Get activities 401-500 (limit 100, skip 400)
To make this loop more intelligent, you would add in some if statements to check the timestamps to see if you should pull more records.
Reminder, the limit/skip properties are not nested under the filters property. Here is a simple example of a body that skips 10 records, gets 10 records, and pulls only activities for Salesforce.
{
"skip":10,
"limit":10,
"filters":{
"service":{
"eq":[11114]
}
}
}
P.S. I would also like to mention that I and a colleague wrote a PowerShell module to make it easier to do these ad-hoc queries straight form command line. Have a look at the following link for how to get started: https://github.com/Microsoft/Cloud-App-Security/wiki/2.-Getting-Started
I hope this helps.
I just added ' ' around the $i which was missing in earlier case.
Its working perfectly fine now. so final script is. I m playing with timestamp now :)
#!/bin/bash
for (( i=0;i<200;i=i+100));
do curl -XPOST "https://myportal/api/v1/activities/" -H "Authorization: Token mykey" -d '{ "filters": { "activity.actionType" : { "eq" : "EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL"} }, "limit":100, "skip" : '$i' } ' >> events.txt
Thanks
Glad to hear you got it all working! :)