SOLVED

Fetch Activity with Curl retrieves limited entries

%3CLINGO-SUB%20id%3D%22lingo-sub-126912%22%20slang%3D%22en-US%22%3EFetch%20Activity%20with%20Curl%20retrieves%20limited%20entries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-126912%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20All%2C%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20trying%20to%20fetch%20activities%20of%20last%20few%26nbsp%3Bdays%20and%20I%20am%20using%20below%20command%2C%3C%2FP%3E%3CP%3Ecurl%20%22%3CA%20href%3D%22http%3A%2F%2Fmydomain.cloudsecurity.com%2Fapi%2Fv1%2Factivities%2F%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttp%3A%2F%2Fmydomain.cloudsecurity.com%2Fapi%2Fv1%2Factivities%2F%3C%2FA%3E%22%20-H%20'Authorization%3A%20Token%20mytoken%22%3CBR%20%2F%3E-d%20'%20%7B%20%22filters%22%20%3A%20%7B%20%22activity.actionType%22%26nbsp%3B%3A%20%7B%20%22eq%22%20%3A%20%22someevent%22%20%7D%2C%20%22date%22%20%3A%20%7B%20%22gte%22%20%3A%20%22i%3Axxxxxx%22%20%7D%26nbsp%3B%20%7D%26nbsp%3B%20%26nbsp%3B%20%7D%20'%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20am%20getting%20only%20limited%20entries%2C%20I%20tried%20with%20%22limits%22%20parameter%20as%20well%2C%20but%20its%20giving%20%22internal%20error%22.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ELet%20us%20know%2C%20what%20query%20need%20to%20put%20in%20order%20to%20get%20the%20specific%20activities%20of%20last%20few%20days.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-126912%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3ECloud%20App%20Security%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-129250%22%20slang%3D%22en-US%22%3ERe%3A%20Fetch%20Activity%20with%20Curl%20retrieves%20limited%20entries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-129250%22%20slang%3D%22en-US%22%3E%3CP%3EGlad%20to%20hear%20you%20got%20it%20all%20working!%20%3A)%3C%2Fimg%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-129246%22%20slang%3D%22en-US%22%3ERe%3A%20Fetch%20Activity%20with%20Curl%20retrieves%20limited%20entries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-129246%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F95282%22%20target%3D%22_blank%22%3E%40Mike%20Kassis%3C%2FA%3E%3C%2FP%3E%3CP%3EI%20just%20added%20'%20'%20around%20the%20%26nbsp%3B%3CSPAN%3E%24i%20which%20was%20missing%20in%20earlier%20case.%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3EIts%20working%20perfectly%20fine%20now.%20so%20final%20script%20is.%20I%20m%20playing%20with%20timestamp%20now%20%3A)%3C%2Fimg%3E%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%3CSPAN%3E%23!%2Fbin%2Fbash%3CBR%20%2F%3Efor%20((%20i%3D0%3Bi%26lt%3B200%3Bi%3Di%2B100))%3B%3CBR%20%2F%3Edo%20curl%20-XPOST%20%22https%3A%2F%2Fmyportal%2Fapi%2Fv1%2Factivities%2F%22%20-H%20%22Authorization%3A%20Token%20mykey%22%20-d%20'%7B%20%22filters%22%3A%20%7B%20%22activity.actionType%22%20%3A%20%7B%20%22eq%22%20%3A%20%22EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL%22%7D%20%7D%2C%20%22limit%22%3A100%2C%20%22skip%22%20%3A%20'%24i'%20%7D%20'%20%26gt%3B%26gt%3B%20events.txt%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CSPAN%3EThanks%3C%2FSPAN%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-129234%22%20slang%3D%22en-US%22%3ERe%3A%20Fetch%20Activity%20with%20Curl%20retrieves%20limited%20entries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-129234%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F95282%22%20target%3D%22_blank%22%3E%40Mike%20Kassis%3C%2FA%3E%26nbsp%3B%3CBR%20%2F%3EThanks%20a%20lot.%3C%2FP%3E%3CP%3EI%20tried%20by%20putting%20it%20into%20script.%3C%2FP%3E%3CP%3EBut%20when%20I%20put%20my%20query%20into%20the%20loop%2C%20it%20starts%20fetching%20all%20the%20events%20including%20the%20one%20I%20m%20interested.%3C%2FP%3E%3CP%3Ee.g.%20I%20m%20interested%20lets%20say%20only%20in%20%22Malware%20Events%22%20which%20%26nbsp%3BI%20have%20shown%20below%2C%20it%20fetches%20malware%20events%20along%20with%20other%2C%20whereas%20I%20want%20only%20malware%20event.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHere%20is%20the%20script%20I%20m%20using.%3C%2FP%3E%3CP%3E%23!%2Fbin%2Fbash%3CBR%20%2F%3Efor%20((%20i%3D0%3Bi%26lt%3B200%3Bi%3Di%2B100))%3B%3CBR%20%2F%3Edo%20curl%20-XPOST%20%22https%3A%2F%2Fmyportal%2Fapi%2Fv1%2Factivities%2F%22%20-H%20%22Authorization%3A%20Token%20mykey%22%20-d%20'%7B%20%22filters%22%3A%20%7B%20%22activity.actionType%22%20%3A%20%7B%20%22eq%22%20%3A%20%22EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL%22%7D%20%7D%2C%20%22limit%22%3A100%2C%20%22skip%22%20%3A%20%24i%20%7D%20'%20%26gt%3B%26gt%3B%20events.txt%3CBR%20%2F%3Edone%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThanks.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-128245%22%20slang%3D%22en-US%22%3ERe%3A%20Fetch%20Activity%20with%20Curl%20retrieves%20limited%20entries%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-128245%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Sanket%2C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20activities%20API%20endpoint%20has%20a%20query%20limit%20of%20100%20records%20to%20prevent%20overloading%20the%20endpoint%20with%20any%20one%20request.%20If%20you%20want%20to%20retrieve%20more%20than%20100%20records%2C%20you%20will%20need%20to%20add%20that%20logic%20to%20your%20script%20to%20update%20the%20parameters%20of%20the%20curl%20request%20through%20a%20loop.%20For%20example%2C%20if%20you%20wanted%20500%20records%20you%20would%20use%20this%20pseudocode%3A%3C%2FP%3E%0A%3CP%3E-%20Get%20activities%201-100%20(limit%20100%2C%20skip%200)%3C%2FP%3E%0A%3CP%3E-%20Get%20activities%20101-200%20(limit%20100%2C%20skip%20100)%3C%2FP%3E%0A%3CP%3E-%20...%3C%2FP%3E%0A%3CP%3E-%20Get%20activities%20401-500%20(limit%20100%2C%20skip%20400)%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETo%20make%20this%20loop%20more%20intelligent%2C%20you%20would%20add%20in%20some%20if%20statements%20to%20check%20the%20timestamps%20to%20see%20if%20you%20should%20pull%20more%20records.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EReminder%2C%20the%20limit%2Fskip%20properties%20are%20not%20nested%20under%20the%20filters%20property.%20Here%20is%20a%20simple%20example%20of%20a%20body%20that%20skips%2010%20records%2C%20gets%2010%20records%2C%20and%20pulls%20only%20activities%20for%20Salesforce.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%7B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%22skip%22%3A10%2C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%22limit%22%3A10%2C%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%22filters%22%3A%7B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%22service%22%3A%7B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%22eq%22%3A%5B11114%5D%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%26nbsp%3B%20%7D%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%20%26nbsp%3B%20%7D%3C%2FP%3E%0A%3CP%3E%7D%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EP.S.%20I%20would%20also%20like%20to%20mention%20that%20I%20and%20a%20colleague%20wrote%20a%20PowerShell%20module%20to%20make%20it%20easier%20to%20do%20these%20ad-hoc%20queries%20straight%20form%20command%20line.%20Have%20a%20look%20at%20the%20following%20link%20for%20how%20to%20get%20started%3A%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fgithub.com%2FMicrosoft%2FCloud-App-Security%2Fwiki%2F2.-Getting-Started%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fgithub.com%2FMicrosoft%2FCloud-App-Security%2Fwiki%2F2.-Getting-Started%3C%2FA%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20hope%20this%20helps.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
New Contributor

Hi All,

 

I am trying to fetch activities of last few days and I am using below command,

curl "http://mydomain.cloudsecurity.com/api/v1/activities/" -H 'Authorization: Token mytoken"
-d ' { "filters" : { "activity.actionType" : { "eq" : "someevent" }, "date" : { "gte" : "i:xxxxxx" }  }    } '

 

I am getting only limited entries, I tried with "limits" parameter as well, but its giving "internal error".

 

Let us know, what query need to put in order to get the specific activities of last few days.

 

 

Thanks.

4 Replies
Highlighted
Best Response confirmed by Sanket Yeram (New Contributor)
Solution

Hi Sanket,

 

The activities API endpoint has a query limit of 100 records to prevent overloading the endpoint with any one request. If you want to retrieve more than 100 records, you will need to add that logic to your script to update the parameters of the curl request through a loop. For example, if you wanted 500 records you would use this pseudocode:

- Get activities 1-100 (limit 100, skip 0)

- Get activities 101-200 (limit 100, skip 100)

- ...

- Get activities 401-500 (limit 100, skip 400)

 

To make this loop more intelligent, you would add in some if statements to check the timestamps to see if you should pull more records.

 

Reminder, the limit/skip properties are not nested under the filters property. Here is a simple example of a body that skips 10 records, gets 10 records, and pulls only activities for Salesforce.

 

 

{

    "skip":10,

    "limit":10,

    "filters":{

        "service":{

            "eq":[11114]

        }

    }

}

 

P.S. I would also like to mention that I and a colleague wrote a PowerShell module to make it easier to do these ad-hoc queries straight form command line. Have a look at the following link for how to get started: https://github.com/Microsoft/Cloud-App-Security/wiki/2.-Getting-Started

 

I hope this helps.

 

Highlighted

@Mike Kassis 
Thanks a lot.

I tried by putting it into script.

But when I put my query into the loop, it starts fetching all the events including the one I m interested.

e.g. I m interested lets say only in "Malware Events" which  I have shown below, it fetches malware events along with other, whereas I want only malware event.

 

Here is the script I m using.

#!/bin/bash
for (( i=0;i<200;i=i+100));
do curl -XPOST "https://myportal/api/v1/activities/" -H "Authorization: Token mykey" -d '{ "filters": { "activity.actionType" : { "eq" : "EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL"} }, "limit":100, "skip" : $i } ' >> events.txt
done

 

 

Thanks.

Highlighted

@Mike Kassis

I just added ' ' around the  $i which was missing in earlier case.

Its working perfectly fine now. so final script is. I m playing with timestamp now :)

 

#!/bin/bash
for (( i=0;i<200;i=i+100));
do curl -XPOST "https://myportal/api/v1/activities/" -H "Authorization: Token mykey" -d '{ "filters": { "activity.actionType" : { "eq" : "EVENT_CATEGORY_MALWARE_DETECTED_IN_EMAIL"} }, "limit":100, "skip" : '$i' } ' >> events.txt

 

Thanks

 

Highlighted

Glad to hear you got it all working! :)