Logic Apps: how to pull large volumes of data from a REST API

[SocInABox]

Hi everyone,

I'm pulling from a REST API that only allows 1000 rows at a time.

This works for a single pull.

My goal is to pull all 50k rows and dump it to a log analytics table.

Can you please suggest the best approach to do this with logic apps?

I'm concerned there may be limitations like dumping to an array and then writing to the table?

Your suggestions are appreciated.

(@liortamir would you have any suggestions? or anyone?)

[#2 · Accepted solution]

@SocInABox It depends if the REST API you are calling allows you to skip a certain number of items.  If it does, you will need to keep track of which group of 1000 you are calling. Then, in a loop, call the REST API making sure to skip the correct amount.  Repeat until you get less than 1000 items returned.  Pseudo code would be 

Initialize variable, SKIPAMOUNT, to zero
Do until amountReturned<1000
  Call REST API passing value from SKIPAMOUNT as the amount to skip
  Add 1000 to SKIPAMOUNT
  Process returned values

[#3]

Thank you Gary! We will try this and let you all know how it went.
Like you said not all APIs are created equal so this is very helpful.