Forum Discussion
Solved
Logic Apps: how to pull large volumes of data from a REST API
Hi everyone,
I'm pulling from a REST API that only allows 1000 rows at a time.
This works for a single pull.
My goal is to pull all 50k rows and dump it to a log analytics table.
Can you please suggest the best approach to do this with logic apps?
I'm concerned there may be limitations like dumping to an array and then writing to the table?
Your suggestions are appreciated.
(@liortamir would you have any suggestions? or anyone?)
SocInABox It depends if the REST API you are calling allows you to skip a certain number of items. If it does, you will need to keep track of which group of 1000 you are calling. Then, in a loop, call the REST API making sure to skip the correct amount. Repeat until you get less than 1000 items returned. Pseudo code would be
Initialize variable, SKIPAMOUNT, to zero Do until amountReturned<1000 Call REST API passing value from SKIPAMOUNT as the amount to skip Add 1000 to SKIPAMOUNT Process returned values
SocInABox It depends if the REST API you are calling allows you to skip a certain number of items. If it does, you will need to keep track of which group of 1000 you are calling. Then, in a loop, call the REST API making sure to skip the correct amount. Repeat until you get less than 1000 items returned. Pseudo code would be
Initialize variable, SKIPAMOUNT, to zero Do until amountReturned<1000 Call REST API passing value from SKIPAMOUNT as the amount to skip Add 1000 to SKIPAMOUNT Process returned values- Thank you Gary! We will try this and let you all know how it went.
Like you said not all APIs are created equal so this is very helpful.
