Forum Discussion
REST API and Pagination
So, I'm having some issues getting all the data from a Rest API into Azure Data Factory. Currently I have everything setup correctly the mapping the source etc. The REST API returns only 100 results and there are pagination rules. I'm not 100% sure how-to setup these rules in Azure. Currently the Copy Data activity only gets the first page of 100 and that's it.
},
"pagination": {
"endCursor": "965125e2-b907-493e-b0fd-c938531f60e2",
"hasNextPage": true
}
}
Each page ends with this Json response. Where endCursor needs to be added to the relative URL like this
?after=965125e2-b907-493e-b0fd-c938531f60e2
For Example, the first request would be
"https://api.samsara.com/fleet/reports/vehicles/fuel-energy?startDate=2023-12-29T00%3A00%3A00.00-06%3A00&endDate=2023-12-29T00%3A00%3A00.00-06%3A00"
Then the next request
"https://api.samsara.com/fleet/reports/vehicles/fuel-energy?startDate=2023-12-29T00%3A00%3A00.00-06%3A00&endDate=2023-12-29T00%3A00%3A00.00-06%3A00?after=965125e2-b907-493e-b0fd-c938531f60e2"In the source dataset the relative URL is also set as dynamic content.
@{concat('/fleet/reports/vehicles/fuel-energy?startDate=', encodeURIComponent(formatDateTime(subtractFromTime(utcnow(), 96, 'Hour'), 'yyyy-MM-ddTHH:mm:ss.fffffffZ')),'&endDate=', encodeURIComponent(formatDateTime(subtractFromTime(utcnow(), 72, 'Hour'), 'yyyy-MM-ddTHH:mm:ss.fffffffZ')))}How would I define this requirement here? Or am I looking at this totally wrong?
- Use an "Until" activity and a "Lookup", the "Copy" activity & "Set Variable" within it to achieve your goal.
The variable will be initially defined as @{string('')} and the output of the "endCursor" response should to set to it in each iteration of the Until activity, till no records fetched. Hope it helps. For anyone who might find this page via search engine I did figure out what I needed to do.
As long as "?after=" was in the relative URL value all I needed to do was put the word "after" as a QueryParameter and where it was located in the body response of the json data. For this API it was Body->pagination.endCursor.
