Forum Discussion
Copy activity Rest API Pagination
I am ingesting data from Rest API source using Copy activity and need help in setting up Pagination.
Update - Increment should be 1 instead of 100 and now it works but now I want to see if I can do it only offset url and remove end condition. I want range condition to be dynamic as "RANGE:1:($.count/pagesize):1" Is this possible?
Rest API source returns following response.
{
status : 200
data: []
count: 156
}
It will always returns total count of records and based on it I need to read through different pages. By default only 100 records are returned each page.
I tried this setup as mentioned in https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-4variables-are-in-absoluteurlqueryparametersheaders-the-end-variable-is-not-pre-defined-and-the-end-condition-is-based-on-the-response
It only got through first 100 records. I see the same behaviour when doing a Preview wherein second call is probably going through but response is empty. I feel probably the offset is not passed as expected which is causing the issue. Through postman I see responses for 2 pages and 3 rd page data array is empty.