User Profile
GayatriKrishnan
Copper Contributor
Joined Sep 13, 2023
User Widgets
Recent Discussions
Copy activity Rest API Pagination
I am ingesting data from Rest API source using Copy activity and need help in setting up Pagination. Update - Increment should be 1 instead of 100 and now it works but now I want to see if I can do it only offset url and remove end condition. I want range condition to be dynamic as "RANGE:1:($.count/pagesize):1" Is this possible? Rest API source returns following response. { status : 200 data: [] count: 156 } It will always returns total count of records and based on it I need to read through different pages. By default only 100 records are returned each page. I tried this setup as mentioned in https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-4variables-are-in-absoluteurlqueryparametersheaders-the-end-variable-is-not-pre-defined-and-the-end-condition-is-based-on-the-response It only got through first 100 records. I see the same behaviour when doing a Preview wherein second call is probably going through but response is empty. I feel probably the offset is not passed as expected which is causing the issue. Through postman I see responses for 2 pages and 3 rd page data array is empty.
