Copy activity Rest API Pagination

Copper Contributor

I am ingesting data from Rest API source using Copy activity and need help in setting up Pagination.

 

Update - Increment should be 1 instead of 100 and now it works but now I want to see if I can do it only offset url and remove end condition. I want range condition to be dynamic as "RANGE:1:($.count/pagesize):1"  Is this possible?

 

Rest API source returns following response. 

{

status : 200

data: []

count: 156

}

It will always returns total count of records and based on it I need to read through different pages. By default only 100 records are returned each page. 

I tried this setup as mentioned in https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-4varia...

GayatriKrishnan_1-1694664614246.png

It only got through first 100 records. I see the same behaviour when doing a Preview wherein second call is probably going through but response is empty. I feel probably the offset is not passed as expected which is causing the issue. Through postman I see responses for 2 pages and 3 rd page data array is empty.

 

0 Replies