Pagination issues during rest api calls

Frequent Visitor

Hi,

I have a pipeline which basically firsts gets a access token by making a http request and once I have the token I'm doing a get request to an API with the access token as a header.

As the REST API o/p is limited I need to make multiple requests for pagination so I did follow the steps mentioned here : https://docs.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

There are few problems with the NextPageUrl which i receive from the REST get request, The NextPageUrl starts with http and based on my experience if the Linked_service Baseurl does not start with HTTPS it does not accept the headers passed so I need to change the url from this  "http://xyz.com/api/v1/?page=1" to "https://xyz.com/api/v1/?page=1" so i tried the replace expression but It says I cant write it this way "@replace('$.ConnectResponse.Metadata.Paging.NextPageURL', 'http', 'https')" or "@replace($.ConnectResponse.Metadata.Paging.NextPageURL, 'http', 'https')"

 

Please advise how can I change the string?

 

Secondly I though okay let me pass the page number manually but how do I add a number to the current page number and stop once it reaches the max?

 

from my API O/p I get example TotalPages=3 and CurrentPage=1 number

so I though I could pass the queryparameter in pagination like below

key = QueryParameters.page and value = @add($.ConnectResponse.Metadata.Paging.NextPageURL, 1)      ----- I've tried with quotes and without quotes --- and it does work as expected.

 

Please advise how can I fix this?

 

Lastly

 

Is this how i pass a header authorization token in pagination?

 

key = Headers['Authorization']

value = Bearer @{activity('Login').output.access_token}

 

I mean this is how I pass the header parameters for the rest source so I'm assuming I just need to pass it the same way correct me if I'm wrong.

 

Thank you in advance.

 

 

0 Replies