APIs often return a large amount of data. Pagination allows you to retrieve a manageable subset of this data at a time, preventing unnecessary strain on resources and reducing the likelihood of timeouts or errors.
In this example, the client starts by requesting the first page of data from the server. The server responds with both the data and metadata indicating the current page, the total number of records. The client then proceeds to request subsequent pages of data until it reaches the last page. This approach allows for efficient data retrieval and processing without overwhelming the client or the server.
we want to get a file in ADLS containing all data from API without the need to use other activities like until/forEach, we want the Copy activity to perform all the pagination needed to collect all the data.
1. Basic knowledge in Rest API.
2. Workspace in Microsoft Fabric.
3. ADLS storage account.
default limit according to pokeapi documentation is 20 records per request. In this tutorial, i want to limit records up to 500 records per request like so:
Call-To-Action: - Make sure to establish all connections in ADLS and in Fabric workspace. - check MS documentation on pagination in copy activity. - Please help us improve by sharing your valuable feedback. - Follow me on LinkedIn: Sally Dabbah | LinkedIn