ADF - REST API Copy Data Activity - Best Practices

Hi,

You are already on the right track, and your observations are valid. There are two main areas to improve here: safe parallelism and pagination strategy.

First, regarding parallelism and data integrity. The issue you are seeing when disabling sequential execution is very likely caused by shared state such as variables or outputs being reused across iterations. In ADF, parallel execution can lead to data mix-ups if global variables are used.

A more reliable approach is to use a ForEach activity with parallel execution enabled, where each iteration processes one franchise independently. Make sure you do not use shared variables inside the loop, and instead rely on item() and pipeline parameters. Also, ensure that your sink path is dynamically generated and unique per franchise and per run.

For example, write files to a path like: franchise_{franchiseId}/report_{runId}.json

This guarantees isolation and avoids overwriting data. It is also recommended to keep pagination sequential within each franchise, while running multiple franchises in parallel. This gives better performance without compromising data integrity.

Second, regarding pagination and loop control. Moving away from Lookup was a good decision from a performance perspective. However, using data read size as a condition is not very robust.

A better approach is to use Copy Activity output metadata, especially rowsCopied. You can stop your loop when rowsCopied equals zero, which is more reliable than checking the data size.

If the API supports it, an even better option is to rely on response metadata such as a hasMore flag or a next page or token value. This is more stable than inferring behavior from indirect indicators.