ADF - REST API Copy Data Activity - Best Practices

Hey everyone,

I'm relatively new to Azure and am using Azure Data Factory (ADF) to extract data from Vonigo via their API.

Everything is working, but I'm looking for ways to improve efficiency. Right now it takes 30+ minutes to pull all franchise data for a given report.

The challenge is that Vonigo only allows reports to be run for one franchise at a time, and switching franchises requires a separate API call. My current process is:

1. Select a franchise from a list
2. Run all required reports for that franchise
3. Save the results to Blob Storage
4. Move to the next franchise and repeat

(We'll eventually ingest the files into our silver layer for transformation.)

Speed Issue

One thing that significantly slows the pipeline down is that many activities are running sequentially. If I disable sequential execution, I've seen cases where data gets written to the wrong destination or associated with the wrong franchise.

Has anyone successfully parallelized a similar process while maintaining data integrity? Are there specific points in the workflow where parallel execution would be safe?

Pagination / Loop Issue

Originally, I used a Lookup activity to inspect the most recently created file. An If activity would then determine whether the file contained any records:

• If records existed, increment the page number and continue.
• If no records existed, end the loop.

This worked, but the Lookup activity added noticeable overhead.

To improve performance, I changed the logic to use the Copy Activity output instead. Specifically, I'm checking the amount of data read from the last API call. Pages with no records appear to consistently return the same data-read value, so I use that to determine when to stop paging.

This approach is much faster, but it feels more fragile since it's relying on an indirect indicator rather than the actual record count.

Would you trust the Copy Activity output in this scenario, stick with the Lookup approach, or recommend a different pattern altogether?

Thanks for any suggestions.